Planning with an Embodied Learnable Memory
Authors: Priyam Parashar
Jacob Krantz
Matthew Chang
Kavit Shah
Xavier Puig
Roozbeh Mottaghi
Venue: ICLR 2026 Poster (2026)
Keywords: Embodied Memory
Planning
Reinforcement Learning
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose a novel learnable memory that, combined with planners, enables agents to plan tasks in large indoor spaces. We also introduce two methods to improve planning using human-in-the-loop data and a novel value-free RL training method.

Abstract: We develop a novel memory representation for embodied planning models performing long-horizon mobile manipulation in dynamic, large-scale indoor environments. Prior memory representations fall short in this setting, as they struggle with object movements, suffer from computational deficiencies, and often depend on the heuristic integration of multiple models. To overcome these limitations, we present the Embodied Perception Memory (EPM), a learnable memory designed for embodied planning. EPM is implemented as a unified Vision-Language Model (VLM) that uses egocentric vision to maintain and update a textual environment representation. We further introduce two complementary methods for training planners to leverage the EPM: an imitation strategy that uses human trajectories for natural exploration and interaction, and a novel reinforcement learning approach, Dynamic Difficulty-Aware Fine-Tuning (DDAFT), which improves planning performance via difficulty-aware exploration. Our memory representation, when integrated with our planning training methods, leads to significant improvements on planning tasks, showing up to a 55% increase in success rate on the PARTNR benchmark compared to strong baselines. Also, our planning method outperforms these baselines even when they have access to groundtruth perception.

Published as a conference paper at ICLR 2026
PLANNING WITH AN EMBODIED LEARNABLE MEMORY
Priyam Parashar" Jacob Krantz"
Matthew Chang"
Kavit Shah
Xavier Puig'
Roozbeh Mottaghi'
ABSTRACT
We develop a novel memory representation for embodied planning models perform- ing long- long-horizon mobile manipulation in dynamic, large-scale indoor environments. Prior memory representations fall short in this setting, as they struggle with object movements, suffer from computational deficiencies, and often depend on the heuris- tie integration of multiple models. To overcome these limitations, we present the Embodied Perception Memory (EPM), a learnable memory designed for embodied planning. EPM is implemented as a unified Vision- AasionLanguuae Model (VLM) that uses egocentric vision to maintain and update a textual environment representa- tion. We further introduce two complementary methods for training planners leverage the EPM: an imitation strategy that uses human trajectories for natural exploration and interaction, and novel reinforcement learning approach, Dynamic DifficultyAAware Fine-Tuning (DDAFT), which improves planning performance via difficulty- exploration. Our memory representation, when integrated with our planning training methods, leads to significant improvements on planning tasks, showing up to a 55% increase in success rate on the PARTNR benchmark compared strong baselines. Also, our planning method outperforms these baselines even when they have access to groundtruth perception.
INTRODUCTION
Imagine a human doing tasks in their home. Without even thinking, they can move towards the cabinet where they last placed a clean mug and plate, bring them to the table, and prepare a coffee to start the day. Our ability to perform these tasks effortlessly rely on an interplay between memory, perception, and planning. Memory provides us with mental map of the objects in the environment, which we update based on our observations, and use to plan effectively. For robots to perform tasks in indoor environments with such efficiency, we need to equip them with the same capabilities: perceive surroundings, remember where objects are over long periods of time, and plan actions accordingly.
Developing an effective memory representation for embodied planning models that tackle long- horizon tasks poses significant challenges. The memory must not only capture the spatial configura- tion, context and state of objects in the scene, but also update dynamically as agents explore or the environment changes. Further, the memory must integrate with a planning system to provide context that enables the agent to perform task.
The current literature on memory representations for planning in embodied tasks has certain limi- tations. A primary shortcoming of many existing approaches their inability to effectively handle dynamic environments (Gu et al., 2024a; Jatavallabhula et al., 2023), where objects are frequently moved and changed by the robot humans in the environment. Many of the existing approaches encode the environment by storing the agent's egocentric observations (Liu et al., 2024a; Yang et al., 2025), or featurized point-cloud of the scene (Jatavallabhula et al., 2023; Liu al., 2024a), which can be queried to retrieve relevant information. While multi-stage pipelines combining standalone detection and segmentation models with planners have been effective, particularly in complex scenes with small or rare objects, such representations often have high memory requirements and compu- tational cost when querying large models multiple times for planning. Furthermore, formulating precise queries can be challenging and impractical for the planner. As found in prior work (Chang et al., 2024), naive feature matching with language queries fails to achieve both high precision and
* Equal contribution. Work done FAIR, Meta.

---

Published as a conference paper at ICLR 2026
Bring scissons, phona credit card from the table to the counter
Agent Observation
Updates
Add (0.3, 0.8 Scieens Fy Add (0.6, 0.9) "ho Chiity"
Remove: Enilyy4
EPM
Entity:: Coffee table
Enoity! Entity:: Phone on Failyy
Entity:::
High-Level Planner Action: Grab Object.
Figure 1: We present memory representation for embodied planing in large-scale environments. Our method includes Embodied Perception Memory (EPM), a VLM-based system that represents the environment from egocentric observations. Our novel LLM-based planner trained with human demonstrations and RL reason over EPM to plan agent actions.
recall. To address these challenges, propose novel memory representation that is adaptable, efficient, can handle dynamic environments, and easily interacts with task planners.
In particular, introduce the Embodied Perception Memory (EPM), novel form of learnable external memory for planning in mobile manipulation tasks. EPM is a single Vision-Language Model (VLM) trained to perform various operations such as adding, removing, and updating informa- tion on scene representations derived from egocentric observations. This end-to-end approach eliminates the need for heuristics and enables faster inference than approaches consisting of multiple models. EPM encodes an object-centric representation of the environment, where each object is characterized by 3D coordinates and a natural language description of the object's state and context. This naturally results in textual representation of the scene, which can be used by LLM-base planners that have demonstrated strong performance in planning tasks.
This representation enables the planner access enviromment information directly, without the need for explicit, bask-based queries (in contrast to approaches that rely on such queries, e.g., Yang et al. (2025)). Hence, the planner can focus only generating actions to interact with the environment, rather than issuing separate API calls to retrieve information. As a result, the planner can be improved with robot interaction data where such queries may not be available. To leverage this, introduce strategy to train the planning model on human demonstrations of mobile manipulation tasks, resulting improved exploration and task execution. Then, we develop novel online RL-based approach that leverages insights from LLM reasoning works (Shao et al., 2024; Yuan et al., 2023; Tong et al., 2024) improve planning from online experience. Specifically, we propose Dynamic Difficulty-Awar Fine- Muni-Tnnng (DDAFT), which is novel form of wwteee-funtio-fffiee RL for LLMs that explores the planning space on difficult instructions with online sampling. This induces a curriculum over training data, leading to sample efficient policy improvement.
We evaluate EPM's s ability track the environment in simulated and real- world egocentric interaction datasets, showing that approach can track the environment in both domains despite being trained only in simulation. We also evaluate EPM together with our proposed planner PARTNR (Chang et al., 2024), simulation benchmark measuring agents' abilities perform tasks specified by language. Our results show significant improvements over strong baselines such as Liu et al. (2024a) and Chang et al. (2025), achieving absolute success rate gains of 55% and 12%, respectively, planning tasks. Additionally, we show that human demonstration training outperforms zero-shot baselines when zero-shot planners provided groundtruth perception. Further, our DDAFT approach improves performance across all settings, providing recipe to enhance embodied planning. In summary, our contributions are:
We introduce the Embodied Perception Memory, novel form of learnable memory for embodied planning tasks that effectively handles dynamic environments as a single model, and can be seamlessly integrated with L--M-based planners.
2

---

Published as a conference paper at ICLR 2026
Open Vocahulary
Object Relationships
Error
Single
Method
Dynamics
No Query
Corectton
Model
Concepiluusion (Jatavallabhula et al,, 2023)
ConceptGraphs (Gu et al., 2024a)
3D-Mem (Yang et al., 2025)
DynaMem (Liu et al., 2024a)
EPM (ours)
Table I: Comparison to different memory representations. We compare EPM with other popular embodied memory representations. Object Relationships: whether object-furniturr relationships are encoded. Dynamics: whether the representation allows for changing environment. No Query: whether the representation needs be queried retrieve information.
We present novel planning approach that leverages human demonstration data and DDAFT, demonstrating significant performance improvements over strong baselines.
2 RELATED WORK
Embodied Planning with LLMs and VLMs. A large body of work has studied LLMs as policies for embodied agents, leveraging their strong capabilities in planning and common sense reasoning, but also highlighting limitations in grounding them agent's 's observations and actions (Ahn et al., 2022; Huang et al., 2023c; Zeng et al., 2022). To improve grounding, some methods integrate language models with external modules, which are exposed to the LLMs as APIs to reason over the agent's observations and commands (Schick et al., 2023). These modules can consist of learned or hand- designed low-level controllers (Wang et al., 2024; Driess al., 2023), or perception models (Liu et al., 2024a; Chang et al., 2025; Rana et al., 2023). DynaMem (Liu et al., 2024a) maintains point-cloud representation of the environment, which queried by an LLM to obtain task-relevant information. As result, planning with such representations requires an LLM to generate both text queries to extract information from the environment and high-level actions. Other approaches build text representation of the environment in the form of scene graph, which is used by the LLM for planning (Gu et al., 2024a; Rana et al., 2023; Chang et al., 2025). This separation decouples planning from perception and interaction, but sensitive the representations of each module, and errors which propagate throughout the system, resulting in planning failures (Chang et al., 2025; Li al., 2025). Additionally, maintaining multiple specialized modules introduces computational overhead that increases overall system requirements. Furthermore, they struggle to model dynamic scenes, requiring heuristic track objects over time. More recently, multiple works have used VLMSs to predict agent actions directly from observations, either by predicting parameters for pre-defined robot skills (Nasiriany et al., 2024; Huang al., 2023b; 2024) or finetuning them to directly predict low-level actions (Shi et al., 2025; Bjorck et al., 2025). These approaches ground actions directly into visual observations, but are limited by VLMs' memory constraints, and thus insufficient for tasks requiring long- horizoorizon exploration in large environments. Our work is closer to the first family of approaches, but instead of relying on off off-the-shelf models, we propose learnable a memory system trained on embodied data, and propose approach allow L.MM planners to effectively work with noisy memory representations.
Spatio-Semantic Memory. There is long history of research spatio- sentinieemanic memory, i.e. remembering the locations of elements of the scene jointly with semantic information about those elements. Early works (Aydemir al., 2013; Gupta al., 2017) focus primarily on navigation and are not designed to handle open vocabulary instructions. A common line of work uses a VLM such as CLIP (Radford al., 2021) or LSeg (Li al., 2022) produce language-aligned features for views of the scene. These features are typically projected into 3D using a point cloud (Huang et al., 2023a; Ha Song, 2022; Liu et al., 2024a; Jatavallabhula al., 2023), neural radiance field (Kerr et al., 2023; Qiu et al., 2024), or Gaussian splat (Ji et al., 2024; Shorinwa et al., 2024). While these methods permit open-vocabee vocabulary queries over the scene, such features do not natively encode inter-entity relationships making referential or contextual queries harder to address. To interface with planning system, these approaches rely on matching spatial feature vectors with some query (typically via cosine similarity), querying VLM on stored images (Yang al., 2025), store per-object fcatures or labels based on location information (Gu et al., 2024a; Maggio et al., 2024; Chang et al., 2024). In contrast, EPM directly produces text representing entities and entity eetity-reaaioonshis, allowing seamless
3

---

Published as a conference paper at ICLR 2026
integration with an LLM planner. Further, all of these works struggle with object re-association and correction of spurious objects, relying on heuristics or thresholds. Re-association and correction are learned within EPM, avoiding these issues. Short-horizon planning over these representations may be achieved by transforming the current scene to a goal image (Wang et al., 2023), or leveraging differentiable scene representation (Bolte al., 2023). However, long-horizon planning with contextual features is most commonly performed with a pre-trained LLM. This can be done by finding locations via cosine similarity with queries produced from the LLM (Chen al., 2023; Yan et al., 2025), or explicitly building a text representation of the scene (Gu et al., 2024a; Koch et al., 2024). EPM uses the latter approach, using the descriptions produced by the learned model directly in the context of an LLM. In Table we provide an overview of different memory representations.
Finally, several works build dynamic representations during simultaneous localization and mapping (SLAM) (Jiang et al., 2024; Wu et al., 2024; Fan et al., 2022; Yu et al., 2018). Such representations generally fail to meet the needs of embodied planning: e.g.. object re-initialization instead of tracking, limited vocabularies, and insufficient interpretability, We expand this discussion in Appendix A.
3 THE EMBODIED PERCEPTION MEMORY (EPM)
We describe the memory component of system which builds environment representation from ego-centric observations. We define the setting under which the memory operates and introduce the EPM to build and maintain dynamic representation of the environment.
3.1 PROBLEM SETTING
We aim to detect and track objects in the scene as the agent interacts with the environment. Formally, let M denote the environment representation at time L. We learn update function Í such that M S(M o², a²), where a' are observations and actions. Dynamic objects are unknown and can be moved the agent executes the task. Below, define the environment state, observations, and actions.
Environment State. We represent the en- vironment as a list of entities correspond- ing to the furniture and objects. Entity index is associated with a unique id,, a
M'
Agent Obs.
Updates
Environment
M 2-1
Cocrdinates
Add pot on Emity x Ramove: Entiyy4
State
3D coordinate & E R of the approximate centroid of the object or furniture, and a
RGB image
Entiy: luyal collee table
buyal
Ensty:3 buyall
natural language description d, containing information such as open- ppen-vocabulary name, state relationship with other entities.
Erttyy Doyal toy on E.2
Agent Action = Navigote[Baiity]]
Observations. At every timestep t, the agent given egocentric observations o² from the environment: RGBD images, the camera pose, and intrinsic parameters.
VLM
Figure 2: Overview of the EPM.
The agent takes high-level actions in the environment. Action is tuple (u, id., id...) where u is a verb (e.g., Navigate, Pick, Place) and v.... Un index into the entities of environment state M. We use the same action space Chang et al. (2025) (Details in Appendix A.2 and A.4).
Actions.
3.2 TRACKING THE ENVIRONMENT STATE WITH THE EPM
Instead of generating a new environment state M the EPM outputs of discrete operations that are combined with M and the agent's 's egocentric observations to obtain M We consider four operation types (we first list the format of the operation, then describe the function):
Add (<coords>) :descriptionn
Adds a new entity to M with the natural language description d <descriptions The entity coordinates are derived by unprojecting the pixel coordinates <coords> into the world frame.
Update k (<coords>) : :<descripti ippiionn Updates the entity with index j in M such that id, k modifying its coordinates and description.
4

---

Published as a conference paper at ICLR 2026
Remove k: Removes the entity at index from M such that id, k.
No updates: Maintains the same memory state M M
The Add operation appends previously-unsee unnoeen objects to the environment state. Update tracks changes of given object (for instance, if a cabinet was opened or an object was grabbed), and enables correction of perception errors (e.g., correcting the category of a mislabeled object once is seen from closer distance). Remove removes entities that have disappeared from the scene or were previously misdetected. If there is no new information, the EPM outputs No updates. We train a VLM to predict EPM updates. Given the RGB observation, textual environment state, robot action, and agent's pose, the model predicts a set of the above operations that update the memory.
Training. We generate dataset of agent interactions PARTNR (Chang et al., 2025), simulation benchmark requiring agents to explore and move objects in indoor environments. We initialize M with the static environment layout and furniture (e.g., cabinets and large tables). We obtain this information directly from the simulator. Alternatively, we could leverage recent powerful techniques obtain geometric layouts using structure- cuneturr-frommmotion techniques (Wang al., 2025; Tang et al., 2025), and furniture by systems that lift 3D reconstructions to static object instances, e.g. EgoLifter (Gu al., 2024b). We initialize EPM with M/ and use heuristics to derive a sequence of operations that update it over time using privileged information. This produces dataset of observations, actions, environment states, and update operations. Using this as supervision, we finetune LLaVa -arraa-nneiiiinnn-- 7b base model (Li et al., 2024) with LoRA (Hu et al., 2022).
4 PLANNING WITH AN EMBODIED MEMORY
We develop a planner that leverages EPM memory to perform mobile manipulation tasks specified in natural language (e.g., tasks in Figure 1). We present the agent, its use of EPM, and two methods for training the planner to achieve increased task performance and perception robustness.
4.1 ARCHITECTURE
We adopt two-layer control architecture following Chang et al. (2025) as illustrated in Appendix A.4, The high-level planner is given a task description т, obtains environment state M from EPM (Sec. 3.1), and selects an action which are translated into low-level controls via skill policy. We use privileged skills to focus our study on perception and planning and note that our framework is agnostic to skill implementation. Skills execute until failure or termination, at which time the planner is called again. For a fair comparison, we adopt the same skills for the baselines.
parameterized by entities from M',
High-Level Planner. Following Chang et al. (2025), we use ReAct (Yao et al., 2023) to enable an LLM to take actions in the environment. The prompt at time t concatenated text representing objects in M', where each object consists a UID and a description from EPM (Sec. 3.2). Auto- regressive action predictions format (Grattafiori et al., 2024). To avoid excessive context lengths, at each step, only objects updated by EPM are added the context. See Appendix A.4 for details and prompt examples.
interleaved with world representations, resembling an instruct L.M
4.2 TRAINING THE HIGH-LEVEL PLANNER
While the prompting format described above enables non-trival zero-shot performance from pre- trained LLMs (see Table 3), the domain of embodied planning is underrepresented in typical LLM training data. The primary challenge sourcing high-quality data for embodied planning further improve performance. We procure this data in two ways: first, by transforming human demonstrations teleoperated in simulation into robot- coottcoomatiibee plans, and second, using novel online reinforcement learning method to improve the planner with experience in the environment.
Training With Human Demonstrations. We use data derived from human demonstrations to train planning models to solve PARTNR tasks (Chang et al., 2025). The factorization of perception and action (Liang et al., 2023) poses the following challenge: how can we learn to plan from demonstrations when different systems of perception induce different optimal actions? For example,
Note that these heuristics only used to generate training data, inference time.
5

---

Published as a conference paper at ICLR 2026
Fanctior:;
(demonstration):
interExploration. Exploration is required the object was not observed at the end of the previous action.
env LnitEnvizommenti
perception init ()
tzace
1. hnier exploration largets:
2. Sample Exploration
3i Inject Objects into
for Frane in demonet taa iat
whish fumitrrr??
Actions: Now many?
the EPM
observation cav, ctrpptttptrr.
peccaptiionsaeee lobservation) if frame.action in |Pick| PI 10pen Close): infer itrace, perception, francl infersaavggation (Leabe, [zane) trace, agpend (trane)
candidates
pa paccepti add earget x objects
task, 5ccne
N
optimize: Remove interactions that progress the task.
optimize (trace)
appeed [Dor returr t:5ce
Pick (epoon)
Place (apoon, DiningTablel
Pl.i:: (epoon)
Place (Spoen, DaningTabiee
Figure 3: Planning Traces from Human Demonstration. We replay human demonstrations in simulation while rolling out a perception system in-the-loop. To adapt the trace to the perception system and teach the agent exploration, we infer a sequence of exploration actions. Finally, we mitigate sub-optimal human demonstrations by removing interactions that fail to make task progress.
an agent with "pixel-perfect" perception can simply spin in place to identify many objects in a room, whereas a nearsighted agent must move about the space to achieve similar understanding (Gibson, 1979). We address this problem by replaying telcoperations in simulation while running perception in-the-loop. This enables prrceptiionsspciffie planning traces without needing to collect perception- specific demonstrations, We fine-tune LLMs on these traces using LoRA (Hu et al., 2022).
Our approach to deriving training data for high-level planning is illustrated in Fig. 3. Each human demonstration consists of a sequence of agent and object poses along with interaction label (Pickl P Place/Open / Clos e/oone.. At each step, we update the environment state and EPM. We derive high-level planning actions as follows. Each interaction label implies planning action. The trace must also reflect object discovery and navigation, so we add Expl fore and Navi i gate actions accordingly. In particular, if the EPM has not yet detected the interaction object by the end of the previous action, we add Exploxs ore actions to find it. To teach exhaustive search behavior, we sample exploration actions that fail to reveal the object. To teach robustness to EPM hallucinations, we sample Nav-Pick actions to hallucinated objects during exploration. Finally, we optimize the resulting trace by removing interaction sequences that fail make progress toward success according to the episode's 's evaluation function. This enables us to mitigate occasional suboptimal behaviors in the human demonstration dataset. Collectively, this process yields a planning trace we use for training. We provide further details in Appendix A.6.
Improving Planning With DDAFT. To improve planning performance with online experience, we finetune our planner using novel formulation of value-function-frre RL for LLMs we call Dynamic Difficult-AWonn FineTuming, or DDAFT. The goal of DDAFT is to improve performance by efficiently searching the planning space for higher-quality planning behavior. While typically, methods focused on improving reasoning (Shao et al., 2024; Ethayarajh et al., 2024) sample instructions uniformly from the training data, DDAFT shapes the sampling space toward difficult instruction improve sample efficiency. This is similar to DART-Math (Tong et al., 2024), however, DART-Math produces static finetuning dataset, while DDAFT is run iteratively, using the current policy to dynamically estimate the difficulty of instructions, leading to superior performance and sample efficiency.
Given a set of tasks available for training T T and task-conditioned initial planning policy To(a|s, T) (e.g.. the one we learned above from human demonstrations), we first roll out on all episodes in E producing an initial dataset of traces (denoted with x)) with task rewards (denoted with r), Do ( ro),... (xn, The process continues by alternating between finetuning planning model on the dataset D. and sampling new traces to form a new dataset. Many different loss functions commonly used in reinforcement learning or language model finetuning are compatible with this formulation (Schulman et al., 2017; Shao et al., 2024; Ethayarajh et al., 2024). For the experiments in this work we fit the RFT objective (Yuan et al., 2023).
Instead of sampling new traces uniformly across the training episodes, as is typically done for LLM reasoning finetuning (Shao et al., 2024; Yuan al., 2023), DDAFT biases sampling so that exploration is focused on episodes for which the dataset currently contains no successful examples. We do this by sampling which episodes to generate new traces from distribution induced by the

---

Published as a conference paper at ICLR 2026
Simulation Results (PARTNR)
Real-World Results (Spot-Indoor)
EPM lours)
EPM (ouns)
DynaMem
GPT-fo
Aidi cred contakle Aide sclssooocccooo on mabie
Add Adkip chac
4-14 op moutte
cadionnn Roor
X
Add bow
AdM
Figure 4: Qualitative Perception Results. Left: Offline evaluation results of the EPM on the PARTNR dataset, with examples of object addition (top) and removal of mislabeled objects (bottom). Right: Results for EPM, along with the proposed baselines in the real world. The orange dots and bounding box highlight the objects localized by the different methods.
softmax over the failure rates of each episode in the dataset so far. The formalism of this approach, along with comparisons against DART-Math and Online-RFT can be found in Appendix A.7.
5 EXPERIMENTS
We measure the ability of our system (1) to perceive and represent dynamic environments and (2) to enable agents to perform long-horizon planning. We first evaluate the EPM in egocentric videos, with annotated actions and camera poses, that depict robot interacting with objects in indoor enviromment. This experiment demonstrates perception capabilities of EPM independent of planning. Next, we present our main result by evaluating EPM within our planning framework on the PARTNR benchmark (Chang et al., 2025). This evaluation demonstrates the joint ability of EPM and our planning methods perform long-horizon natural language tasks in large indoor environments.
5.1 PERCEPTION EVALUATION
We first evaluate the EPM independently of planning: given agent taking actions in an environment, we measure the ability to track the environment over time. We consider two datasets:
PARTNR (Chang et al., 2025): evaluate the performance of our memory representation in large indoor environments, we generate single-agent demonstrations from the PARTNR validation set and obtain the environment state every agent steps, as described in See 3.2. This results in 1000 demonstrations in 12 different scenes with different object configurations in each episode.
Spot-Indoor: To measure whether our representation transfers to real-world scenarios, we collect dataset of 30 interaction sequences (15 human, 15 robot) recorded 1155m² apartment using Spot robot. While large-scale datasets like EPIC Kitchens (Damen et al., 2020) and EPIC Fields (Tschernezki et al., 2023) provide dynamic scenes and global poses, they lack the dense depth data required for our approach. Our dataset contains 1,0 frames where, in each sequence, target and distractor objects are relocated between various receptacles. In half of the sequences, humans make changes the environment well. We label the dataset by manually annotating the relevant objects and distractors, listing the first frame where a given object seen and the frames at which an object moves.
Metrics. To evaluate the spatial memory, we construct at each timestep & an environment graph G given the environment state M Nodes correspond to entities in M (objects or furniture) and edges denote relations. We define a partially observable graph G, using groundtruth, representing the environment that can be observed given the agent's observations up to time f. We align G, and G with bipartite matching based node distances (matching entities if the name matches and distance is <0.5) and report Precision, Recall and Fl scores for nodes and edges averaged across time.
Baselines. We compare our approach with the following external memory baselines:
7

---

Published as a conference paper at ICLR 2026
Node Metries
Edge Metrics
Method
Dataset
Precision
Recall
Precision
Recall
F1
GT
PARTNR
0.86.0.01
0.54g0.02
0.60
0.83
0.54
0.63+0.02
GPT-4o
PARTNR
0.00
0.05
4.00
2.00
4.00
4.000
DynaMem
PARTNR
0.04
0.10
0.05
EPM
PARTNR
0.36
0.42
0.3410.01
0.66 0.662
0.38
0.46
GPT-4o
Spot-Indoor
0.00
0.65+.0.04
0.02+20.00
0.00
0.00
0.00
Dynamem
Spot-Indoor
0.424
0.70
0..0
EPM
Spot-Indoon
0.30
0.33
0.24
0. 0.28
0. 0.12+0.03
0. 0.16
Table 2: Perception-only Results. Perception evaluation outside the planning loop.
To demonstrate the challenges of the PARTNR dataset, we design privileged version of EPM, where objects are added based on a groundtruth sensor if they are within a distance range from the agent and occupy a minimal area of the agent's s observation.
GT:
GPT-4o: We design the memory as in EPM, but obtain update operations via GPT-4o.
DynaMem: We choose DynaMem (Liu et al., 2024a) as our main baseline it is open-vocabulary, shown to work with LLM planners in the real-world, and handles dynamic scenes. DynaMem stores aggregated object embeddings in global voxel grid, along with RGB-D and pose data. For a query. it retrieves the closest voxel via feature-matching (Zhai et al., 2023) and then uses an object-detector (Minderer et al., 2023) the associated frame to locate the object. If found, the bounding box and depth provide the 3D position. We query each frame using all object categories in our dataset. As object relationships require exhaustive pairwise queries, we omit edge metrics.
5.1.1 PERCEPTION-ONEE RESULTS
In this section, we report perception results in offline setting (i.e., outside the context of planning). EPM outperforms baselines significantly in simulation (Table top). However, the performance is far from saturation (F1-node 0.34), due the challenging nature of the dataset, where objects are often partially visible or seen from far away (see GT performance). Our approach can localize and add objects to the memory, along with their relationships, and remove incorrectly-added objects (Fig. left). To evaluate real-world generalization, we report the results on the Spot-Indoor Indoor dataset (Table 2 bottom). Note that our memory operations are learned using only simulation data. Generally, the model is able to localize objects and relationships in the real world, but often mis- misecategssizes objects and fails to associate multiple instances of the same object (Fig. 4 right). More analysis can be found in Appendix A.3. DynaMem obtains the best performance, but requires querying an object detector with every object from a vocabulary, which is unfeasible with large vocabularies. As we show later in our planning results, these limitations lead to significant drops in planning performance and efficiency compared to our model. Also, note that DynaMem cannot infer relationships (hence, no edge metric reported). GPT-4o tends to hallucinate objects and labels, resulting in much lower node precision.
5.2 PLANNING EVALUATION
We evaluate the planning framework with EPM memory against benchmark of household tasks. We use this online evaluation to draw conclusions about the utility of EPM as a dynamic memory for task planning and the efficacy of our planner to successfully perform tasks given this system of perception. We evaluate against single-agent version of the PARTNR benchmark (Chang et al., 2025), and report results against the held-out validation split of 1,000 episodes covering 12 unique scenes from the Habitat Synthetic Scenes dataset (HSSD (Khanna et al., 2024)).
Metrics. Following PARTNR evaluation protocols, report metrics across axes of task completion and efficiency. Success Rate: The percentage of episodes completed successfully, Percent Complete: the average progress through each task, even if not fully completed, Simulation Steps: the average number of steps taken in the simulator across all episodes, Planning Cycles: the average number of ligh-level actions, and Extraneous Actions: the ratio of unnecessary rearrangements over total rearrangements. The last two are not relevant to the DynaMem baseline as it uses open-loop planning.
Baselines. We compare against the following baselines:
8

---

Published as a conference paper at ICLR 2026
Success Rate
Percent
Simulation
Planning Cycles +
Extraneous Actions +
Row
Method
Perception
Complete 1
Steps 4
DynaMem (Liu et al., 2024a)
GT
0.17 001
0.33 not
7611 130
PP (Chang et al., 2025)
GT
0.51 000
0.67 0.0
1570 #:40
17.6 04
0.19 + 0.01
PP+DDAFT (ours)
GT
0.66 i 0.02
0.78 a: 0.01
2160 60
22.3 0.5
0.19 1: 0.01
HD (ours)
GT
0.63 # 0.02
0.74 = 0.0
2110 s0
21.6 0.
0.17 + 0.01
IID+DAAFT (ours)
GT
0.68 0.002
0.80 0.00
2550
22.7 1.5
0.15 0.01
DynaMem (Liu et al., 2024a)
DynaMem
0.03 000
0.11 = 0.01
5090 120
PP (Chang et al., 2025)
Learned
0.46 + 0.02
= 0.01
1850 50
19.7 = 0.4
0.28 + 0.01
PP+DDAFT (ours)
Learned
0.58 0.0
0.74 = 0.01
2200
23.2 0.5
0.27 = 0.01
HD (ours)
Learned
0.55 0.2
0.69 0:01
3040
30,2
0.17 0.00
10
HD+DDAFT (ours)
Learned
0.58 0.02
0.74 0.01
2250
70
23.6
0.23 0.01
Table 3: Planning Results, Results on the PARTNR single-agent benchmark. We pair planners with various forms of memory in the groundtruth (top) and learned (bottom) perception settings.
The perception system described in Sec. 5.1 is combined with a grid-based navigation system that linearly combines voxel-last-seen timestamps and voxel-to-query alignment scores to identify exploration frontiers. The system explores until input query is found then picks or places it. Based on proposed system in Liu al. (2024b), we add few-shot open-loop LLM planner to decompose tasks into pick-and-place objectives. We tested several ablations of DynaMem specifically testing performance if we used: (a) GT object segmentation and categories instead of object-detector, (b) room room-labels along-witg with object categories (referred to as Room in appendix), and (c) only voxel-last-seen timestamp (TimeOnly) for exploration instead of linear combination. We report the best performing variant in Table 3. See Appendix A.5 for details.
DynaMem:
PP, Preetrreet (Chang ct al., 2025): LLM planning with EPM using the prompting strat- egy deseribed in Section 4.1 using 1.1 amaa 300-3----Innttt
with no additional finetuning. HD: Planning with EPM with LLama 1-88------- Instruct fine-tuned on traces derived from
Human Demonstrations as described in Section 4.2. For training, we use demonstrations from the 100,000 tasks in the PARTNR Train split (Chang et al., 2025).
PP+DDAFT: Starting with the PP model as the base model,
5 iterations of DDAFT
2,000 episode subset of the PARTNR training data, sampling 2,000 episodes per iteration.
HD+DDAFT: Planning with EPM with a model fine-tuned on demonstration and RL data. Using HD as the base model, we perform policy improvement following PP+DDAFT.
5.2.1 PLANNING RESULTS
We report PARTNR planning results in Table 3 with groundtruth (top) and learned (bottom) perception and highlight key findings below.
HD plans effectively from EPM representations. Our HD method outperforms PP in the GT perception setting by 0.12 success (row 4 vs 2) despite a smaller model size (8b vs 70b parameters). Following DDAFT, this trend continues (0.68 0.66 SR; row 5 3). This shows that deriving traces from demonstrations is an effective approach to learn embodied task planning.
HD adapts to EPM failures. With learned perception, HD outperforms PP (row 8 vs and outperforms PP with perfect perception (row 8 vs 2). This shows that demonstrations teach effective planning without the need for collecting perception-speeiff planning traces.
DDAFT enables policy improvement over both PP and HD policies. With GT perception, DDAFT leads to success improvements of 0.15 for PP and 0.05 for HD (comparing row 3 to row 2, and row 5 to row 4, respectively). This extends to learned perception, where we observe improvement of 0.12 and 0.03 success for PP and HD, respectively (row vs 7, row 10 vs 9). This showcases the generality of DDAFT to improve text-based planning policies that leverage the EPM.
Our methods with EPM in the loop outperform strong baselines in task planning. Our methods (rows 8- 80) outperform strong baselines (rows 6&7) by a large margin (55% and 12% improvement in success rate over DynaMem and PP, respectively). To ensure that DynaMem's lower performance is not attributable to simulation visuals, we also provide DynaMem with groundtruth perception (row 1). Nevertheless, its performance remains inferior to ours. In addition to exploration challenges, tasks may require specific instance multiple instances of objects ('all forks') and furniture ('table in office") ) which are not handled well by the eetrieval-then-ettee noted that method is 3.5x faster than DynaMem. PP shows lower sim steps and planning cycles
based algorithms. It should be

---

Published as a conference paper at ICLR 2026
than our methods. That is primarily due to solving shorter horizon tasks. See Appendix A.8-A.10 for more results and a further discussion.
5.2.2 REAL WORLD EVALUATION
To evaluate the effectiveness of our method in real-world scenarios, we test the PP+DDAFT planner together with the learned EPM in an indoor environment, using the Spot robot. We use our planning and memory system to predict high-level actions given a language instruction and adapt the work from Yokoyama et al. (2023) to parse the high-level instructions into motor skills. We test the system in 20 different scenarios, which require the robot to move objects to target locations in presence of clutter. The robot succeeds in 55% of the tasks, with failures coming from both planning and skill execution errors. In 70% of the tasks, the EPM succeeds at predicting the correct plan. More details can be found in the Appendix A.
CONCLUSION AND LIMITATIONS
We present EPM, novel learnable memory for embodied planning models. We show that the explicit world representation of EPM allows for seamless integration with LLM-based planners. Further, we show that signal for training planning models on this representation can be extracted from general human demonstrations, and novel DDAFT method improves planning performance in all settings. Our proposed planning approach outperforms baselines even when they are provided with ground-truth perception.
Our training is limited to simulation data due to the complex nature of collecting a large-scale real-world dataset for dynamic scenes with detailed planning actions. We leave real-world training for future work. We also note that purely text-based representations could inhibit planning performance in certain settings. Our EPM formulation general enough incorporate visual information as continuous features, provided the training data includes such modalities. For example, if a task requires reasoning about object properties not represented text, the planner has no way to solve the task. In those scenarios, hybrid system could pair the EPM with specialized visuomotor or VLA modules that operate directly on the current observation and do require persistent memory. We leave this exploration for future work.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang- Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jomell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can and not as say: Grounding language in robotic affordances. CoRL, 2022.
Alper Aydemir, Andrzej Pronobis, Moritz Göbelbecker, and Patric Jensfelt. Active visual object search in unknown environments using uncertain semanties. IEEE Transactions on Robotics, 2013.
Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. GrOOt nl: An open foundation model for generalist humanoid robots. arXiv, 2025. 3
Ben Bolte, Austin Wang. Jimmy Yang, Mustafa Mukadam, Mrinal Kalakrishnan, and Chris Paxton. Usa-net: Unified semantic and affordance representations for robot memory. In IROS, 2023. 4
Matthew Chang, Theophile Gervet, Mukul Khanna, Sriram Yenamandra, Dhruv Shah, So Yeon Min, Kavit Shah, Chris Paxton, Saurabh Gupta, Dhruv Batra, Roozbeh Mottaghi, Jitendra Malik, and Devendra Singh Chaplot. Goat: Go any thing. In RSS, 2024. 2,
Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, et al. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. ICLR, 2025. 2,44, 5, 7, 8, 9, 18, 19, 21, 24, 28, 32
Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S Ryoo, Austin Stone, and Daniel Kappler. opeabvocab queryable scene representations for real world planning. In ICRA, 2023.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. The epic- ppichkit dataset: Collection, challenges and baselines. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. In ICML, 2023.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. In ICML, 2024. 6
Yingchun Fan, Qichi Zhang, Yuliang Tang, Shaofen Liu, and Hong Han. Blitz-slam: A semantic
slam in dynamic environments. Pattern Recognition, 2022. 4, 15
James J Gibson. The Ecological Approach to Visual Perception. Houghton, Mifflin and Company, 1979. 6
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv, 2024. 5, 22
Qiao Gu. Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen. Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In ICRA, 2024a. 1, 1,3,4 4
Qiao Gu, Zhaoyang Lv, Duncan Frost, Simon Green, Julian Straub, and Chris Sweeney. Egolifter: Open-world 3d segmentation for egocentric perception. In ECCV, 2024b. 5
11

---

Published as a conference paper at ICLR 2026
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In CVPR, 2017.
Huy Ha and Shuran Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models. CoRL, 2022. 3
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022. 5, 6, 15, 24
Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In ICRA, 2023a.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In CoRL, 2023b.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and brian ichter. Inner monologue: Embodied reasoning through planning with language models. In CoRI., 2023c.
Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. In CoRL., 2024.
Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer. Soroush Saryazdi, Nikhil Keetha, et al. Conceptfusion: Open-set multimodal 3d mapping. In RSS, 2023. 1,
Mazeyu Ji, Ri-Zhao Qiu, Xueyan Zou, and Xiaolong Wang. Graspsplats: Efficient manipulation with 3d feature splatting. In CoRL, 2024. 3
Haochen Jiang, Yueming Xu, Kejic Li, Jianfeng Feng, and Li Zhang. Rodyn-slam: Robust dynamic dense rgb-d slam with neural radiance fields. IEEE Robotics and Automation Letters, 2024. 4, 15
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In ICCV, 2023.
Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X Chang, and Manolis Savva. Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene scale and realism tradeoffs for objectgoal navigation. In CVPR, 2024.
32
Sebastian Koch, Narunas Vaskevicius, Mirco Colosi, Pedro Hermosilla, and Timo Ropinski. Open3dsg: Open- vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships. In CVPR, 2024. 4
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv, 2024. 5, 33
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. In ICLR, 2022. 3
Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Raymond Yu, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, et al. Hamster: Hierarchical action models for open-world robot manipulation. In ICLR, 2025.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman. Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In ICRA, 2023.
Peiqi Liu, Zhanqiu Guo. Mohit Warke, Soumith Chintala, Chris Paxton, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Dynamem: Online dynamic seatio-semantic memory for open world mobile manipulation. arXiv, 2024a. 2, 8, 9, 23
12

---

Published as a conference paper at ICLR 2026
Peiqi Liu, Yaswanth Orru, Jay Vakil, Chris Paxton, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Ok-robot: What really matters integrating open-knowledge models for robotics. In RSS, 2024b. 9,
Dominic Maggio, Yun Chang, Nathan Hughes, Matthew Trang, Dan Griffith, Carlyn Dougherty, Eric Cristofalo, Lukas Schmid, and Luca Carlone. Clio: Real-time task-driven open-set 3d scene graphs. IEEE Robotics and Automation Letters, 2024. 3
Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open object detection. In NeurlPS, 2023. 8
Soroush Nasiriany. Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong. Tingnan Zhang, Tsang- Wei Edward Lee, Kuang- Huci Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Finn, Sergey Levine, and Brian lchter. Pivot: Iterative visual prompting elicits actionable knowledge for vims. In ICML, 2024. 3
Ri-Zhao Qiu, Yafei Hu, Yuchen Song, Ge Yang, Yang Fu, Jianglong Ye, Jiteng Mu, Ruihan Yang, Nikolay Atanasov, Sebastian Scherer, al. Learning generalizable feature fields for mobile manipulation. arXiv, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 3
Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning. In CoRL, 2023.
Hello Robot. Stretch ai: Dynamem code, 2025. URL https: http://:///..tctcoooo//o/ooooooo stretchhai/bloke /tto///aain//////i// momain/ddcss. docs/ dynamem. md. 21
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to tools. In NeurlPS, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv, 2024. 2, 6
Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open- Ondedended instruction following with hierarchical vision-language-action models. arXiv, 2025. 3
Ola Shorinwa, Johnathan Tucker, Aliyah Smith, Aiden Swann, Timothy Chen, Roya Firoozi, Monroe Kennedy III, and Mac Schwager. Splat-mover: Multi-stage, open-vocabulary robotic manipulation via editable gaussian splatting. In CoRL, 2024. 3
Zhenggang Tang, Yuchen Fan, Dilin Wang, Hongyu Xu, Rakesh Ranjan, Alexander Schwing, and Zhicheng Yan. Mvdust3r+: Single-stage scene reconstruction from sparse views in 2 seconds, In CVPR, 2025.
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficulty-awe aware rejection tuning for mathematical problem-solving. In NewrIPS, 2024. 2, 6, 26
Vadim Tschernezki, Ahmad Darkhalil, Zhifan Zhu, David Fouhey, Iro Larina, Diane Larlus, Dima Damen, and Andrea Vedaldi. EPIC Fields: Marrying 3D Geometry and Video Understanding. In NeurIPS, 2023.7
13

---

Published as a conference paper at ICLR 2026
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In CVPR, 2025. 5
Shu Wang, Muzhi Han, Ziyuan Jiao, Zeyu Zhang, Ying Nian Wu, Song-Chun Zhu, and Hangxin Liu. Llm Large language model-based task and motion planning with motion failure reasoning. In IROS, 2024. 3
Yixuan Wang, Mingtong Zhang, Zhuoran Li, Katherine Rose Fei, and Yunzhu Li. D3 fields: Dynamic 3d descriptor fields for zero-shot generalizable robotic manipulation. In ICRA 2024 Workshop on 3D Visual Representations for Robot Manipulation, 2023.
Jiajun Wu, Li Fei-
Wenhua Wu, Guangming Wang. Ting Deng, Sebastian Aegidius, Stuart Shanks, Valerio Modugno, Dimitrios Kanoulas, and Hesheng Wang. Dvn-slam: Dynamic visual neural slam based local-global encoding. In ICRA, 2024. 4, 15
Boyang Xue, Qi Zhu, Hongru Wang, Rui Wang, Sheng Wang, Hongling Xu, Fei Mi, Yasheng Wang, Lifeng Shang, Qun Liu, al. Dast: Difficulty-awan self-training on large language models. arXiv, 2025. 27
Zhijie Yan, Shufei Li, Zuoxu Wang. Lixiu Wu, Han Wang, Jun Zhu, Lijiang Chen, and Jihong Liu. Dynamic opea-vocabulary scene graphs for long-term language-guidee guided mobile manipulation. IEEE Robotics and Automation Letters, 2025. 4
Yuncong Yang, Han Yang. Jiachen Zhou, Peihao Chen, Hongxin Zhang, Yilun Du, and Chuang Gan. 3d-mem: 3d scene memory for embodied exploration and reasoning, In CVPR, 2025. 1, 2,
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In ICIR,, 2023. 5
Naoki Yokoyama, Alexander William Clegg, Joanne Truong, Eric Undersander, Tsung- Yen Yang, Sergio Amaud, Sehoon Ha, Dhruv Batra, and Akshara Rai. Adaptive skill coordination for robotic mobile manipulation. arXiv, 2023. 10, 31
Chao Yu, Zuxin Liu, Xin-Jun Liu, Fugui Xie, Yi Yang, Qi Wei, and Qiao Fei. Ds-slam: A semantic visual slam towards dynamic environments. In IROS, 2018. 15
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou Scaling relationship on learning mathematical reasoning with large language models. arXiv, 2023. 2, 6, 27
Andy Zeng, Maria Attarian, Krzysztof Marcin Choromanski, Adrian Wong Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, et al. Socratic models: Composing zero-shot multimodal reasoning with language. In ICLR, 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 8
14


Reviews:
[{"review_id": "2jkpMx66jk", "reviewer": "Reviewer_ACax", "summary": "This paper introduces Embodied Perception Memory (EPM), a unified vision–language model based memory system for embodied planning that represents the environment as a textual, object-centric world model updated from egocentric observations. EPM dynamically tracks objects and changes in long-horizon and enables seamless integration with LLM-based planners. To train planners to effectively leverage this representation, the authors propose two complementary methods: (i) imitation learning from human demonstration trajectories and (ii) Dynamic Difficulty-Aware Fine-Tuning (DDAFT), a value-function-free reinforcement learning algorithm that prioritizes training on harder tasks by sampling episodes based on failure rates. They evaluate EPM both as a perception module in isolation and as part of full embodied planning tasks. In particular, EPM outperforms existing memory architectures such as DynaMem and also surpasses LLM-based planners trained solely on the PARTNR dataset.", "questions": "1. How do you plan to evaluate the complete EPM planning pipeline in real-world robotic settings, and what challenges do you anticipate in transferring beyond simulation?\n\n2. Have you considered fine-tuning EPM on large-scale real-world embodied datasets (e.g., Ego4D, RT-1/RT-2, BEHAVIOR-1K) to improve generalization for real objects not present in simulation?\n\n3. How does EPM handle conflicting observations, object occlusions, or reappearances? Is there an object permanence or confidence mechanism to avoid stale or incorrect memory states?\n\n4. Can you provide memory drift or long-horizon error analyses? How does the textual memory behave when small memory errors compound over long sequences?\n\n5. What motivates DDAFT over more established RL fine-tuning methods, and have you compared against RFT, PPO-based finetuning, or GRPO?\n\n6. Could you report computational metrics such as update latency, memory insertion cost, and query time, to substantiate the claimed scalability benefits?\n\n7. Why were recent concept-centric memory systems (e.g., ConceptFusion, ConceptGraph) not included as baselines, and how do you expect EPM to compare conceptually and empirically?\n\n8. Do you anticipate difficulty scaling EPM to domains requiring fine-grained perceptual grounding (e.g., grasp point reasoning, deformable objects, tool use), and how might hybrid visual-textual memory help?\n\n9. Have you explored mechanisms to reset, compress, or prune memory to avoid uncontrolled growth or hallucinated persistence in long-horizon tasks?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing is generally clear, with strong motivation and context provided for the problem setting.\n\n2. The paper introduces a novel memory mechanism for embodied planning which is a conceptual advance over prior approaches that rely on separate perception modules, graph-based representations, or retrieval interfaces. The integration of a single VLM that simultaneously maintains, updates and exposes a textual world representation represents a meaningful rethinking of memory design for embodied agents.\n\n3. The proposed approach tackles an increasingly important problem in embodied A, long-horizon planning in dynamic environments. The ability to maintain and evolve a textual world model from egocentric observations, while integrating seamlessly with language-based planners, has direct implications for future real-world embodied AI systems.\n\n4. By unifying perception and memory within a single model, the approach reduces reliance on heavy symbolic querying or multi-module pipelines, which can be brittle and computationally expensive. This end-to-end design offers a promising direction toward scalable, real-time embodied planning systems.\n\n5. The paper evaluates both perception and planning. The perception system is tested on both simulaton and real-world egocentric data, demonstrating promising transfer beyond simulation. Planning performance is assessed in challenging simulated environments, where EPM outperforms prior memory architectures (e.g., DynaMem) and LLM planners trained solely on PARTNR.", "weaknesses": "1. Real-world perception results do not show clear advantage over DynaMem: While the perception module is evaluated on real-world egocentric data, the reported results do not show a clear performance improvement over DynaMem in this setting. The authors attribute this to out-of-distribution objects that appear in real environments but are absent in simulation training. However, this highlights a practical limitation of the current approach: the system’s ability to generalize in open-world settings remains uncertain. It would be valuable to explore whether fine-tuning EPM on diverse real-world embodied datasets could reduce this gap. For example, large-scale egocentric datasets such as Ego4D, BEHAVIOR-1K real-world captures, or RT-1/RT-2 robot interaction datasets contain a wide variety of real objects and household interactions that could support domain adaptation. \n\n2. No real-world validation of the planning pipeline: While the perception module is evaluated on real-world egocentric data, the planning results are restricted to simulation. Consequently, it remains uncertain how well the complete EPM-based planning pipeline would transfer to physical robots, where challenges such as sensor noise, actuation inaccuracies, and occlusions are more pronounced. Even small-scale real-robot experiments, or analysis demonstrating robustness to real-world observation and control noise, would significantly strengthen the claim that the method scales beyond simulation.\n\n3. Incomplete comparison to recent memory systems and lack of computational analysis: While the paper compares against DynaMem, the evaluation omits other recent memory representations that also maintain object-centric world models, such as ConceptFusion, ConceptGraph etc. which would provide a more complete picture of performance relative to the state of the art in embodied memory systems. Additionally, although success rates and perception F1 scores are reported, the paper does not quantify computational efficiency or update overhead, which is central to the claimed scalability benefits. Reporting metrics such as time per memory update, insertion cost for new objects, query latency, and overall inference load per planning step would clarify the practical efficiency gains of EPM over modular graph-based or retrieval-based memory architectures.\n\n4. Limitations of a text-only memory representation: A purely textual memory, while clean and LLM-friendly, may fail to capture fine-grained perceptual details needed for precise manipulation. Object shape, texture, grasp points, and continuous spatial relationships are difficult to encode in text, and tasks requiring such information may exceed the capabilities of the current representation. Although the authors acknowledge hybrid memories as future work, the paper does not explore mechanisms to integrate visual or geometric cues, limiting applicability to more physically detailed settings.\n\n5. DDAFT lacks formal justification and comparison to standard RL fine-tuning techniques: While DDAFT intuitively prioritizes difficult trajectories, the method is introduced without strong grounding, and its advantages over established RL-from-feedback or policy fine-tuning approaches remain unclear. In particular, comparisons to more standard baselines such as RFT-style updates, GRPO, or PPO-based fine-tuning would help determine whether DDAFT offers a principled improvement or functions primarily as a heuristic curriculum strategy. Without these comparisons, it is difficult to isolate the contribution of the proposed RL component.\n\n6. Limited analysis of failure cases and memory degradation over long horizons: The paper reports aggregate success metrics but offers limited qualitative analysis of failure modes. In particular, it remains unclear how EPM behaves when memory errors accumulate over long sequences (e.g., misplaced objects, overwritten states, hallucinated object persistence). Without studying memory drift, error correction, or mechanisms for uncertainty estimation, it is difficult to assess reliability in extended, multi-room or multi-task settings.\n\n7. Ambiguity in update policies and memory maintenance strategy: EPM continuously rewrites textual memory, but the rules governing updates (e.g., when to delete stale information, how to handle conflicting observations, how to resolve object identity uncertainty) are not fully articulated. For instance, if an object is temporarily occluded or repositioned off-camera, the system may incorrectly maintain outdated state. An explicit treatment of object permanence, confidence tracking, or uncertainty-aware updates would strengthen the method’s robustness claims.\n\n8. Task and domain diversity is limited: Although the results on PARTNR are compelling, the generality of EPM to broader embodied domains such as navigation, deformable object manipulation, tool use, or embodied household tasks in other environments like BEHAVIOR-1K or Habitat 2.0 is not established. Without cross-domain evaluation, the system’s scalability across diverse embodied tasks remains open.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nl28oU2iXW", "reviewer": "Reviewer_U9Bz", "summary": "This paper proposes the Embodied Perception Memory (EPM), a textual list of entities (objects and furniture) and their relationships that uses a Vision-Language Model (VLM) to maintain and update. For planning, the authors employ a two-stage training strategy: (1) Imitation learning on planning trajectorirs derived from human demonstrations, which are post-processed to incorporate perception-specific exploration and robustness to memory errors; (2) A reinforcement fine-tuning method named Dynamic Difficulty-Aware Fine-Tuning (DDAFT), which adapts the DART-Math idea to the planning task during online policy improvement. The primary experimental validation on embodied planning is conducted on the PARTNR benchmark.", "questions": "1. How does the system handle multi-object disambiguation? For example, if there are four apples and five milk bottles in the environment, how does EPM distinguish and track each instance in its textual representation?\n2. For the Spot-Indoor dataset, are the annotations for adding/removing objects manually labeled, or are they constructed using the method described in Section 4.2 (i.e., via simulation replay and heuristics)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The EPM provides a unified and interpretable text-based memory that seamlessly integrates with LLM planners, avoiding the need for complex querying mechanisms used in some prior works (e.g., feature matching in point clouds).\n\n- The proposed DDAFT method effectively improves planning performance by dynamically focusing on difficult tasks during training.", "weaknesses": "- The idea of using a unified VLM to convert visual observations into a textual memory for planning has been explored in prior works [1-3].\n- The experiments are conducted only on the PARTNR validation set. It is unclear how the method would perform on the test set, which is important for assessing generalization and fairness in benchmarking.\n- The training data for EPM relies on simulator-privileged information (e.g., ground-truth object states and poses), which limits the scalability and applicability of the method to real-world settings where such information is unavailable.\n\n[1] Context-aware planning and environment-aware memory for instruction following embodied agents, ICCV 2023   \n[2] STMA: A Spatio-Temporal Memory Agent for Long-Horizon Embodied Task Planning, arxiv 2025.02    \n[3] KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systems, ICRA 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mSWY7228OW", "reviewer": "Reviewer_JRZU", "summary": "This paper proposes an embodied planning framework that introduces a *dynamically learnable memory* to assist high-level task planning. The key idea is to maintain a textual memory that records the positions and states of currently observed objects, which is then injected into the input context of a large language model (LLM) planner. The planner conditions on this memory to produce high-level action decisions such as *Navigate*, *Pick*, and *Place*, allowing the system to reason about dynamic environments through language-conditioned planning.", "questions": "1. **Why use an LLM planner instead of a VLM?**\n   Since the memory content is purely a textual description of visual states, a Vision-Language Model could directly access equivalent spatial information from images. What is the rationale for choosing a text-only LLM over a multimodal model that already encodes spatial and visual grounding?\n2. **Can you compare LLM w/ memory vs. VLM w/o memory vs. VLM w/ memory for the task planning?**\n   Such a comparison would more convincingly isolate the contribution of the memory mechanism and clarify whether the observed gains come from the memory design \n3. **Can you include additional benchmarks?**\n   Evaluating on environments like **ALFWorld** or **EmbodiedBench** would strengthen the claim that the proposed approach generalizes across different embodied planning settings.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Incorporating a memory mechanism into embodied planning is conceptually valuable. It aligns with the broader goal of enhancing situational awareness and long-horizon reasoning in embodied agents.\n- The proposed pipeline is relatively complete, combining human demonstration data and DDAFT to improve the planning capability of the LLM. \n- The authors provide detailed implementation descriptions, which improve the reproducibility of the proposed approach.", "weaknesses": "The overall method is straightforward and lacks sufficient evidence to justify the necessity of the proposed memory mechanism. Essentially, the “memory” represents the textual description of observed object positions, which is appended as additional prompt context for the LLM planner to make decisions. This design raises questions about its true effectiveness: a Vision-Language Model (VLM) could directly perceive and infer object locations from images without requiring a separate process to extract and feed this information in textual form.\n\nIn the experiments, the authors compare models of different scales (7B and 70B) and their fine-tuned variants. The results show that supervised fine-tuning and DDAFT (a DPO-like reward fine-tuning method) indeed enhance model performance, but such improvement mainly reflects the benefit of additional data and optimization rather than the specific contribution of the memory mechanism. From the presented evidence, it remains unclear why this kind of dynamic memory is indispensable for embodied task planning.\n\nMoreover, the evaluation is limited to the PARTNR benchmark. While this environment is a reasonable testbed, the empirical validation would be more convincing if the authors could extend experiments to other widely used embodied task planning benchmarks such as **ALFWorld** or **EmbodiedBench**. Including results from multiple environments would help demonstrate the generality and robustness of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IjLCUr2eKN", "reviewer": "Reviewer_esNZ", "summary": "This paper presents Embodied Perception Memory (EPM), a learnable memory representation for long-horizon embodied planning in dynamic indoor environments. This representation enables end-to-end scene information operations (addition, removal, update), adopts object-centric encoding (3D coordinates + textual state/context descriptions), and eliminates explicit task-based queries by generating LLM-compatible textual scene representations. In addition, a two-stage tranining recipe is proposed to finetune LLM planners to leverage the EPM: first train the model on human demostration data to improve exploration and task execution, then leverage a novel online RL-based approach, named Dynamic Difficulty-Aware Fine-Tuning (DDAFT), to optimize planning via online sampling of difficult instructions.", "questions": "See weakness above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is clear and logically coherent, ensuring easy readability for readers.\n2. The proposed memory representation is well-suited for tracking object states in dynamic indoor environments, and it exhibits several key advantages:\n (1) Easy maintenance: Unlike complex scene graphs that detail the relative relationships between all objects, EPM adopts a simpler spatial representation—one that focuses only on the relationships between dynamic objects and infrequently movable furniture.\n (2) Lightweight design: Compared with explicit semantic mapping techniques, EPM eliminates the need to perform segmentation, open-vocabulary encoding, and dynamic map updates.\n3. The performance results are robust, particularly when planners are paired with the proposed memory under learned perception settings.", "weaknesses": "1. In Section 4.2, the use of human demonstration data for training is introduced; however, the description of how to acquire human demonstration data based on tasks and memory lacks clarity.\n2. How is the instance association problem in the perception process addressed? It appears that the same instance may be recorded as multiple instances due to perception and localization errors. When applied to real world, has consideration been given to adding an operation type for merging entities?\n3. In the inference phase, how is the information about which specific room each piece of furniture is located in (stored in memory) acquired? In the planning example in Appendix A.4, the descriptive information of the furniture is inconsistent with the content described in the Entity List and Embodied Perception Memory Input in Appendix A.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)