Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilization
Authors: Taeyoon Kwon
Dongwook Choi
Hyojun Kim
Sunghwan Kim
Seungjun Moon
Beong-woo Kwak
Kuan-Hao Huang
Jinyoung Yeo
Venue: ICLR 2026 Poster (2026)
Keywords: Embodied agent
Memory
Agent
LLM
Personalization
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: LLM-powered embodied agents have shown success on conventional object-rearrangement tasks, but providing personalized assistance that leverages user-specific knowledge from past interactions presents new challenges. We investigate these challenges through the lens of agents' memory utilization along two critical dimensions: object semantics (identifying objects based on personal meaning) and user patterns (recalling sequences from behavioral routines). To assess these capabilities, we construct Memento, an end-to-end two-stage evaluation framework comprising single-memory and joint-memory tasks. Our experiments reveal that current agents can recall simple object semantics but struggle to apply sequential user patterns to planning. Through in-depth analysis, we identify two critical bottlenecks: information overload and coordination failures when handling multiple memories. Based on these findings, we explore memory architectural approaches to address these challenges. Given our observation that episodic memory provides both personalized knowledge and in-context learning benefits, we design a hierarchical knowledge graph-based user-profile memory module that separately manages personalized knowledge, achieving substantial improvements on both single and joint-memory tasks.

Published as a conference paper at ICLR 2026
EMBODIED AGENTS MEET PERSONALIZATION: INVES- TIGATING CHALLENGES AND SOLUTIONS THROUGH THE LENS OF MEMORY UTILIZATION
Taeyoon Kwon" Dongwook Choi
Hyojun Kim-
Sunghwan Kim-
Seungjun Moon Beong-woo Kwak!
Kuan-Hao Huang? Jinyoung Yeo!
'Department of Artificial Intelligence, Yonsei University
"Department of Computer Science and Engineering. Texas A&M University
ABSTRACT
LLM-powered embodied agents have shown success on conventional object- rearrangement tasks, but providing personalized assistance that leverages user- specific knowledge from past interactions presents new challenges. We investi- gate these challenges through the lens of agents' memory utilization along two critical dimensions: object semantics (identifying objects based on personal mean- ing) and user patterns (recalling sequences from behavioral routines). To assess these capabilities, we construct MEMENTO, an end-to-end two-stage evaluation framework comprising single-memory and joint-memory tasks. Our experiments reveal that current agents can recall simple object semantics but struggle to ap- ply sequential user patterns to planning. Through in in-depth analysis, we identify two critical bottlenecks: information overload and coordination failures when handling multiple memories. Based on these findings, we explore memory chitectural approaches to address these challenges. Given our observation that episodic memory provides both personalized knowledge and in-context learning benefits, we design a hierarchical knowledge graph-based user-profile memory module that separately manages personalized knowledge, achieving substantial im- provements on both single and joint- memory tasks. Our code and data is available at [|ct:p////gEtabbccco////oooooagp
INTRODUCTION
Embodied agents empowered by large language models (LLMs) have recently demonstrated remark- able success in executing object rearrangement tasks in household environments (Huang et al. 2022b Wu et al. 2023 Song
2023 Chang al. 2024, Li et al. 2025b). ) As the primary objective of
embodied agents is to provide assistance to users while interacting with the physical world, leveraging
LLMs' natural language understanding and planning capabilities leads embodied agents to effectively interpret user instructions to successfully accomplish the task.
But do such tasks truly reflect the challenges in providing practical assistance to the users? As illustrated in Figure [ conventional embodied tasks predominantly focus single-turn interactions with static and simplified instructions that the agents could simply follow without implicit reasoning to
comprehend user intentions (Ahn et al. 2022 Huang et al. 2022a Song al. 2023 Wu et al. 2024). However, for personalized embodied agents, is important to understand personalized knowledge that users assign unique semantics to the physical world (e.g.. favorite cup, breakfast routine) to interpret dynamic instructions. To provide personalized assistance, agents must understand user instructions and conduct task planning by leveraging personalized knowledge retained from previous interactions esspciall episodic memory, which enables the recall of specific events grounded in time and space (Huet et al. 2025). Therefore, understanding embodied agents' memory utilization capabilities is crucial for providing practical assistance to users.
In this work, we investigate the challenges faced by LLM-peeered embodied agents in personalized assistance tasks through the lens of memory utilization. We analyze agents' memory utilization
"quual contribution

---

Published as a conference paper at ICLR 2026
Previous Embodied Tasks
Personalized Assistance Tasks
"Bring the cup and place the right side of the table, then bring... ..."
What the faverite cup?
routine?
My Favrrte Oup
First, plak the cup.
My Breakdayt Routine
Bring my favorite cup and prepare my light breakfast setup
is simply following instructions enough for Rersonalized embodied agent?
Comparison between conventional embodied tasks and personalized assistance tasks. Previous works focus on following simple instructions, while personalized assistance agents must know user-specific knowledge, which require grounding in past interactions.
Figure 1:
capabilities along two critical dimensions based on personalized knowledge: (1) the ability to identify target objects based on personal meaning (object semantics), and (2) the ability to recall sequences of object-location pairs from consistent user behavioral patterns, such as daily routines (user patterns). To evaluate task performance and quantify how memory utilization affects performance on personalized assistance tasks, we construct MEMENTO, an end-to-end two-stage personalized embodied agent evaluation framework comprising both single-memory and joint-memory tasks. Using MEMENTO, we address the following three research questions:
RQ1: Can current LLM-powered embodied agents effectively utilize memory to perform personalized assistance tasks? (Section 4p Initially, we evaluate various LLM-powered embodied agents using MEMENTO assess their memory utilization capabilities. Our findings reveal key limitation: while agents can effectively recall simple object-related memory for object semantics, they struggle to apply sequential information such as user patterns their planning process. Qualitative analysis shows that conflicts between parametric commonsense knowledge and non-parametric personalized knowledge can cause agents neglect memory utilization, indicating that current I.LM-powered embodied agents cannot reliably integrate personalized knowledge for task execution.
RQ2: What are the key factors that negatively affect embodied agents' memory utilization capabilities? (Section 5) To understand the underlying factors that limit memory utilization perfor- mance, we conduct an in-depth analysis of agents' behavior under varying memory conditions. Our investigation reveals two critical bottlenecks: (1) information overload: increasing the number of retrieved memories (top-k) introduces noise that degrades performance, and (2) coordination failures: agents cannot effectively coordinate multiple memories, failing even simple joint-memory tasks combining two object semantic memories. These findings underscore the essential need for improved memory architectures that support agents personalized assistance tasks.
RQ3: How can we design memory architectures to better support personalized assistance tasks? (Section 6) Based on our findings, we explore architectural approaches to improve agents' memory utilization capabilities. Initially, we investigate memory summarization to remove unnecessary infor- mation but find marginal improvements and even degradation in smaller LLMs. This result reveals that episodic memory provides both personalized knowledge and in-context learning benefits, making it an essential memory module for embodied agents. Therefore, we develop hierarchical knowledge braph user-profile memory module that manages personalized information independently. Adapting this memory module shows substantial improvements on both single and joint- memory tasks demonstrating the potential for future research in developing personalized embodied agents.
2 RELATED WORK
LLM- -.wwerd embodied agents. LLMs have significantly advanced embodied agents' reasoning and planning capabilities in recent years. Researchers have explored LLMs for interpreting user goals (Ahn et al.) 2022), high-level task planning (Huang al. 2022a), and integrating LLMs into comprehensive embodied agent frameworks (Huang et al. 2022b Li et al., 2022] Mu et al.
2023: Song et al. 2023 Huang al. 2023b). Other research directions have focused generating
2

---

Published as a conference paper at ICLR 2026
executable code for embodied tasks directly from language instructions (Liang et al. 2022; Wang 2003al Singh et al. (203), while various benchmarks have been developed to evaluate embodied reasoning abilities (Li et al. 2023] Choi et al. 2024 Chang et al. 2024 Li et al. 2025b). Collectively. these studies highlight the promise of LLM --wwereedd agents in bridging language understanding and
physical interaction.
Memory systems for embodied agents. Previous studies on memory systems for embodied agents have primarily focused on semantic memory (e.g., scene graph, semantic map), which store and provide state information about the current environment (Rana et al. 2023; Kim et al. 2023 Gu
al., 2024; Xie et al. [024] Wang et al., 2024c), or on procedural memory (e.g., skill library) that stores action primitives, focusing on how to perform tasks to enhance the efficiency in generating the low-level action code (Wang et al. 2023a Sarch et al. 2023 Zhang et al. 2023). Another important category is episodic memory, which captures specific past interactions and experiences with users. However, prior uses of episodic memory have mostly treated it as passive task buffers (Ahn et al.,
2022
Singh et al. 2023) or histories for in-context (Huang et al. 2022b Liu et al. 2023 Song et al.
2023
Chen et al. 2024). without explicitly evaluating its role in personalized task grounding or
systematic memory utilization.
Personalization for embodied agents.
The importance of personalization in robotics has long
been recognized (Dautenhahn 2004: Lee et al.) 2012; Clabaugh & Matarić 2018), particularly
in the context of hoban-robot interaction where robots adapt their interactive behaviors to align
with individual users. Recent works have focused on reflecting individual user's preferences during
embodied agents' task execution, such as spatial arrangement (Kapelyukh & Johns 2021 Wu et al.
2023), table settings (Puig et al. (202T), personalized object navigations (Dai et al. 2024 Barsellotti
et al. 2024). Recently, Xu et al. (224)) aim to infer user preferences from a few demonstrations
and adapt planning behavior accordingly. However, these approaches primarily address implicit preference adaptation or short-tern reactive behaviors derived from repeated user demonstrations, without leveraging explicitly provided, user-defined knowledge in a structured manner.
3 EVALUATION SETUP
3.1 TASK & FOCUS
Task: personalized object rearrangement. We formulate the personalized object rearrangement task as Partially Observable Markov Decision Process (POMDP) (S, A, R, 0,7), where S denotes the set of environment states, A is the set of actions, T is the transition function, R is the reward function, At timestep f, the agent receives a text observation Wi E 52 describing nearby objects based on the previous action a--1 and generates action a, via LLM policy æ. Given natural language instruction I that may contain personalized references (e.g., "Place my favorite mug on the table"), the agent must derive the goal representation g using both the instruction and personalized knowledge from memory: d(, M) - = (o,, 4)==) where M = h1, hy,.... hn represents the set of episodic memories from previous interactions, and each (o,, 14) pair specifies target object o; and location 7,- The policy generates actions based on the intevvattonaaatiion trajectory T₁
the observation space, O is the observation function, and 7 the discount factor.
ा (1, M, Tt) + as, Te = (W1, a1, W2, .2..... Wt-1, a;-1, we)
The goal is produce action sequence ar such that the resulting state s, satisfies all locactnnocation specifications in g.
Focus: memory utilization. Personalized assistance tasks require agents to integrate multiple capa- bilities including natural language understanding, memory retrieval, planning, and action execution. Among these interconnected abilities, we focus specifically on memory utilization-the agent's 's capacity to effectively recall and apply relevant personalized knowledge from episodic memory M to interpret instructions and guide task planning. This focus is motivated by memory utilization being the primary distinguishing capability from an agent perspective that separates personalized assistance tasks from conventional embodied tasks, making it essential to investigate for potential challenges and solutions for developing personalized embodied agents.
3

---

Published as a conference paper at ICLR 2026
1. Memory Acquisition Stage
MEMENTO (Perooalised Emaodied Agent exstall Framnwsrk
2. Memory Utilization Stage
Epiacde #1
Spisods #2
Single memory Taskk
Seint memory Tast
Ofject Semantics Paster:
Nowa livng reamn rakkee and move bowl...
spplic the bank serring!
Move my Erverite cun to kiigiing room sobli
the lving room tabee... and chan prezare
(Thrt's my faverioe cupt
baok ection + Additional info
Environment
Personalized Instruction
Joint Personaliodd Instruction
Pertonaliaed knovledge
Personal Inowdadge
Conduct object rearaaggmmet task and accumulate episodić memory
Eplsodic Memory
Ephode #1
Episods
21 Utilie episodi memory for assistance
Move cap sotraliing roar
ind team ou appk tect
DesorbaOo enttunn Resuts SMM
Raruht Auscess
thouglt
History
Hittery
prspute un prreicus
tir pocuers fannta
(bwar)
Seauta
Success Rate (SR)
(5R)
memory
capability comparing the Success Rate (25R)
Memory Utilzation Capability (2BR)
Figure 2: Overview of MEMENTO. The framework evaluates memory utilization capability by comparing agent performance on tasks with identical goals but varying instructions on each stage.
3.2 DIMENSIONS OF PERSONALIZED KNOWLEDGE ANALYSIS
We analyze embodied agents' memory utilization capabilities along two dimensions that reflect how personalized knowledge affects goal interpretation (M) - g = (o,, 4) Therefore. we categorize personalized knowledge into two types, each designed to isolate distinct reasoning challenges that agents must resolve using episodic memory:
Object semantics evaluates agents' ability to identify target objects o, based on personal meaning assigned by users, including ownership (my cup). preference (my favorite running gear), personal history (a graduation gift from my grandma), or grouped references (my childhood toy collections). This dimension tests whether agents can recall specific object semanties from prior interactions to resolve ambiguous object references.
User patterns assesses agents' capacity to reconstruct complete goals by leveraging consistent behavioral patterns, including personal routines (my remole work setup) and arrangement prefer- ences (my cozy dinner atmosphere). This dimension evaluates whether agents can apply previously observed patterns across multiple objects and locations infer task requirements.
Each dimension targets different aspects of the grounding function ΦiT, J, M) - g: object seman- ties focuses resolving individual object references, while patterns addresses holistic goal reconstruction from behavioral context. Detailed explanation of the sub-categories of personalized knowledge be found in Appendix C.2
3.3 MEMENTO: PERSONALIZED EMBODIED AGENT EVALUATION FRAMEWORK
Framework design. The major challenge of evaluating embodied agents' memory utilization capability is quantifying the memory effect overall task performance. To address this, as shown in Figure we design an end-to-end two-stage evaluation process that measures memory utilization capabilities through instruction interpretation. Given the base object rearrangement task episode defined as the tuple = (S, J, T,g), the key concept of our evaluation process and goal representation g. while varying the instruction / across the two stages to isolate instruction interpretation capability as the primary factor influencing performance differences.
share the scene
In the memory acquisition stage, agents perform conventional object rearrangement tasks with instructions that contain sufficient personalized knowledge. These episodes are defined as cacg = (S, Jacq. g) where Φ(Tacg) - g. meaning agents directly infer the goal from instructions. During this stage, agents establish a reference performance baseline while accumulating episodic memory haog from their interactions. The memory utilization stage presents agents with identical tasks
4

---

Published as a conference paper at ICLR 2026
but uses underspecified instructions Cutal = Jutst g). These tasks can only be completed when agents recall and apply relevant personalized knowledge, requiring the extended grounding function Φ haog) - g. Through this evaluation process, utilization capability by comparing performance between the two stages.
able to quantify the agent's 's memory
Furthermore, to evaluate different complexity levels, assess both (1) single-memory tasks requir- ing information from one episodic memory, and (2) joint-memory tasks necessitating synthesis from
two distinct memories, formulated as josnt = (S, ! [g g where i, denote corresponding reference episodes from the acquisition stage. As shown Figure
3 we conduct experiments without memory usage to validate our framework design, confirming that agents struggle significantly with underspecified instructions.
Original Acquistion
Single-Mamory Utilization Joint-Memory Utilization
Dataset construction. We construct the dataset for ME- MENTO using Habitat 3.0 simulator (Puig et al. with simulated Spot robot (Boston Dynamics, 2025
2023
Success Rate (%)
80
Yokoyama et al. 2023). Our dataset spans 12 scenes with
total of 438 episodes, using PartNR (Chang et al. (024)
60
test set as foundation data since its multi-object rearrange- ment tasks align with our evaluation design.
40
20
Using GPT-4o, we generate personalized knowledge and create two instruction types: Jucg concatenating base in- struction, visual captions, and personalized knowledge for direct goal inference; and Tutat containing personalized references requiring memory recall. To prevent agents from identifying targets without personalized knowledge, we augmented scenes by placing distractors of the same type but with different appearances near target objects (e.g., placing a "red cup" next to the target "blue cup"). For quality control, we filtered episodes with similar memories referencing identical objects and manually reviewed failed cases to ensure instruction quality. We further validate the dataset with an inter-rater reliability analysis, achieving average Krippendorff's alpha of 0.69 across four key criteria. Details of the MEMENTO are provided in Appendix CK
GPT-4o
Llama3. 1-70b Llama3.1-8
Figure 3: The performance results with- out using episodic memory. Original in- dicates episodes from PartNR dataset.
3.4 METRICS & RETRIEVAL SETUP
Evaluation metrics. Following Chang et al. (2024), we use two main metrics: Percent Complete (PC) for the proportion of goal completion, and Success Rate (SR) for full task completion. We also report Sim Steps, which show the number of simulation steps required to complete the task, and Planning Cycles, which indicate the number of LLM inference calls made during task execution. Importantly, we report performance differences between acquisition and utilization stages as APC and ASR quantify agents' memory utilization capabilities. Note that for joint-memory tasks, these differences are computed relative to the average score of the corresponding acquisition stage episodes.
Retrieval setup. We first run the memory acquisition stage for all episodes to accumulate episodic memories, which are then used in the memory utilization stage. For memory retrieval in the utilization stage, we use top-k similarity- mamleeity-bbseed retrieval with fll-mpett-base-yy (Reimers & Gurevych, 2019). treating the current instruction a query and episodic memory instructions keys. Since our primary objective is evaluating memory utilization capability rather than retrieval performance, we ensure the corresponding gold memories are always included in the top-k results through randomly replacing retrieved candidates when necessary. We set k = 5 as our default configuration hecause this achieves 96.5% recall for gold memories in realistic retrieval scenarios, compared only 86. 1% recall at k 3. Comprehensive analysis of various retrieval strategies without gold memory guarantees is provided in Appendix E.3
4 EVALUATING MEMORY UTILIZATION ACROSS EMBODIED AGENTS
Through MEMENTO, memory utilization capabilities across multiple LLM- LLM--owered embodied agents of varying parameter sizes to understand how effectively they provide personalized assistance and identify their limitations (RQttionnn(()))
5

---

Published as a conference paper at ICLR 2026
Table 1: Model performance across memory acquisition and utilization stage in MEMENTO.
Model
Stage
Task Type
Planning Cycles
Sim
Percent Complete
PC
Success Rate
ARR
(Memory)
Steps 1
Acquisition
16.5
2156.1
96.3
95.0
GPT-4o
Utilization
Single Joint
16.1
2450.8
88.0
-8.3
85.1
-9.9
3480.7
86.7
63.9
Acquisition
16.0
Claudo-3.5 3.5id--3.--Snneet
Utilization
Single Joint
15.3
2258.8
69.3
-26.9
63.7
-30.3
27.8
3198.8
64.6
-30.1
33.3
-57.0
Acquisition
17.5
2,81.9
93.5
91.0
Qwen-2.5-726
Utilization
Single Joint
17.5
2691.2
72.6
-20.9
67.2
-23.8
31.3
4027.1
68.9
-27.9
36.1
-58.3
Acquisition
17.7
2162.1
92.9
90.0
Llama-3.1-700
Utilization
Single Joint
19.0
2566.6
72.2
200.7
66.7
-23.3
31.4
3425.2
51.3
44.9
8.3
Acquisition
19.3
2377.0
78.1
68.5
Llama 3.1 8h
Utilization
Single Joint
19.0
3131.7
48.1
-30.0
35,0
-33.5
27.4
3478.2
35.3
-5.5
8.3
-59.8
Acquisition
21.7
2476.8
64.1
53.2
Qwen-2.5-7b
Utilization
Single Joint
21.8
3271.0
39.1
-25.0
27.4
-25.8
26.9
4149.0
33.7
-34.2
5.6
-52.7
4.1 MODELS & IMPLEMENTATION DETAILS
We evaluate both proprietary models (GPT-4o (Hurst et al., 2024). Claude-3.5-Sonnet (Anthropic
2024)) and open-source models 1llma-3.1-70b/$5 GGattafiori et al.
2024). Qwen-2.5-72b/7b (Yang
et al. 2024) with different model families and sizes. We implement [.M L.M--poeredd embodied agents
following established architectures (Szot al.) 2021] Puig et al.) [023] Chang al. 2024). where
LLM serves as high-level policy planner that selects appropriate skills from predetined skill library using ReAct (Yao et al., 2023) prompting format. Note that, we use gold perception and motor skills
to focus on challenges related to embodied agents' cognitive abilities. Complete implementation
details are provided in Appendix C.I
4.2 RESULTS
L.M--poeree embodied agents struggle with un- derstanding personalized knowledge. In Table [ we observe that while GPT-4o maintains a relatively high success rate in the single-memory tasks, all mod- els show a success rate drop over 20% compared to the memory acquisition stage. This substantial per- formance decline across all evaluated models demon- strates that current embodied agents cannot reliably integrate personalized knowledge for task execution.
Acquisition Memory Utilication
Object Semantics
-15,1
-54.5
-38,4
3.4
Success Rate
88
Current embodied agents can effectively recall object semantics but struggle to comprehend user patterns. We then analyze the agents performance on single-memory task by personalized knowledge types. Notably, as shown in Figure 4 all models show minimal performance degradation for object semantics tasks, while user pattern tasks exhibit substantial performance drops across all evaluated models, indicating that current embodied agents face significant limitations in comprehending sequential information within behavioral contexts.
GPT-40 Claude Sonnet Qwen2.5-22b Llama3.1
3.-5onet
Figure 4: The results of personalized knowl- edge type based analysis (single-memoryy)
Agents default to commonsense over personalized knowledge. To understand the agents' behavior on personalized assistance tasks, we conduct qualitative case study which is shown in Figure 5 Through this analysis, we identify an interesting pattern: agents tend to rely on commonsense knowledge rather than personalized memory. For example, when reasoning about "afternoon refresh setup" (C.1), the agent relies on commonsense knowledge rather than referencing personalized memory, correctly predicting "jug and cup" because this commonsense happens to align with the

---

Published as a conference paper at ICLR 2026
Object Semantios Success
X Objecct Semanties Fallure
Reasoning
Personalization
Inaccurate
Couldlyou place the broad from
Raece put dennarware sut
Can you bring the gifted tay
Can you ng gift boot from my best thled?
fsuortin hustery....
togatter n anatte catliw.....
from my best friend bette?
Result
Result
Result
Result
Thought... found multipe natances
Thought The description of plate_.
Thought Sinco the toy is gift from
Thought Nothher bock..1 nor book.. malchhss
of oreeedd.... ined ideridly
mulches the typica aennaraare sul.
my bust
Action: Oeettoo (bread_o))
Thought Oreed_ matches the
Beasar duecsed erorp ziato 61 torget comimersonne reasonnnng
Bossion Mesendereetedd user's intentt
voAze personskae knowlogge
Bacsee: Salled pocurolaty recat the aescrption ohe book.
User Pattern Success
User Parteern Failure
Commonsense Reasoning
Reference
Commonsense Reasoning
Inacourate
Can you a rrance my afternoon refreshment setup??
Coud you sut my plants their usual spet!
Con you arange the toys for playtime?
Couls setup the cand for ocay
Result
Result
Result
Result
Thought .. will start recoioing the
Thought desermnn their usual
Thoug aghtt The bable the ty room conms line good spor to stacn the toys tocess cur playtime,
Thought Usesed on previtus ecamppies placing the condle on couch i5
ketchoon as it common place
spot wwll referto previous examples,
seoup for caey
Thought...
Secsso Hinolacod hova alla &e
of the afttafooot rufreshment setup.
jcundle tabll couet))
Figure 5: Taxonomy of successful and failed cases in memory utilization with illustrative examples. Top: success and failure cases of object semantics; Bottom: success and failure cases of user patterns.
user's actual behavior. However, when applying the same commonsense-bae based reasoning approach to "playtime" (D.1), the agent incorrectly places toys on the TV room table instead of consulting personalized memory that specifies the user's preference for placing toys on the bench. This indicates that when agents struggle to reference personalized knowledge from memory, they default to parametric commonsense knowledge rather than leveraging non-parametric personalized knowledge, leading to inconsistent performance across user contexts.
SCENARIO-BASED ANALYSIS FOR MEMORY UTILIZATION BOTTLENECKS
In this section, conduct scenario-basee analysis to understand the factors that negatively affect agents' memory utilization capabilities in personalized assistance tasks (RQ2) First, we examine agent performance when increasing the number of retrieved memories in the context ($ 5.Ip. Second, we analyze agent behavior when tasks require coordinating multiple memories to achieve task completion ($ 5.2
Memory Acquisttinn -
Memooy Utilication Stage
GPT-4o
Claude-3.-SSomeet
Qwen2.5-72
Llamak31-70h
Success Rate (%6)
80
50
70
40
30
50
Object Semantics
User Patterns
Figure 6: Success rate comparison across models as top-k value increases. Dashed lines represent memory acquisition stage baselines for each model.
5.1 ANALYZING AGENT PERFORMANCE UNDER INCREASED MEMORY CONTEXT
To understand whether agents remain robust in identifying correct information as memory volume increases, we vary the number of retrieved memories (top-k) in single-memon tasks. We ensure gold memory is always included and focus on whether agents can reference the correct memory.
Information overload degrades memory utilization performance. shown in Figure increasing consistently degrades performance across all models and both personalized knowledge types, indicating that agents struggle to extract relevant information when presented with larger sets of retrieved memories. We found that agents increasingly rely on commonsense knowledge rather patterns, suggesting
than personalized memories retrieved memories increase, especially that embodied agents suffer from information overload. Therefore, even though LLMs demonstrate impressive capabilities in processing long-context information (Wang al.) 2024b Liu et al. 2025)
7

---

Published as a conference paper at ICLR 2026
which confirm in Appendix D.5 memory systems that retrieve appropriate numbers of memories are crucial for personalized embodied agents.
5.2 INVESTIGATING AGENT BEHAVIOR IN JOINT-MEMORY TASKS
In realistic scenarios, users often refer multiple pieces of personalized knowledge within single instruction. Accordingly, we fix the top-k parameter to & = and employ joint-memory tasks in MEMENTO analyze the agents' performance tasks that require synthesizing information from two distinct episodic memories.
Agents fail to coordinate multiple memories. As shown in Table all models exhibit substan- tially larger performance drops in joint-memory tasks compared with single-memory tasks. Even GPT-4o's success rate declines by 30.5%, re- vealing serious bottlenecks when agents must coordinate multiple memories for personalized assistance. Figure 7 breaks down the joint- memory results by weeoonalizee-knowleedee type and, consistent with earlier findings, agents per- form particularly poorly on instructions involv- ing user patterns. Notably, unlike in single- memory tasks where the ASR for object seman- ties was only marginal, GPT-4o shows -20.8% drop when combining two object-semantic mem- ories. Because episodic memory inherently en- codes interaction history, the in-context learning nature of LLMs makes it difficult to simultancously reference multiple distinet memories. These findings underscore that agents struggle to effectively ex- tract personalized knowledge using current memory architectures, highlighting the need for improved memory designs to support personalized assistance tasks.
Overal
Memory Acquisition Stage
Object Semantics Object Semantics
Memory Stage
Object Semantics User Pattern
Ratter Pattem
100
30,5
224
Rata
GPT-40
Soude-15-onnee
Qwen225-722
Llarma. 1-200
Figure 7: Personalized knowledge type-based anal- ysis on joint- memory tasks.
6 EXPLORING MEMORY DESIGN FOR PERSONALIZED EMBODIED AGENTS
Based on our previous findings that demonstrate crit- ical bottlenecks in memory utilization for person- alized assistance, we identify that these limitations stem from the fundamental challenge of extracting relevant personalized knowledge from episodic mem- ories. We hypothesize that clearly providing personal- ized knowledge information an approach consistent with recent advances in context engineering (Mei will
Table
Model performance with memory format simplification approaches. Full mem- ory indicates the episodic memory.
Model
Memory
PC (%)
SR (%)
Full Memory
90.0
83.3
GPT-4e
Summarization
88.0
Instruction
62.4
50.0
Full Memory
77.2
66.7
et al. 2025) wwil enhance agents' ability to leverage
Qwen: 2.5m-2.5.73
Summarization
77.4
memory effectively. Therefore, in this section, we explore memory architectural approaches to improve agents' memory utilization capabilities for personal- ized assistance tasks (RQ3)
Instmation-only
51.3
40.0
Full Memory
72.8
63.3
Llama-3.1-88
Summarization
Instruction-only
40.0
30.0
Full Memory
50.1
43.3
Qwen-2.5-7h
Summarization
43.9
36.7
Insiruction only
35.6
23.3
6.1 INVESTIGATING MEMORY FORMAT SIMPLIFICATION
As episodic memory contains full interaction histories, it includes contextual information irrelevant to personalized knowledge extraction. Therefore, we investigate memory format simplification approaches that provide agents with refined information by removing irrelevant content, hypothesizing that this reduction noise will improve performance. We compare full episodic memory against two simplified alternatives: (1) GPT-4o summarized episodic memory that retains key information while removing trajectory details, and (2) instruction-onle setups that rely solely on task instructions, since personalized knowledge be extracted from instructions alone.
8

---

Published as a conference paper at ICLR 2026
Baseline
User Proffie Memory
Object Sementics
Overall
Object Semantics
Ovenall
Object Semantics
User Pattem
Object Semantics
User Pattern User Pattern
-7.
-20
+11.1
-30.2
+0.3
-60
-60
-60
Qwen-2.5-728 Uamo311-88
Qwen-25577
GPT-40
owen 2.5728
Liama 3.1-58
Qwen-255.7 7B
(o) Mimmor-Memory Tasks
(b) Joint- Memory Tasks
Figure 8: Agent performance with user profile memory across single- and joint-memory tasks.
Episodic memory provides both personalized knowledge and in-context learning benefits. As shown in Table summarized memory shows minimal impact on large models (GPT-4o, Qwen- 2.5-72b). Notably, smaller models (Llama-3.1-8) 3.bb.. Qwen-2.5-7b) exhibit performance degradation when using summarized memory compared full episodic memory. This performance drop occurs because summarizing removes the trajectory information that provides crucial in-context learning benefits for LLMs, reducing the overall task success rate. These findings demonstrate that episodic memory is essential module for embodied agents.
6.2 DEVELOPING HIERARCHICAL KNOWLEDGE GRAPHI-BSSED USER PROFILE MEMORY
Building upon our finding that episodic memory is an essential memory module, we devise a hierar- chical knowledge graph-based user profile memory module that manages personalized knowledge independently to provide cleaner, more accessible personalized information to the agent.
Memory module design. A key challenge in devel- oping personalized knowledge module is organiz- ing knowledge relationships effectively. Sequential knowledge like user patterns requires sophisticated memory structure to prevent information loss or fail- ing to adapt to evolving knowledge (e.g., "Add the mug from my coffee set to my morning routine and prepare my routine"). Inspired by previous works
"Add the from coffee to propare my momiing routtine"
Knowledge
Ferura
Or Anooslation
routine Dont parlons
my coffee set lnfuea ireience
pattem
Elements
O: otject
Of locelom
on knowledge graphs (Speer et al. 2017] [Bosellt et al. 2019), we construct personalized knowledge as a hicrarchical knowledge graph that captures both object semantics and user patterns.
conpoooet of
edge
Torger poccted
my edge
the
on the
syelza piote
As shown in Figure 9 user profile memory employs a Figure 9: Illustration of user profile memory. the top, knowledge
three-level structure with types (object semantics and user patterns) in the middle, and specific elements (objects, patterns, locations) the bottom. The graph connects these levels using hierarchical edges for structural relationships and temporal edges for sequential ordering within user patterns. This design enables systematic representation and easy updates of personalized knowledge-foo instance, red mug from a coffee set can be dynamically added to morning routine by updating the relevant knowledge relationships. More details of the user profile memory design, schema, extraction and retrieval algorithms are provided in Appendix D.I
User profile memory enhances embodied agents' memory utilization capability. As shown in Figure 8 agents with user profile memory achieves significant performance improvements across all models for both single-memory and joint-memory tasks. The performance gains are especially notable user pattern tasks, where the sequential structure had previously posed challenges. These results indicate that user profile memory effectively manages personalized knowledge, demonstrating its ability enhance agents' memory utilization capabilities for personalized assistance tasks.
CONCLUSION
This work investigates memory utilization challenges in LLM L.M--ooeerd embodied agents for person alized assistance through MEMENTO, revealing that while agents recall simple object semanties,
9

---

Published as a conference paper at ICLR 2026
they struggle with sequential user pattems, suffer from information overload when processing multi- ple retrieved memories, and fail to coordinate joint- joint-memory tasks effectively. We demonstrate that episodic memory provides both personalized knowledge and in-context learning benefits, explaining why simple summarization approaches fail. To address these bottlenecks, we design hierarchical knowledge graph-based user-profile memory module that separately manages personalized knowl- edge while preserving episodic memory benefits, achieving substantial improvements across both single-memory and joint-memory tasks. We hope these findings will inspire future research toward more sophisticated memory architectures that enable truly effective personalized embodied agents.
ACKNOWLEDGEMENTS
This project is supported by Microsoft Research Asia. This research was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT), under the Global Research Support Program in the Digital Field program (RS-2024-00436680) National AI Research Lab Project (RS-2024-00457788)) Artificial Intelligence Graduate School Program (Yonsei University) (RS-2020-1120136) We also thank Donghyun Kim, Juhyun Park, and Hyeonjong Ju for their helpful discussions and support. Jinyoung Yeo is the corresponding author.
ETHICS STATEMENT
This work involves no human subjects, conflicts interest, or legal compliance concerns. We use licensed resources (e.g., PARTNR (Chang al. 2024) under the MIT License) and commonsense knowledge without personal data, copyrighted content, or bias-amplifying material. We generated personalized knowledge from L.M--yythesizedd descriptions with prompts designed to reduce biased or discriminatory content, and we manually verified that the resulting user knowledge was free of such concerns. We acknowledge that embodied memory systems may pose privacy and security risks in real-world deployment. To mitigate these issues, future systems should adopt on-device processing, minimize retention, require user consent, and apply secure, transparent memory handling.
REPRODUCIBILITY STATEMENT
To facilitate full reproducibility of our results, we include complete evaluation dataset details and experimental procedures in the appendices. Furthermore, publish code at https:// githuub.com/Conooriggaaaalaa with thorough documentation.
REFERENCES
Christopher Agia, Krishna Murthy Jatavallabhula, Mohamed Khodeir, Ondrej Miksik, Vibhav Vineet, Mustafa Mukadam, Liam Paull, and Florian Shkurti. Taskography: Evaluating robot task planning over large 3d scene graphs. In Conference Robot Learning, pp. 48.58. PMLR, 2022.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
Anthropic. Introducing claude 3.5 sonnet, June 2024. URL Accessed: 2025-05-07.
news/ 3-tp//aaauee-----e
Luca Barsellotti, Roberto Bigazzi, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Person- alized instance-based navigation toward user- specifcc objects in realistic environments. In The Thirty-eight Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024, /tt.tttpps////gggeeeeeeeeeeeeeee
wiw.rrummmiuuuniin 1= F1sbp
Johan Bjorek, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr00t nl: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503. 14734, 2025.
10

---

Published as a conference paper at ICLR 2026
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. Comet: Commonsense transformers for automatic knowledge graph construction. arXiv 2019.
preprint
Boston Dynamics. Spot robot. httppp///bsstooppppeeasc//////// Accessed: 2025-04-28.
2025.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 15818, 2023.
Matthew Chang. Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, al. Parter: A benchmark for planning and reasoning in embodied multi-agent tasks. arXiv preprint arXiv:2411.00081, 2024.
Yongchao Chen, Jacob Arkin, Charles Dawson, Yang Zhang, Nicholas Roy, and Chuchu Fan.
Autotamp: Autoregressive task and motion planning with llms as translators and checkers. In 2024 IEEE International conference robotics and automation (ICRA), pp. 6695-6702. IEEE, 2024.
Jac-Woo Choi, Youngwoo Yoon, Hyobin Ong, Jachong Kim, and Minsu Jang. Lota-bench: Bench- marking language-oriented task planners for embodied agents. In International Conference on Learning Representations (ICLR), 2024.
Caitlyn Clabaugh and Maja Matarić. Robots for the people, by the people: Personalizing human- machine interaction. Science Robotics, 3011)ccaaaa711 2018. doi: 10. 106/sc/ccoooosccccss..o aat745..
URL https:://// /ttp://ceeereeerrro/////////////
Yinpei Dai, Run Peng, Sikai Li, and Joyce Chai. Think, act, and ask: Open-world interactive
personalized robot navigation, 2024.
Payel Das, Subhajit Chaudhury, Elliot Nelson, Igor Melnyk, Sarath Swaminathan, Sihui Dai, Aurélie Lozano, Georgios Kollias, Vijil Chenthamarakshan, Jiřf, Navrátil, Soham Dan, and Pin-Yu Chen. Larimar: Large language models with episodic memory control, 2024. URL
https:
K. Dautenhahn. Robots like live with?! developmental perspective personalized, life-long robot companion. In RO-MAN 2004. 13th IEEE International Workshop on Robot and Human Interactive Communication (IEEE Catalog No.04TH8859), pp. 17-22, 2004. doi:
10. 1091009// 200000 13747220.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference Robotics and Automation (ICRA), pp. 3553-2560 JEEE, 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinay Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407. 27788. 2024.
Qiao Gu Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open- vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 5021-5028. IEEE, 2024.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pp. 9118-9147. PMLR, 2022a
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207 050000 2022b.

---

Published as a conference paper at ICLR 2026
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023a.
Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng. Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, and brian ichter. Grounded decoding: Guiding text generation with grounded models for embodied agents. In seirth Conference on Neural Information Processing Systems, 2023b. URL htttps::/ggenneee.e..eeetttteeee JCC1 58 1Ush
Alexis Huet, Zied Ben Houidi, and Dario Rossi. Episodic memories generation and evaluation benchmark for large language models. arXiv preprint arXiv:2501.33121 2025.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21277 2024.
Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang. and Michael Bendersky. Query expansion by prompting large language models. arXiv preprint ww.ivv23305003655 2023.
Ivan Kapelyukh and Edward Johns. My house, my rules: Learning tidying preferences prefecnessss with graph neu- 2021.
ral networks. CoRR,
Byeonghwi Kim, Jinyeon Kim, Yuyeong Kim, Cheolhong Min, and Jonghyun Choi. Context- aware planning and environment-awwan memory for instruction following embodied agents. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV) pp. 10936- 10946, October 2023.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-actinn model. arXiv preprint 2.XY:22406.09266, 2024.
Klaus Krippendorff. Computing krippendorff's alpha-reliability. 2011.
Min Kyung Lee, Jodi Forlizzi, Sara Kiesler, Paul Rybski, John Antanitis, and Sarun Savetsila. Personalization in hri: A longitudinal field experiment. In 2012 7th ACM/IEEE International Conference on Human-Robol Interuction (HRI), pp. 319-326, 2012.
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Young Kim, Alan Lou, Caleb R Matthews, Ivan
Jerry Huayang Tang, Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, Karen Liu, Jiajun Wu, and Li Fei-Fei. Behavior- k: benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Karen Liu, Dana Kulic, and Jeff Ichnowski (eds.), Proceedings The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pp. 80 93-93, PMLR, 14-18 Dec 2023. URL http://pcocceeennnoaaaaaaaaaaaas http://w....p....//////aa... http:///02//1122atmmmm
https://proceeddnggs
Hao Li, Chenghao Yang, An Zhang, Yang Deng, Xiang Wang, and Tat-Seng Chua. Hello again! LLM L.M--ooereed personalized agent for long-term dialogue. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. $259-5276 Albuquerque, New Mexico, April 2025a. Association for Computational Linguistics. ISBN 881-8-888976-----8
htthtt::: http:://acaaatttacc
1ong.2127
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking Ilms for embodied decision making. Advances in Neural Information Processing Systems, 37:100428- 100534, 2025b.
12

---

Published as a conference paper at ICLR 2026
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, and Yuke Zhu. Pre-trained language models for interactive recision-making In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, 2022. ISBN 9781713871088.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian lchter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In arXiv preprint arXiv:2209.07753. 2022.
Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.17407, 2025.
Zeyi Liu, Arpit Bahety, and Shuran Song. Reflect: Summarizing robot experiences for failure explanation and correction. arXiv preprint arXiv:2300./5572, 2023.
Lingrui Mei, Jiayu Yao, Yuyao Ge, Yiwei Wang, Baolong Bi, Yujun Cai, Jiazhi Liu, Mingyu Li, Zhong-Zhi Li, Duzhen Zhang, et al. A survey of context engineering for large language models. arXiv preprint arXiv:2507. 13334, 2025.
Meta. Introducing llama 3.1: Our most capable models to date, July 2024. URL https:// www.meta.com/bblcom 1atp://mmmmttmmaaaaaaaaaaaa Accessed: 2025-05-16.
Yao Mu, Qinglong Zhang. Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. EmbodiedGPT: Vision-language pre-training via embodied chain of thought. In Thirty-sevnnth Conference Neural Information Processing Systems, 2023. URL
htttps://openreview..c for:m? w II5Z 1o
Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, lon Stoica, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems, 2024. URL http:::://axxi..orrrrr
abs/ / 2310.08560
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In In the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23), UIST '23, New York, NY, USA, 2023. Association for Computing Machinery.
Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B. Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help: A challenge for social perception and human-ai collaboration, 2021. URL httphhttss//aaaii...rrteee /abp 2ttp://111109900
Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung- Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, et al. Habitat 3.0: A co-habitat for humans, avatars and robots. arXiv preprint arXiv:2310.13724, 2023.
Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning. arXiv preprint arXiv:2307.06134 2023.
Nils Reimers and Iryna Gurevych. Sentence-bert Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
Matthew Rueben and William D Smart. Privacy in human-robot interaction: Survey and future work. We robot, 2016:5th, 2016.
Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. Improving passage retrieval with zero-shot question generation. arXiv preprint arXiv:2204.07449 2022.
Ranjan Sapkota, Yang Cao, Konstantinos models: Concepts, progress, applications and challenges. arXiv preprint 0:X10:25005.0469, 2025.
Roumeliotis, and Manoj Karkee. Vision-languuge-action
13

---

Published as a conference paper at ICLR 2026
Gabriel Sarch, Yue Wu, Michael Tarr, and Katerina Fragkiadaki. Open-ended instructable embodied agents with memory augnened large language models. arXiv preprint XXi::2310.151277 2023.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11523-1 11523-1133,, 2023. doi: 10.1109//CR448811 2222.1116133
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference computer vision, pp. 2998-3009, 2023.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents. arXiv preprint 0rX452230.00952,, 2023.
Andrew Szot, Alexander Clegg. Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in neural information processing systems, 34:251-266, 2021.
Bing Wang, Xinnian Liang, Jian Yang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. Enhancing large language model with self-controlled memory framework, 2024a.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint Xrvv:22305. 16291, 2023a.
Liang Wang. Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models. arXiv preprint arXiv:2303.07672 2023b.
Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadch, and Armaghan Eshaghi. Beyond the limits: A survey of techniques to extend the context length in large language models. arXiv preprint arXiv:240..02244 2024b.
Zixuan Wang, Bo Yu, Junzhe Zhao, Wenhao Sun, Sai Hou, Shuai Liang, Xing Hu, Yinhe Han, and Yiming Gan. Karma: Augmenting embodied ai agents with long-and-short term memory systems. arXiv preprint arXiv:2409.1 022240999490,,, 14908, 2024c.
Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng. Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. Autonomous Robots, 47(8):1087-1102 2023.
Yike Wu, Jiatao Zhang, Nan Hu, Lanling Tang, Guilin Qi, Jun Shao, Jie Ren, and Wei Song. Mldt: Multi-level decomposition for complex long-horizon robotic task planning with open-source large language model. In International Conference on Database Systems for Advanced Applications, pp. 251-267. Springer, 2024.
Quanting Xie, So Yeon Min, Pengliang Ji, Yue Yang, Tianyi Zhang, Kedi Xu, Aarav Bajaj. Ruslan Salakhutdinov, Matthew Johnson-Roberson and Yonatan Bisk. Embodied-rag: General non- parametric embodied memory for retrieval and generation. arXiv preprint arXiv:2409.1833 2024.
Manjie Xu, Xinyi Yang, Wei Liang, Chi Zhang, and Yixin Zhu. Learning to plan with personalized preferences. arXiv preprint arXiv:2502 00858, 2024.
Wujiang Xu, Zujie Liang, Kai Mei. Hang Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for Iim agents. arXiv preprint arXiv:2502. 12/10, 2025.
14

---

Published as a conference paper at ICLR 2026
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
Rui Yang. Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking mulli-moda large language models for vision-driven embodied agents. arXiv preprint 00Xiv::5502.096660 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference Learning Representations (ICLR), 2023.
Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung- Yen Yang. Vidhi Jain, Alexander William Clegg. John Turner, al. Homerobot: Open-vocabulary mobile manipulation. arXiv preprint arXiv:2306. 11565, 2023.
Naoki Yokoyama, Alex Clegg, Joanne Truong. Eric Undersander, Tsung- Yen Yang. Sergio Amaud, Sehoon Ha, Dhruv Batra, and Akshara Rai. Asc: Adaptive skill coordination for robotic mobile manipulation. IEEE Robotics and Automation Letters, 911):799-7788 2023.
Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. arXiv preprint 2rxXiv:307.02288. 2023.
Wanjun Zhong, Lianghong Guo, Qiqi Gao, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory, arXiv preprint 20X.:22305.10250, 2023.
A LIMITATIONS, PRIVACY, SOCIETAL IMPACTS AND LLM USAGE
A.1 LIMITATIONS
This work focuses on agents' memory utilization capabilities for providing personalized assistance. While our study provides valuable insights, several limitations must be acknowledged. First, our experiments were conducted in a controlled simulator environment (Puig et al. 2023 Although we focused on agents' cognitive abilities and expect marginal impact on our results, this controlled envi- ronment may not fully capture the complexity and variability of real-world, presenting potential risks for generalization. Second, we utilize oracle perception and motor skills to isolate memory utilization as the primary performance factor. Recent research has increasingly incorporated visual components into embodied agents (Huang et al. [223a Yang al. (2025) developed
Sssion-Language--ee Action (VLA) models that generate end-to-end actions (Brohan et al. 2023 Kim et al., 2024| [Borck et al., 2025). Additionally, recent studies have shown that primary limitations in embodied tasks often
exist at the observation and action levels (Chang et al., 2024 Sapkota et al. 2025). Therefore, while our focus on language-base specific aspect, it may not capture all failure modes present in actual embodied agents. Finally, our framework lacks conversational components that are crucial for effective personalized assistance. Beyond memory utilization,, personalized embodied agents require robust communication capabilities, including the ability to clarify ambiguous instructions, proactively engage with users, and adapt their communication style to individual preferences. Investigating how memory utilization integrates with conversational abilities to enable more comprehensive personalized assistance represents an important direction for our ongoing research.
memory utilization capabilities provides clear insights into this
A.2 PRIVACY AND SECURITY CONSIDERATIONS
As our work involves personalized knowledge and agent memory systems, we recognize potential privacy risks including unauthorized access to user-specific knowledge and inference of sensitive personal information from interaction patterns. While our current study utilizes synthetic personalized data to mitigate privacy concerns, weal-world deployment of personalized embodied agents requires robust privacy safeguards including encryption of user-specific knowledge bases, secure access
15

---

Published as a conference paper at ICLR 2026
controls with proper authentication protocols, data minimization strategies, and mechanisms for user control over personal data. The specific implementation of these privacy protection mechanisms should depend on deployment context and applicable regulatory requirements, and recommend thorough privacy impact assessments before real-world deployment.
A.3 SOCIETAL IMPACTS
Our work explores how embodied agents can remember and adapt to user-specific preferences through episodic memory, enabling more personalized and natural interactions. This capability has the poten- tial enhance convenience, efficiency, and user satisfaction in everyday eavironmentts-beneeine wide range users regardless of age or ability. However, personalization also raises concerns about privacy, bias reinforcement, and over-dependence 2012). Since episodic memory involves storing interaction history, future systems must consider secure and transparent memory handling. Although study is conducted in controlled simulator, these considerations will become crucial as such systems move toward real-world deployment. We hope that MEMENTO serves foundation for future research in building safe, privacy-aware, and user-aligned personalized embodied agents.
Al agents (Rueben & Smart 2016 Lee et al.)
A.4
LLM USAGE
Our use of LLMs was for the following specific purposes:
Writing and Editing Assistance: We utilize LLMs for improving the clarity, grammar, and readability of the text. This included correcting grammatical errors, and ensuring consistent terminology throughout the paper.
Code Generation for Experiments: We utilize L.MMs accelerate the implementation of our experimental framework.
The authors meticulously reviewed, edited, and validated all content generated or modified by LLMs. The conceptual arguments, experimental design, analysis, and final conclusions presented in this paper are the original work the authors. We take full responsibility for content of this work.
B EXTENDED RELATED WORK
Our work focuses memory and personalization in embodied agents. Here, we broaden the related work to include studies on memory-augmented agents, personalization, and memory architectures within the wider research context.
augmened agents have increasingly been studied as a means to support long-term rea- soning, plamning, and adaptive behavior in LLM agents. Park et al. (2023) propose Generative Agents, which simulate human- behavior by maintaining a memory stream of past experiences in natural language. This memory stream enables agents reflect prior events, retrieve relevant information, and plan future actions based on their histories. In a similar direction, Xu et al.) (2025) introduce A-Mem, dynamic memory system in which memory is represented evolving and interconnected notes. The agent can generate, retrieve, and update these notes over time, thereby supporting adaptability.
Memory for dialogue and personalization has emerged as a key direction, highlighting how memory supports user-specific context and long-term consistency in interactions. [i al. (2025a) present system that leverages both short-term and long-term memories to maintain user-specific context across multiple conversation sessions, reporting improvements dialogue coherence and persona consistency. Meanwhile, Wang et al. (2024a) propose the where the agent dynamically decides when and what to store or retrieve, achieving greater coherence and knowledge retention over extended interactions. Alongside these approaches, structured long- term and episodic memory have been introduced to enhance personalization. Zhong et al.) (2023) develop MemoryBank, which stores and retrieves interaction history to support consistent responses across turns. Das et al. (2024) present Larimar, a framework that enables selective recall and forgetting, providing controllable knowledge updates that support personalization scale.
Memory (SCM) framework,
16

---

Published as a conference paper at ICLR 2026
Architectural perspectives memory focus not only building memory components for individ- ual tasks but on treating memory as a fundamental part of the overall system. Packer et al.) (2024) illustrate this idea with MemGPT, which frames memory management through an operating-systere analogy. In this view, the LLM functions like an operating system that autonomously organizes and manages both its in-context and external memory, rather than relying solely user-provided prompts. This perspective emphasizes memory as central architectural layer, enabling scalable and autonomous agents that can handle complex, long-horizon tasks.
C DETAILS OF EVALUATION SETUP
C.I LLM-POWERED EMBODIED AGENT ARCHITECTURE
Following Szot et al. 2021) Puig et al. (2023) Chang et al. 2024),, we adopt a two-layer hierarchical control architecture for our LLM-powent embodied agent. We utilize the LLM high-level policy planner that selects appropriate skills from the predefined skill library. The selected skill then provides control signals to the simulator. For memory systems, we implement a textual scene- graph as our semantic memory alongside an episodic memory.
Skill library. The skill library consists of oracle low-level skills that the LLM policy can select as actions. These action skills are divided into motor skills (e.g., navigate, pick, place) and perception skills (e.g., describe ceribe_objeeeeeee ject, find_object, find_reece rend_receptacle) Note that we exclusively used oracle skills in our skill library to focus episodic memory-centric analysis. Further descriptions are provided in Table and Table
Table 3: List of available agent motor skills.
Skill
Description
Navigate
Used for navigating to entity. You must provide the name of the entity you want navigate Example: Navigate[counter 22].
Pick
Used for picking up an object. You must provide the name object. Example: Pickfcup 1].
Place
Used for placing an object on a target location. Example: Place[book_ O, on, table_ 2, None,
None].
Open
Used for opening articulated entity. Example: Open[chest_ _f drawers_ 1].
Close
Used closing an articulated entity. Example: Closefchest drawers I].
Explore
Doing exploration towards a target object or receptacle, you need to provide the name of the place you want to explore.
Wait
Used to make agent stay idle for some time. Example (Wait())
Table 4: List of available agent perception skills.
Skill
Description
FindObjecrTool
Used to find the exact name/names of the object/objects of interest. you want find exact names objects on specific receptacles fürnitures, please include that the query. Example (FindOhjeerToolfff) Tool[toys on the floor) or FindOhjectToollapplae Used to know the exact name of a receptacle. A receptacle is furniture or entity (like chair, table, hed etc.) where you can place an object. Example (FindRecepta- cleTool[a kitchen counter]).
FindReceptacleTTon
FindRoomTool
Used to know the exact name a room in the house. room a region in the house where furniture is placed. Example (FindRoomTooll) room which might have something to eat)).
DescribeObjectTool
Used to retrieve a bricf descriptive semantic meaning of given object or furniturc name. Example (oeseribcOhjeetTolllss) (oonee
Semantic memory. For semantic memory, we implement a acene-graph style hierarchical rep- resentation, which has demonstrated effecttivnness for planning problems (Agia al.) 2022 Rana ct al. 2023) Gu et al. 2024). Following [Cang et al. (2024), we utilize a multi-edge directed graph with three distinct levels to represent environmental entities. The top level contains single root.
17

---

Published as a conference paper at ICLR 2026
node representing the house environment, the second level comprises encompasses furniture, objects, and agents. Each node stores the corresponding entity's s 3D location and relevant state information. This graph structure is initialized and continuously updated with ground-truth information from the simulator at each state 8t- In our system, this structured semantic memory provides the LLM planner with interpretable representation of the environment, which can be flexibly queried and reasoned over through natural language descriptions.
nodes, and the third level
Episodic memory. Our episodic memory is configured to store the ReAct-style formatting that guides the LLMs' reasoning process (Yao et al., 2023). This memory structure captures both the user's instruction and the complete sequence of < <Thought, Action, Observat ion> triplets generated during task execution. The episodic memory is accessed at the beginning of each task by retrieval and updated upon task completion, enabling the agent recall previous interactions.
Memory retrieval. For retrieval, we encode instructions and memory entries using the all-mpnet- base-v2 sentence transformer (Reimers & Gurevych] 2019) and use the current task instruction the query. To avoid ambiguous or irrelevant memories, we retrieve candidate memories only from the history within the same scene the current task.
LLM inference setup. We configured the language model with a temperature of 0 to ensure deterministic outputs. For sampling parameters, set top_p and top_ to 50.
C.2 PERSONALIZED KNOWLEDGE DETAILS
We categorize knowledge about personal items as object semantics and knowledge about consistent behaviors user patterns structure our evaluation approach. Object semantics can be further classified into four sub-categories naive ownership (e.g.. "my cup"). object preference (e.g., "a chessboard play chess with my brother"), history (e.g.. "graduation gift from my grandma") "). and group (e.g.. "my favorite toys", where toys indicate toy airplane and toy truck). User patterns encompass consistent action sequences in specific contexts, with two sub-categories: personal routine (e.g., "my remote work setup") and arrangement preference (e.g.. "my movie night setup"). Based on these personalized knowledge categories, we designed tasks that specifically require agents to recall and apply this information to evaluate their memory utilization capabilities. Further details provided in Table and Table
Table 5: List of the suhcategories for object semantics.
Type
Description
Example
Ownership
Possessive reference to the user's belonging
My cup, My laptop
Preference
Object aligned with the user's individual taste or selee- tion
Bread from my favorite bakery, Jug for serving drink
History
Object linked to personal memory past experience
photo of the my beloved pet, travel souvenir vase
Groups
Conceptual functional grouping of multiple related objects
my home office setup, my travel essentials
Table 6: List of the subcategories for user patterns.
Type
Description
Example
Routine
A sequence or setup the user follows as habit regular activity.
meal time setting, setup for cooking routine
Preference
A specific way the user prefers to prepare or arrange their environment when a particular situation occurs.
my coffee break, cozy decora- tion spot
18

---

Published as a conference paper at ICLR 2026
C.3 DETAILS OF MEMENTO
C.3.1 FOUNDATION DATASET
We adopt the test set of PartNR (Chang et al. 2024) foundation object rearrangement task dataset. PartNR is designed to evaluate planning and reasoning capabilities in embodied tasks and includes four primary task types: (1) constraint-free basic rearrangement, (2) spatial tasks requiring reasoning about object positions, (3) temporal tasks with sequential dependencies, and (4) heterogeneous tasks involving human-only actions. We select PartNR episodes for their complexity beyond simple scenarios. The rich linguistic structure and diverse task requirements make PartNR particularly suitable for evaluating personalized user patterns, enabling us to create scenarios that effectively test agents' ability to adapt to user-specific behavioral routines.
operations, filtering out tasks irrelevant to personalized assistance
C.3.2 DATASET GENERATION PROCESS DETAILS
We leverage GPT-4o and a systematic process to incorporate personalized knowledge into existing task structures, creating tasks tailored for each evaluation stage.
Captioning process. Since we employ LLMs high-level policy planners for embodied agents, we generate natural language descriptions of objects in the scene using GPT-4o, enabling agents to reason over object properties without relying on direct visual perception. We collect object models from the OVMM dataset (Yenamandra et al. (023) and use GPT-4o to generate descriptions from rendered object images. In particular, the Google Scanned dataset (Downs al. 2022) included within OVMM provides object identities in file names, which we leverage alongside the images to produce more realistic and accurate descriptions. Through this process, we generate 1,920 object descriptions spanning 66 object categories. The prompts used for generating object descriptions are provided in Appendix
Preprocessing scenes. To collect suitable episodes for personalized assistance tasks, we prepro- cessed and filtered episodes using three criteria. We only gathered non-heterogeneous. episodes that can be executed by robotic agents, excluding tasks that require human-only actions. We filtered out tasks where target objects were not uniquely specified (e.g.. "Bring one apple to the kitchen table"), since our task setting requires identification specific objects based on personalized knowledge unavailable,
that distinguishes them from similar objects. When captions for target objects substituted alternative objects from the same category, or excluded the entire episode no suitable alternatives existed.
Distractor sampling. To sample distractor objects for episodes focusing object semanties, utilized PartNR's 's dataset generation methods. This approach systematically selects objects located adjacent target objects on the same receptacles floor surfaces, requiring agents differentiate between objects of the same category.
Task instruction generation process. We prompt GPT-4o to generate personalized knowledge tasks based on the knowledge categories defined in Sec- tion
with Sta idarrd Destation
For object semantics tasks, we provide target object captions and guide GPT-4o generate natural se- mantic descriptions. We first instruct the model to create the most plausible subcategory of object se- mantics, then generate corresponding personalized knowledge. The resulting instruction comprises three components: the command instruction, additional ob- ject information, and personalized knowledge. For example: "Bring the cup the kitchen table" (com- mand) + "The cup is a white mug with fancy handle"
COART
Figure 10: Episodes with zero success rate (31 in total) were excluded from the analysis. (description) "That mug is my coffee mug" (personalized knowledge).
19

---

Published as a conference paper at ICLR 2026
For user pattern tasks, we provide base instructions and allow GPT-4o to infer plausible user patterns corresponding to action sequences. Memory acquisition stage instructions concatenate the command with personalized knowledge (e.g., "Bring the cup and dish from the kitchen table to living room. That's my dinner setup"). Detailed prompts are provided in Appendix
Quality control. To ensure dataset quality, we implement a two-stage filtering process after generating episodes for MEMENTO. We first apply rule-based heuristic filters guided by six explicit criteria to identify problematic episodes:
Semantic misalignment - Instructions inconsistent with stored memory (e.g., memory states "my favorite cup" but instruction refers to "my cup")
Triviality Instructions solvable without memory access (e.g., "Prepare my coffee time cup setup on the kitchen table" when setup details are explicitly provided)
Redundancy - Instructions repeating knowledge from previous episodes in the same scene (e.g.. "my usual morning cup" when earlier episode already established "the cup I usually use in the morning")
Environment inconsistency Innssstttnnyy-ssuuuuttininn ing target cup as red when it's actually blue, but distractor cup is red)
Instructions conflicting with actual object properties (e.g., describ-
Instruction inconsistency Semantic inconsistency between acquisition and utilization stages (e.g., changing from "pick from A and place at B to only "place at B")
Temporal irrelevance Instructions with expired temporal context (e.g., referencing "today's temperature" when the memory is no longer current)
Following this rule-based filtering, we test each remaining episode with GPT-4o. excluding episodes where the model fails five consecutive attempts as this indicates potential feasibility clarity issues. This comprehensive quality control process filters out 31 episodes (13.4% showing consistently poor performance, all with zero success rates indicating complete task failure. The remaining 201 episodes demonstrate strong performance with 95% confidence intervals of [0.755, 0.844] for success rate and [0.823, 0.893] for completion rate. The high correlation (r = 0.908) between these metrics confirms consistent performance across evaluation measures.
C.3.3 ANTER-RATER RELIABILITY ANALYSIS
To validate the consistency of our heuristic filtering process, conduct inter-rater reliability analysis us-
Semantics
60
User Pattern
ing Krippendorff's ' alpha (Krippendorfi 2011). Mul- tiple annotators independently evaluate held-out subset of instructions using our six filtering criteria. Since Environment inconsistency and Temporal relevance are strictly controlled during instruction generation, they achieve near- perfeet agreement and exclude them from reliability analysis. We com- pute Krippendorff's alpha for the remaining four cat- egories, obtaining values of 0.75 for Semantic mis- alignment, 0.87 for Instruction inconsistency. 0.58 for Triviality, and 0.57 for Redundancy. The average alpha of 0.69 across these criteria indicates moderate agreement, supporting the reliability of our filtering process for maintaining dataset quality.
Percmtgg (6)
80
3
Number (object, location)
Figure 11: Distribution required (object, location) pairs across object semantics and user pattern tasks.
C.3.4 DATASET STATISTICS
The dataset comprises 438 episodes, divided into two main stages. The memory acquisition stage contains 201 episodes (89 object semantics tasks and 112 user patterns tasks). The memory utilization stage also contains 201 single-memor episodes (89 object semantics and 1 122 user patterns tasks), along with 36 mulli-memoon episodes, which include 12 object semantics pairs, 12 user patterns pairs, and 12 mixed pairs. All episodes constructed using the Habitat 3.0 simulator.
20

---

Published as a conference paper at ICLR 2026
Number of (target, location) pairs. We compared the number of (object, location) pairs required for successful completion across both object semantics and user pattern tasks. As shown in Figure [T the distribution of required (object, location) pairs is nearly identical between the two personalized knowledge types. This indicates that the performance difference between object semantics and user patterns stems from the intrinsic characteristics of each knowledge type rather than differences in task complexity measured by the number of required object-location pairs.
C.4 EVALUATION METHOD
We adopt the official evaluation protocol from the PartNR benchmark (Chang et al. 2024). which pro- vides python-based framework for assessing multi-step rearrangement tasks. We use this framework without modification. The evaluator analyzes simulator states using three components: (1) proposi- tions that define object relationships to satisfy (e.g., is_on_top [sspoon_]], [table_1]] (2) dependencies that define temporal conditions for multi-step instructions after_ unsatis insstissfi sf id), and (3) constraints that enforce execution requirements (e.g., ordering, object consistency across steps). We rely on this system to evaluate tasks involving ambiguous references and sequential dependencies. The evaluation produces percent-complete score and binary success indicator.
C.5 COMPUTING RESOURCES
Our experiments primarily utilized commercial API services rather than local computing resources. We used OpenAl's Chat API for accessing GPT-4o, Claude models through Anthropic's 's API, and OpenRouter's API service for accessing 1lama-3.1 (Meta 2024), Qwen-2.5 YYngg et al. 2024). For running simulation environment we used 8 NVIDIA GeForce RT' 3090 GPUS. For our implementa- tion and evaluation, we use Huggingface library2, vLLMM library. Both libraries are licensed under Apache License, Version 2.0. And we used langchain library, under MIT License. We have confirmed that all the artifacts used this paper are available for non-commercial scientific use.
D EXPERIMENT & ANALYSIS DETAILS
In this section, we provide a detailed description of the experimental setup, analysis results, and the memory module, complementing the explanations provided in the main text.
D.1
USER PROFILE MEMORY
Our work introduces a user profile memory module and offers initial evidence that, when combined with episodic memory, structured personalized knowledge can enhance performance on personalized assistance tasks.
Knowledge graph design. We define the user profile memory hierarchical knowledge graph that organizes personalized knowledge into three levels Usee, Knowledge, and Elements- to provide structured representation that is both scalable across from overlapping entities. The hierarchical partitioning enables the system to scale to many users while preserving efficiency, since overlapping objects can be represented connected via edges instead of being redundantly replicated. To support this representation, each nodes connected through two types of edges: hierarchical edges, which capture inclusion relations across and within levels, and temporal edges, which encode the sequential structure of user patterns. Together, these edge types allow the graph to comprehensively represent the personalized knowledge that users may require an embodied agent to leverage in real-world interactions.
and resistant to redundancy
canonical form and
Knowledge graph update procedure. While the hierarchical design specifies how knowledge is organized, the graph must also evolve users provide new information or refine existing entries. Algorithm presents the procedure for updating the knowledge graph. Given a user instruction and the current graph G, the LLM processes the instruction by extracting candidate knowledge IC and associated elements E. For each candidate knowledge k. the relevant subgraph Gt, is filtered, and similarity search is applied to collect matching nodes C. These candidates then expanded by collecting all connected edges and their descendant nodes, and the resulting subgraphs are passed to L.M. The LLM determines whether the instruction corresponds to addition or update,
21

---

Published as a conference paper at ICLR 2026
executes the corresponding operation. For update, the LLM removes the existing knowledge node and generates new one, and for an addition it simply creates a new knowledge node. both cases, the LLM determines whether to reuse or instantiate the associated object, location nodes based on whether their similarity scores exceed predefined threshold.
Algorithm 1: Update Knowledge Graph
Input: User instruction I, Knowledge graph G Updated knowledge graph G"
Output:
// Step Extract candidate knowledges from instruction K + Eatract // K ((k1,t1), (22,t))....
C+ Ø
foreach ()) E Co
// Step 2: Extract elements (potteems/dbjects//lcrrooro
tied to this knowledge
8 Elementsl) (2);
// Step 3: Select subgraph by type G1 + SSbbrupph( type lx):
1/ Step 4: Retrieve candidates for knowledge and elements
C-CUU CUSiiitrriySSeeehhCCC
4x);
foreach (e,te) E E do
L C CU SimilaritySearchhCC
// Step 5: Decide add or update op Decide peciideA
if op update then return G' <- Update Nooleeeeeoeee Expand((),
else if op add then
return G' <- AdddKKoonledgeeooeeCC....e
Algorithm 2: RetrievalProcedure
Input: User instruction I, User profile memory graph G Output:
Retrieved natural language descriptions R
!/ Step Parse instruction
K. Fatract
C.R. Φ
11 Step 2: Filter subgraph and run similarity search foreach (k,t)) EC
G f- Subgraph(G, type = t);
C C SimilarittSSarcchC.,, k,44;;
C + RemoveDDplicates(():
// Step 3: Reformulate results with LLM
foreach (c, tz) do
R + RU Re formulale (Expand(e))
return
Retrieval procedure. To leverage personalized knowledge during task execution, the agent retrieves information from the user profile memory using hybrid retrieval pipeline that combines embedding- based search with LLM reasoning. This procedure ensures that the retrieved knowledge is both semantically aligned with user instructions and interpretable in natural language.
Knowledge graph schema. We summarize the node and edge types in Tables 7 and and provide complete examples user profile memory instances in Figures 12 and 13
22

---

Published as a conference paper at ICLR 2026
Table 7: Edge schema of the user profile memory, including relation types and JSON-style examples.
Edge Type
Source
Relation
Target
Example
Hierarchical
user
refers_to
knowledge
[ "source": "ul", "target": "k1", "type": "Hierarchical", "relation": "refers_to" { "source": "kI", "target": "pl". "type": "Hierarchical", "relation": "entails")
Hierarchical
knowledge
entails
pattern
Hierarchical
pattern
target
object
[ "source": "pl". "target"; "ol", "type"; "Hierarchical", "relation": "target")
Hierarchical
pattern
target
location
{ "source": "pl", "target": "I1", "type": "Hierarchical", "relation": "target")
Hierarchical
knowledge
composed_ov of
object
{ "source": "kl", "target": "o2", "type": "Hierarchical", "relation": "composed_of" oo
Temporal
pattern
before
pattern
[ "source": "pl". "target": "p2", "type": "Temporal", "relation": "be- fore")
Table 8: Node schema of the user profile memory.
Node Type
Attributes
Example
User
id, type,
{ul, "user", "james")
Knowledge
type, subtype, alias, description
{k_objl, "knowledge", object_semantics "my favorite cup", yellow cup with wooden handle")
Pattern
id, type, name, args
{pl. "pattern". "place". [cup, on. livin- groom_tablle
Object
id, type, name, granularity id, type, name, expression
(o2, "object", "red dog image cup". instance)
Location
[II. "location", livingroom table, "on blc")
Object Semantic Knowledge: James' dinnerware set
"nodes": [
{ ("id": "ut", "type": "User", "name": "James"), (iid": "k1", "type": "Knowledge", "subtype": "object semantics", "alias": "my dinnerware
set"),
("id": "o1", "type": "Object", "name": "plate", "granularity": "instancee" "attributes": ["square", "white", "black geometric pattern"]), ("id": "o2", "type": "Object", "name": "bowl", "granularity": "instance", "attributes": ""white", "brown lid"])
J.
"edges": [
("source": "u1", "target": "k1", "type": "Hierarchical", "relation": "refers to"),
("source": "k1", "target": "o1", "type": "Hierarchical", "relation": "target"),
("source": "k1", "target": "o2", "type": "Hierarchical", "relation": "target")
Figure 12: Example of user profile memory graph representing object semantics knowledge.
23

---

Published as a conference paper at ICLR 2026
User Pattern Knowledge: James' living room play setup
"nodes": [
("Id": "ut", "type": "User", "name": "James"),
("id": "k1", "type": "Knowiedge", "subtype": "user pattern", "alias": "living room play
setup", "description": "keep toy vehicle on the living room couch it's easy to find when playing"),
("id":-p1", "type": "Pattern", "name": "move", "args": ["toy vehicle", "from bedroom", "to
living room"]),
"p'", "type": "ttppe "Pattern,, "name": "place", "args": ("toy vehicle", "on couch", "in living room"]),
("id": "ol", "type":
"Object"" "Object", "name": "toy vehicle"),
("id": "type": "Location", "name": "bedroom", "expression": "in the bedroom"), ("id": "12", "type": "Location,, "name": "livingroom", "expression": "in the living room"), """"","3", "type": "Location, "name": "couch", "expression": "on the in the living room")
1,
"edges":
("source": "u1", "target:: "k1", "Her,""ttt""""""aarccchiea "relation": "refers to"),
("source": "target": "p1", "type": "Hierarchical", "relation": "entails"),
("source": "target": "p",, "type": "Hlerarchical", "relation": "entails") ) "Hierarchical", "o", "Hiierrcc"""""""""rcccccii""" "relation": "target"",
("source": "tartttt "target":
"Hierarchical,, "relation": "target"), ("source": "p1", "p1", "target":: : "I2", "Hierarciiir "relation": "relattio"",,,..aiii"n "target"), ("source": "p2", "target": "o1", "type": "Hierarchical,, "relation": "target"), ("source": "p2", "target": "13", "type": "Hierarchical,, "relation": "targett",
"I",
("source": "p1", "target": "p2", "type": "Temporal", "relation": "before")
Figure 13: Example of a user profile memory graph representing user pattern knowledge.
D.2 SUCCESS AND ERROR CASE ANALYSIS
As shown in Figure 5 we sampled success and error cases in memory utilization stage.
Tasks require object semanties knowledge. Success cases demonstrate that agents can effectively reference personalized object attributes from episodic memory, correctly identifying and applying the specific information needed for task completion. (A.1) However, we observed distinct eπro patterns when agents needed to utilize this type of knowledge: the most prominent being commonsense reasoning errors (B.1), where agents incorrectly relied generic object descriptions instead of personalized cues; missed personalization cues (B.2), where agents failed to recognize the need to access personalized knowledge; and memory recall failures (B.3), where agents unable to locate relevant information despite its presence in the provided context.
Tasks require user patterns. For user patterns tasks, we observed that agents employed two distinct strategies: commonsense reasoning (C.1) treating memory as exemplars for step-by-step reasoning, and direct reference (C.2) for distinctive patterns like "my go-to breakfast." However, both approaches introduced specific vulnerabilities leading to two common failure patterns. First, com- monsense reasoning (D.1) happened when agents attempted to apply the reasoning-basee approach encountered gaps they couldn't bridge, leading them substitute commonsense knowledge that seemed plausible but contradicted established personalized routines. Second, inaccurate recall (D.2) occured when agents recognized the need for personalized knowledge but retrieved imprecise or incomplete information from memory. The sequential nature of these approaches made them particularly error-prons as mistakes at any intermediate step propagated through subsequent actions. This vulnerability explains the consistently higher failure rates in patterns tasks across all models, highlighting fundamental challenges in maintaining coherence through multi-step reasoning over episodic memory.
24

---

Published as a conference paper at ICLR 2026
D.3 MEMORY DESIGN EXPERIMENT DETAILS
The following details clarify the episode sampling, memory construction, and retrieval configurations used in this experiment.
We describe here the experimental setup for Section 6
D.3.1 DETAILS OF MEMORY FORMAT SIMPLIFICATION EXPERIMENT
We sampled 30 episodes by selecting 10 episodes from each of three scenes, with careful balancing across task types and difficulty levels informed by our preliminary experimental results. To minimize the influence of erroneous trajectories, we provided GPP--4-geenerate gold memory as the given episodic memory, thereby reducing the impact of noisy interaction histories. We also used GPT-4o generate summaries of the action trajectories, including user instructions, facilitate compact memory representations. For each evaluation, one exemplar shot was given to the agent, and top-k memory entries were retrieved based on the current query, where k 10 for frontier models (GPT-4o and Qwen-2.5 (wwe-22.5-22B) and k = 7 for smaller models (LLaMA-3.J 1-BB and Qwen-2.5- 2.5--.,, considering context length limitations. The prompts used for these experiments are provided in Appendix F3
D.3.2 DETAILS OF USER PROFILE MEMORY EXPERIMENT
We conduct experiments by incorporating user profile memory in addition to episodic memory. The experiments are conducted with the same models as in Section 6.1 (GPT-40, Qwen-2.5-72B- Instruct, Llama-3.1 --0--...------ntteectt and Qwenn2.55-7BInnstruue to construct the user profile memory and to carry out the retrieving procedure, which combines embedding- search with LLM reasoning: the LLM parses user instructions and reformulates retrieved graph content into natural language descriptions. User profile memory entries are retrieved under
These models also employed both
setup without gold memory, while episodic memory supplied as each agent's trajectory under & = 5 setup with gold memory. We evaluate performance both the single and joint memory utilization stages, and the prompts for memory construction and retrieval are provided in Appendix
D.4 EXAMPLE TRAJECTORY AND EVALUATION
To provide readers with concrete understanding of our evaluation framework and typical agent performance patterns, we present detailed examples of both successful (Tables TO and failure
cases (Tables 12 14 These examples include the complete instruction metadata, agent execution trajectories, and evaluation outcomes, offering insights into when and why agents succeed or struggle with personalized knowledge utilization.
Table 9: Episode metadata for the example task.
Field
Value
episode_typs
user_pattern
subtype
preference
task
Could you decorate bedroom as like?
task percent tttplpeeeeoopppllllee
complete
1.0
task state success
1.0
25

---

Published as a conference paper at ICLR 2026
Table 10: Trajectory for the example task.
Thought
Action
Result State A
Since there objects found yet. I should by exploring the bedroom locate any objects that can be used for decoration.
Explore
No objects found
(bedroom_
2 Since no objects were found bedroom 1, should explore another room to locate objects that can be used for decoration. will explore the living room decorative items.
Navigate (to living_room__)
Location changed
it is likely to contain
3 have found plant container and statue the shelves in the living room, which can be used for decoration in the bedroom. will start by picking up the plant container from the shelves.
Pick (plant_containees
plant container held
4 Now that am close to the shelves, will attempt to pick the plant container to move it to the bedroom for decoration.
(continue)
Success
5 have successfully picked up the plant container. Now, I need to navigate to the bedroom to place it for decoration.
Navigate (to bedroom_
Arrived in bedroom_
6 have successfully navigated to bedroom_L Place (plant_containg on chest_of ofw.wffdddr__Saaa
Object placed
Now. need to find suitable location to place the plant container for decoration. will place on the chest of drawers in the bedroom.
7 Now that am close to chest of drawers 54, will place the plant container on it for decoration.
(continue)
Confirmed placement
8 have successfully placed the plant container on chest_of ____aarr__55 in bedroom_ Now, need return the living room to pick up the statue from the shelves for further decoration in the bedroom.
Navigate (to living_room_
Location changed
back the living room. I need navigate closer to the shelves to pick up the statue for further decoration in the bedroom.
Positioning pick
10 Now that am close to the shelves, will attempt to pick the statue to move it to the bedroom for decoration.
Pick (statue 0))
statue held
11 have successfully picked the statue. Now, need navigate the bedroom to place it for decoration.
Navigate
Location changed
(to bedroom_
12 have successfully navigated to bedroom Now, need find suitable location to place the statue for decoration. will place it on the chest of drawers next to the plant container.
Positioning
13 Now that am close to chest of drawers 54, will place the statue next to the plant container for decoration.
Place (statue_ 0 on chest_on _f drw_ddrwwe__S. _54 next_ plant_ catttt_oootinr___
Object placed
26

---

Published as a conference paper at ICLR 2026
Table 11: Evaluation functions for the example task.
Function
Args
Description
j__on_top
object_ handles=[Cole Hardware_arwaeeeeeeeeeer Harttaree_lletric____ Pntectrc_PPo_______ PotttPottaaa_eeee Aos_AAssottmettS. Barpemtt__500000000000 $5_=0000), receptacle handles=[bedroom chest of drawers handles). number=1
Plant container is drawer
is_on_top top Seeee_aanndcsssDoSSaaeee Statue_4_00000
Statue is drawer
receptacle handles=[bedroom chest of drawers handles], number=1
next to entity handles a=(Cole Hardware Electric Pot Assortment s0peent055_0000000000
Plant statue are next to each other
entity handhes_nddaaaeeee b_[[o_[[D[aaaaaaaa Statue 1aciue___0000 4_:0000) number=1, 12_ thr
Table 12: Episode metadata for the example task.
Field
Value
episode_type
user_pattern
subtype
routine
task
Could you set the candle for cozy evening?
task_ pet_pppeeeeetpppeleee
0
task state success
0
Table 13: Trajectory for the example task.
# Thought
Action
Result / State A
Since there are no objects found yet, should start by exploring the tv room to locate candle that can be set up for cozy evening.
Navigate
Location changed
(to tv
2 have found a candle, candle 0, on table 34 in the
Pick (candle -0)
candle 0 held
To it up for cozy evening. will need to pick it up from the table.
3 Now that am close table 33, will attempt to pick up the candle, candle 0, to up for cozy evening.
(continue)
Success
have successfully picked up the candle, can- dle_0. To set it up for a cozy evening, I will place suitable surface in the tv room, such the table, table 14.
Place
Object placed
(candle_0 0 on lable_14)
5 am now close to table_14 14 in the room. will place the candle, candle 0, this table set up cozy evening.
(continue)
Failed to meet task con- straints
27

---

Published as a conference paper at ICLR 2026
Table 14: Evaluation functions for the example task.
Function
Args
Description
is_on_top object_handdeeeee handles=[candlee, receptacle hhaieaeeesncdiiinahaaaae number=1
Candle must be on chair
2. is_on_top (adee__hnndles(ccnnleee recettale_handless((bbbe number=1
Candle must be table (after chair)
Constraint TemporalConstraint: Placing the candle on the chair before placing it
the table.
D.5 LLMS LONG CONTEXT COMPREHENSION ANALYSIS
To investigate whether the information overload bottleneck in LLM LLM--ooered embodied agents for personalized assistance stems from limitations in processing long-contexi information, we conduct an experiment QA format to evaluate whether LLMs can properly reference personalized information from long contexts.
Experiment setup. We provide LLMs with the episodic memory information required for personal- ized assistance tasks along with the corresponding task instructions. We then prompted the models: "Identify the personalized knowledge necessary perform this instruction based on the context provided below." To evaluate the responses, we use GPT-4o as an LLM-as-a-Judge, supplying it with the stored gold memory and task instruction as references assess whether the LLM's identified personalized knowledge aligns with that in the gold memory.
Table 15: Accuracy on long-contex. QA with varying numbers of retrieved episodic memories (Top-k).
Model
=
5
20
GPT-4o
100%
98.71%
98.28%
96.98%
96.98%
95.69%
Qwen-2.5-72b
100%
93.97%
94.40%
93.53%
93.10%
83.19%
Results. As shown in the Table 15 all models demonstrate high accuracy in correctly identifying personalized knowledge in the QA format. Compared the results in Section [5. Figure 6 this indicates that current LLMs have the ability to reference personalized information from long contexts, but encounter difficulties when applying this personalized information to planning tasks.
E EXTENDED ANALYSIS: MEMORY COMPONENTS AND REAL-WORLD WORLD SCENARIOS
To provide comprehensive insights into memory utilization for personalized assistance, we conduct extended analysis examining multiple factors affecting agent performance. This section covers three primary areas: memory quality effects on agent performance, retrieval strategy effectiveness for episodic memory, and agent behavior in complex real-world scenarios including 3-joint memory tasks and ambiguous instructions.
E. HUMAN BASELINE
Experiment setup. To establish a meaningful comparison point for our agent evaluation and validate that tasks are fundamentally solvable, conduct human baseline study. We recruit four undergraduate participants to complete a curated set 20 tasks that represent the challenges our embodied agents face in memory utilization for personalized assistance.
To ensure comprehensive coverage of task complexity, we categorize task difficulty based on empirical observations of model behavior across agent evaluations:
Easy: Llama-8B successfully completed the task using its memory
28

---

Published as a conference paper at ICLR 2026
Medium: Llama-8B &BB failed with its memory but succeeded when given GPT-4o's answer memory
Hard: Llama-8B failed even when given GPT-4o'ss answer memory
Super hard: GPT-4o failed with its own memory
Random joint-memory tasks
We implement a text-based interactive interface that mirrors the agent environment while accommodat- ing human interaction patterns. Participants issue step-by-step commands and receive observational feedback after each action, similar how our agents operate. Each participant receives 10-minute tutorial on the interface and completes tasks in the same environment as the agents. Crucially, participants have access to the same top-5 retrieved memories that agents receive, allowing direct comparison of memory utilization capabilities.
Results. The human baseline provides both validation and perspective our evaluation framework. All participants successfully completed 100% of the tasks with correct end states, demonstrating that the challenges are fundamentally solvable given appropriate memory access and reasoning capabilities. While humans require more time than -.MM-d planners, they exhibit superior abilities in referencing episodic memory, recognizing errors, and adapting their actions accordingly. This performance establishes meaningful upper-bound reference for agent capabilities and confirms that our tasks capture genuine challenges in memory utilization for personalized assistance.
E.2 MEMORY QUALITY AND ROBUSTNESS ANALYSIS
To better understand the factors affecting memory utilization performance, we conduct additional analysis examining two critical aspects of memory-based personalized assistance. First, we analyze how memory quality impacts agent performance by comparing high quality memory trajectories with acquired memory trajectories from actual agent interactions. Second, we evaluate agent robustness when episodic memories are corrupted or incomplete through various noise and degradation condi- tions. These analyses provide insights into the practical limitations and requirements for deploying personalized embodied agents in real-world scenarios where memory quality cannot be guaranteed.
E.2.1 MEMORY QUALITY ANALYSIS
Experiment setup. We further analyze the effect of memory quality by comparing high quality memory, consisting of successful and shortest-path trajectories that serves as curated references with the memory obtained from interaction histories in the memory acquisition stage, with predicted memory obtained via automated retrieval. By evaluating performance differences between these two settings, we aim understand how memory quality affects agent performance across the memory utilization stage for both tasks (single-mema tasks, joint-memory tasks).
Performance degradation with lower-quality trajectories. Table 16 shows the performance comparison between high quality memory and acquired memory across the memory utilization stage. In the memory utilization stage, high-capacity models show relatively stable performance across gold and retrieved memory, while lower-capacity models exhibit a substantial drop when using retrieved memory. This suggests that less capable models have more difficulty extracting relevant information from an imperfect memory context. When executing more demanding loint-memon tasks, which require combining and reasoning across multiple memory sources, performance degrades sharply across all models with acquired memory. This indicates that the compounded complexity of integrating multiple memories makes it increasingly important for the agent to rely on clear, noise- free, high-quality memory contexts from which synthesis can be performed reliably. Additionally, as shown in the table X, the overall reduction in planning steps when using high-quality memory suggests that episodic memory plays important role in the planning process of embodied agents. Enhancing memory quality through more precise filtering is therefore essential not only for improving reasoning performance but also for increasing the agent's planning efficiency.
29

---

Published as a conference paper at ICLR 2026
Table 16: Performance comparison of models when using high-quality memory versus acquired (self-acquired) memory, evaluated in both single- and joint-memory settings.
Model
Memory
Stage
Replanning Count
Sim Steps
PC (%)
SR (%)
qualitu
Single Memory
14.5
2481.2
80.6
Joint Memory
25.8
3619.4
83.3
63.9
GPT-4o
Single Memory
16.1
2450.8
Baseline
Joint Memory
28.9
3480.7
86.7
63.9
Single Memory
14.1
2266.4
70.1
High-quality
Joint Memory
24.2
3226.6
69.9
38.9
Claude-3.5 3.5-Sonnel
Single Memory
15.3
2258.8
63.7
Baseline
Joint Memory
27.8
3198.8
64.6
33.3
Single Memory
15.2
2630.4
68.2
High-quality
Qwen-2.5-72B
Joint Memory
26.4
4082.
64.8
38.9
Single Memory
17.5
2691.2
72.6
67.2
Baseline
Joint Memory
31.3
4027.1
68.9
36.1
Single Memory
15.1
2364.0
72.6
67.2
quahiquality
Joint Memory
27.1
3374.5
64.1
27.8
Llama-70B
Single Memory
19.0
2566.6
72.2
66.7
Baseline
Joint Memory
31.4
3425.2
51.3
8.3
Single Memory
16.9
2739.0
65.1
59.2
High-quality
Joint Memory
26.4
3753.2
54.0
13.9
Llama-88
Single Memory
19.0
3131.7
48.1
35.0
Baseline
Joint Memory
27.4
3478.2
35.3
8.3
Single Memory
15.7
2991.5
55.7
46.3
High-quality
Joint Memory
23.9
4402.0
34.0
8.3
Qwen-2.5-7B
Single Memory
21.8
3271.4
39.1
27.4
Joint Memory
26.9
4148.6
33.7
5.6
Table 17: Model performance across memory acquisition and utilization stages MEMENTO.
Memory Type
Planning Cycles |
Sim Steps |
Percent Complete 1
Success Rate t
Model
Memory
APC
ASR
Gold
Bascline
14.5
2481.2
85.3
80.6
Random Shuffle
14.6
2759.0
81.5
-3,8
78.6
-2.0
Noisy
14.4
2450.9
84.0
-1.3
80.6
0.0
GPT-4o
Random
14.5
2526.9
84.4
-0.9
80.1
-0.5
w/o Pick
14.6
2555.7
82.9
-2.4
79.1
-1.5
Partial
w/o Place
14.5
2569.8
84.4
-0.9
82.6
+2.0
w/o Nav
15.4
2526.8
81.4
-3.9
78.5
-2.1
E.2.2 ROBUSTNESS TO NOISY AND PARTIAL MEMORIES
Experiment setup. We conduct additional experiments evaluate the model's robustness when episodic memories are corrupted or incomplete. As the target model. We leverage GPT-4o, and we organize the experimental setup into two representative degradation types: Noisy Memory and Partial Memory. For Noisy Memory, we consider two conditions: (i) random step injection, where unrelated steps from other scenes (30%) are inserted into the memory trajectory to simulate retrieval or percep- tion noise, and (ii) temporal shuffle, where the step order within episode is randomly permuted to disrupt temporal coherence. For Partial Memory, we examine two forms of incompleteness: (i) random step removal, where 20% of steps were randomly removed to test agent robustness under incomplete recall, and (ii) cirtion-yye-sppeiffe removal, where all steps of a given action type (Place, Place, Navigate) dropped to analyze their impact planning.
Random noise and missing navigation cues degrade performance. As shown in Table 17 the model remains robust under most forms of incomplete memory, with only minor degradation. However, two factors consistently cause substantial performance drops: (i) the injection of random,
30

---

Published as a conference paper at ICLR 2026
noisy entries and (ii) the absence of navigation- related information. This indicates that the model is more sensitive to misleading or corrupted imputs than to simple omissions, and that navigation cues play disproportionately critical role in planning.
E.3 RETRIEVAL STRATEGY ANALYSIS
Effective retrieval of relevant episodic memories is fundamental to successful memory utilization in personalized assistance tasks. To understand how retrieval quality affects agent performance, analyze both baseline retrieval effectiveness and the impact of advanced retrieval strategies on downstream task outcomes. We first establish baseline missing rates using similarity-ban based retrieval determine optimal retrieval parameters, then examine whether advanced retrieval strategies consistently improve both memory retrieval quality and agent task performance. This analysis provides insights into the relationship between retrieval precision and practical task execution in personalized embodied agents.
E.3.1 BASELINE RETRIEVAL ANALYSIS.
To assess whether the corresponding episodic memory is successfully retrieved during memory utilization, first measure the missing rate under a baseline similarity-bant
Table 18: We use an embedding- baaeed retrieval method with the all-mpnet-based Sentence Transformer (Reimers Gurevych 2019).
retriever, The results show missing rates of 13.9% for top-3 and 3.5% for top-5, 5, indicating that expanding to top-5 nearly eliminates missing cases. Based on this observa- tion, we set top-5 as the default retrieval setup, prioritizing inclusion of the required memory over the risk of distractor entries.
Retriever Strategy
Top-K
Missing Rates (%)
3
13.9
Bascline
5
3.5
E.3.2 ADVANCED RETRIEVAL STRATEGY ANALYSIS
Table 19: Performance comparison across different retrieval strategies.
Missing Rates (%)
Percent Complete 1
Success
Model
Strategy
Top-K
Rate t
3
13.9
82.9 84.7
79.5
Baseline
5
3.5
82.1
GPT-4o
3
2.5
87.2
84.1
Reranker
5
1.5
84.3
80.6
3
9.0
84.9
81.1
Query Expansion
5
1.5
86.2
81.1
3
13.9
75.3
70.1
Baseline
5
3.5
75.4
70.0
Qwen-2.5-72b
3
73.0
67.2
Reranker
5
2.1
70.9
64.1
3
8.5
75.6
69.2
Query Expansion
5
4.0
72.6
66.2
Experiment setup. Building on this baseline, further examine the effect of advanced retrieval strategies on the instruction following capability. We compared three approaches: (i) the baseline similarity-based retreival,
(ii) Granker-based retrieval, which applies an LLM--baed pointwise reranker (Sachan et al. 2022; Sun et al. 2023) an initial pool of 10 candidates before selecting the final top-3 or top-5, and (iii) query expansion based retrieval, where queries are reformulated before retrieval (Jagerman et al.) 2023 Wang et al.) 2023b). These strategies are evaluated with both GPT-40 and Qwe22,.5-72b, using the same models to compute reranker scores and produce expanded queries.
We adopt embedding-baseed method using the all-mpnet-hase-12 Sentence Transformer (Reimers &
Gurevych 2019).
31

---

Published as a conference paper at ICLR 2026
Improved refrieval does not ensure improved outcomes. shown in Table retrieval strategies affect not only memory inclusion but also downstream task performance. First, applying reranker or query expansion substantially reduce the missing rate, particularly under the top-3 setting, where improvements range from 4.9pp to 11 1..4p.. This confirms their effectiveness narrowing the gap between top-3 and top-5 retrieval. Second, performance gains arere not consistent across models. For example, with Qwen, query expansion under the top-5 setting even degraded retrieval, suggesting that the effectiveness of query expansion critically depends on the quality of the model generating the reformulated queries. Finally, except for GPT-4o at top-3, introducing advanced retrieval strategies further reduces downstream performance, likely due the inclusion of semantically similar but task-irrelevant memories. These results suggest that simply increasing retrieval relevance does not necessarily yield better task performance, particularly in real-world contexts with dense and noisy memory pools.
E.4
REAL- WORLD SCENARIO ANALYSIS
While our main evaluation demonstrates fundamental limitations in memory utilization for personal- ized assistance, real-world deployment introduces additional complexities that extend beyond our controlled experimental settings. To explore these practical challenges, we conduct three sets of experiments that examine agent behavior under more realistic conditions. First, we evaluate scalability by testing agents 3-joint memory tasks that require integrating knowledge from multiple episodes. Second, we assess how agents handle ambiguous instructions that indirectly reference personalized knowledge through contextual cues rather than explicit mentions. Finally, we investigate agents' ability to generalize learned user preferences across different but related contexts, such as applying spatial arrangements from one room another. These experiments provide insights into additional hurdles for successful real-world deployment of personalized embodied agents.
E.4.1 REAL-WORd MULTI-EPISODE EVALUATION
In practice, user requirements are rarely limited to information from just one or two episodes, but often require knowledge distributed across multi- ple episodes. To explore this realistic scenario, we extend our joint-memory task cases involving three memories, which require agents to effec- tively integrate three independent memory sources to solve the task. Following the same methodology as our joint-memory task construction, we sequentially concatenated three personalized instructions to
Table 20: Performance results on multiple joint- memory tasks.
Model
Task
PC (9:)
SR (%)
GPT-4o
joint memory task
86.7
63.9
GPT-4o
3-joint memory task
61.2
22.0
Qwen-2.5-726
joint memory task
68.9
36.1
Qwen-2.5-726
3-joint memory task
42.4
7.3
generate total of 41 tasks. We observed substantial performance drops compared to the two-memon, This suggests that reasoning introduces significant difficulty due to temporal gaps and abstraction challenges. We believe this degradation reflects the true complexity of real-world personalized tasks, and addressing this challenge remains critical for future research.
joint- tasks for both GPT-4o and 1.sen-2.5-2b-innsi
E.4.2 AGENT BEHAVIOR UNDER AMBIGUOUS INSTRUCTIONS FROM USERS
Experiment setup. While MEMENTO focuses on evaluating personalized knowledge grounding with explicit references prior interactions, real- world human-agent communication often involves ambiguous or indirect references. To explore this challenge, we conducted a proof-of-concept ex- periment assess whether current models can interpret ambiguous instructions that indirectly re- fer to previously encountered personalized knowl- edge. We created tailored set of 30 tasks by sampling 10 episodes from each three scenes and modifying them to include indirect references to personalized knowledge through contextual cues, synonyms, or causal references (e.g., Can you set my afternoon tea time routine? - I I'm about enjoy my afternoon tea. Could you things
Table 21: Performance under ambiguous queries for personalized knowledge. PC (Percent Com- plete) and SR (Success Rate) (%) indicate how well agents resolve indirect references person- alized knowledge from memory.
Model
PC (%)
SR (%)
GPT-4o (Bascline) GPT-4o Qwen-2.5-72b (Baseline)
92.0
90.0
80.4
73.3
75.1
66.7
Qwen-2.5-72b
59.6
53.3
32

---

Published as a conference paper at ICLR 2026
up as I like them?). To ensure quality and mitigate potential bias, all episodes were jointly created and validated by two authors. For each model, we first generated memory traces during the memory acquisition stage and then evaluated memory usage during the utilization stage. Table provides representative examples of the ambiguous instruction pairs used. In each case, the user rephrases the original request refer indirectly to previously established personalized knowledge, simulating natural variability in real-world human-agent communication.
The results, shown in Table 21 reveal the degradation in performance, indicating that handling ambiguous queries remains a key challenge for future personalized embodied agents. We view this as promising direction for future work. where we plan to systematically extend MEMENTO to evaluate LLW-powered embodied agents' capabilities under ambiguous and implicit reference scenarios, aiming to better reflect the complexity of real-world human-agent interaction. Table 22: Examples of ambiguous instruction modifications in the additional experiment dataset.
Result.
Original Memory Utilization Instruction
Rephrased Ambiguous Instruction
Can you arrange my cozy reading setup on the dining table?
Can you set up the dining table as I prefer so can read the book comfortably?
Could you help me tidy up by moving my grad- uation gifted book, the candle with brown and blue gradient, and round white clock with black numbers to the stand?
Could you help me tidy up by moving my book received to celebrate completing my studies, the candle with a brown and blue gradient, and the round white clock with black numbers to the stand?
E.4.3 PERSONALIZED KNOWLEDGE GENERALIZATION
Experiment setup. A critical aspect of personalized embodied agent is their ability to generalize learned user preferences across different but related contexts. This includes applying knowledge about a user's breakfast routine to dinner preparation tasks, or transferring spatial preferences from one room another. To investigate this capability, we designed generalization tasks that test whether embodied agents can abstract and reuse personalized knowledge as transferable templates. We extended user pattern tasks by sampling existing scenarios and systematically altering target locations, thereby evaluating whether agents can adapt user preferences to novel but structurally similar situations. To encourage explicit memory utilization, we incorporated contextual cues in the instructions, such as "on [another receptacle] this time," clearly signaling that established preferences should apply in the modified context. Our generalization tasks were constructed using relatively simple base cases that GPT-4o consistently solves in standard settings.
Result. As in Table 23 we observed that GPT- 4o occasionally fails by defaulting to common- sense reasoning rather than retrieving and adapting the relevant personalized memory. Interestingly, Qwen-2.5-72b demonstrated improved comple- tion rates on these tasks. We attribute this im- provement to the explicit instruction cues, which likely reduced ambiguity and guided the model toward semorry-bsse reasoning. While these preliminary experiments provide initial insights into
Table 23: Agent performance on personalized knowledge generalization tasks.
Model
Task
PC (%)
SR (%)
GPT-40 GPT-4o
Baseline
100.0
100.0
Generalization
92.5
83.3
Qwen-2.5-72h
Baseline
75.0
75.0
Qwen-22.--26b
Generalization
85.8
75.0
generalization capabilities, they represent an important step toward understanding how agents can flexibly apply personalized knowledge. Future work could build these initial observations to better understand how agents transfer personalized knowledge across different contexts.
E.4.4 USER PROFILE MEMORY ROBUSTNESS
Experiment setup. In real-world interactions, users often refer to knowledge, objects, or locations using different names expressions rather than clean and consistent symbolic identifiers. To directly evaluate the robustness of our proposed User Profile Memory under realistic conditions, we additionally conduct qualitative analysis how well User Profile Memory handles natural noise that arises in real user interactions. We identified two noise types that naturally arise when users interact with personalized embodied agent:
33

---

Published as a conference paper at ICLR 2026
Knowledge Update Noise the user expresses existing knowledge using different phrase (synonyms, paraphrases, alias names).
Node Reference Noise new knowledge refers to an existing object location using a different form.
For each category, three undergraduate annotators modified the original instructions grounded in the scene of MEMENTO, producing 10 noisy instructions for each noise type. This procedure ensures the noise realistically reflects how humans describe object semantics or user patterns. These noisy instructions were crafted using the underlying scene (103997895 1710311822 and ensure an accurate evaluation of noise robustness, all experiments were conducted on graphs constructed from the original instructions. Full datasets are provided in Table 2alee24add and Table
Table 24: Dataset of Knowledge Update Noise Variants.
Original Knowledge
Modified Knowledge
Gift teapot from grandfather Family heirloom vase
Grandfather's gifted teakettle
Inherited family vase
Gift toy airplane from childhood friend Favorite relaxation candle
A toy airplane that childhood friend gave me
The candle always use relax Android collectible figurine
Collectible android figure
Bedroom organization
Organizing sctup for the bedroom space
Morning work session setup
Morning workspace arrangement
Cozy room setup
Arrangement for creating a cozy room atmosphere Nighttime setup before going to bed Organized placement of study materials
Bedtime setup
Study materials organization
Table 25: Dataset of Node Reference Noise Variants.
Original Description
Modified Description
Black teapot with curved spout and nate
curved-spout black teapot featuring an intricately decorated lid A gray laptop equipped with a black key beydoard and touchpad
Gray laptop
with black key-
board/touchpad
Dark green analog alarm clock with twin bells
Dark green twin-bell analog alarm clock
Black and orange keychain with circular pendant
A circular pendant keychain in black and orange
android figure which is a black and white android panda figure
black and white Android panda figurine
Move the toy airplane from the bedroom the living room table. Place it next the couch.
Relocate the toy plane from the bedroom onto the table in the living room, positioning next the sofa.
Move the picture frame and clock to the table in the bedroom. Then, place them next each other on the counter in the kitchen.
Bring the photo frame and the alarm clock to the bedroom table, and afterward arrange both items side by side on the countertop at kitchen.
Place the bread, the potato, and the egg the kitchen table and place them next each other.
Set the loaf, spud, and fresh egg together on the ta- ble at kitchen, arranging them close to another.
Move the laptop and the mouse pad to a new living room table. Move the clock another shelf.
Transfer the notebook computer and mouse mat onto the new table in the living room, and place the clock on separate shelf.
Take the book, pencil case, and multiport hub from the table in the living room and put them the shelves in the bedroom, placing them next each other.
Gather the note, pencil pouch, and usb hub from the living room table and arrange them side by side the bedroom shelves.
34

---

Published as a conference paper at ICLR 2026
knowledge update/add decisions are handled reliably across all samples. These findings indicate that method handles natural linguistic noise reasonably well. However, node reference shows a few missed cases (5 out of 20). Although these missed cases amount to 25%, the model still correctly handles 75% of the reference variations. The specific failure are summarized in Table 27 These failures indicate that reducing unnecessary node duplication remains important challenge that requires further refinement. We believe this direction is important because mitigating node duplication directly improves memory consistency, which is key factor influencing downstream task performance personalized embodied agents.
Result. As shown in Table 26
Table 26: Accuracy under different noise types.
Noise Type
Accuracy (%)
Knowledge Update Noise Node Reference Noise
75.0
Table 27: Failure cases in node reference noise
Existing Node
Add Node
mouse pad clock android figure which black and white android panda figure kitchen counter another shelf
mouse mat
alarm clock
black white Android Panda Figurine countertop at kitchen
separate shelf
35

---

Published as a conference paper at ICLR 2026
F PROMPTS
F.1 PROMPTS FOR DATASET GENERATION
Object Semantics Instruction Generation Prompt
Your task is to generate a user instruction that includes object semantics for an embodied agent that can perform rearrangement tasks.
The instruction should be grounded in personalized object-level semantics based on the original instruction and object descriptions.
The object semantics can be categorized into types: ownership: Indicates that the user personally owns has special claim on an object.
Indicates the user's specific preferences related to object (e.g., placement, condition).
preference:
history: Reflects the user's past interaction or meaningful history with the object.
group: Defines a logical or personalized grouping of multiple objects (e.g., "my coffee set" for mug + saucer + spoon).
You should generate 2 types of instructions and object semantics: Stage Instruction for memorization; The instruction should include the original instruction, descriptions of all the objects, and explicit object semantics of the relevant objects. This will be used to store memory.
Stage 2: Instruction for utilization; The instruction should require the agent to understand and use the previously stored object semantics. should sound natural to humans and be difficult for an agent without access to memory. Keep it short and situated. Relevant objects must be referred to only using their stored semantics, without any descriptive attributes. For all other objects, refer to them using visual or descriptive attributes.
Object Semantics: This is the semantic information associated with each object used in the instruction. Only include the most relevant one object semantics based on the instruction context. Not all target objects need be included, but use as many of them as possible.
Note that if the original instruction involves sequence of object interactions, that order should be preserved in the Stage 2 instruction.
The output format should be as follows:
36

---

Published as a conference paper at ICLR 2026
[Example]
### Input
oriiinall instruction: <original instruction
<list of the objects with short descriptions>
### Output
Stage 1: <instruction> + <object descriptions> <object semantics>
-Sagge 2: <instruction with object semantics formed in a natural way>
-ssed Object: <List about the used objects' categories>
Objeec Semantics: <Object semantics category about the relevant objects>
[Example 1] (shot_ exampeeeamles
### Input
original instruction: finstruction]
handle info: (handle_intl
User Pattern Instruction Generation Prompt
user _eer_patter:: |-
Your task is to generate user instruction that includes user pattern for embodied agent that can perform rearrangment tasks.
The instruction should be related to personalized knowledge based on the original instruction.
The user pattern can be categorized into 2 types:
ppeeeeence: specific way the user prefers prepare or arrange their environment when a particular situation occurs.
routine: A sequence or setup the user follows as habit or regular activity.
You should generate 2 types of instructions, memory, and user pattern: Stage 1: Instruction for memorization; The instruction should be original instruction + user pattern. You should explicitly state the user's preference routine in the instruction.
Stage 2: Instruction for utilization; The instruction should be only about user's preference or routine that human would naturally use in the situated environment. You should make the instruction difficult for the agent without using memory and try to make it short.
37

---

Published as a conference paper at ICLR 2026
-ser pattern: The user pattern should be the user's preference or routine that can be reused for future rearrangement tasks.
Note that if the original instruction requires a sequence of actions, the order of the actions should be followed for the stage 2 instruction.
The output format should be as follows: ### Input
<original instruction>
### Output
Stage 1: <original instruction> + <user pattern>
Stage 2: <user pattern formed in natural way>
-eeoory: <Memory about user's preference or routine>
User pattern: <user pattern>
[Example 1]
Schot_examples)
### Input
{org instruction)
Captioning Prompt for OVMM Objects
captioning:
Generate short, but precise caption for the given object. Focus only on the object, ignoring the background. Include its type, primary colors, and any distinctive features.
Examples:
schot_examples
Image:
Category:
{category)
Image:
(image)
38

---

Published as a conference paper at ICLR 2026
Captioning Prompt for Captioning Google Objects
captioning google: >
Generate short, but precise caption for the given object. Focus only on the object, ignoring the background. Include its type, primary colors, and any distinctive features.
If you can't recognize the object, refer the name of the objects gave.
Examples:
{shot_ )
Image:
Category:
{category)
Name:
(name}
Image:
(image)
39

---

Published as a conference paper at ICLR 2026
F.2
PROMPTS FOR AGENT
Zero Shot Agent ReAct Prompt
prompt: --
fsystem (astte__tgg/eee You are an agent that solves embodied-agent ageet planning problems. The task assigned to you will be situated in house and will generally involve navigating to objects, picking and placing them on different receptacles to achieve rearrangement, You strictly follow any format specifications and pay attention the previous actions taken in order to avoid repeating mistakes.
If there are multiple tasks to complete, please follow them in the order they appear in the instruction.
Rooms do not need to be explored more than once. This means if you have explored the living room and have not found the object, then you should explore the kitchen, if relevant object still not found, you should explore the hallway etc...
Many calls to the same action in a row are a sign that something has gone wrong and you should try different action.(eot_tag.
(input}
dyvorldddescription
Possible Actions:
desoiiessci)ttionss
Done: Used to indicate that the agent has finished the task. Example (Donel))
What is the next action to make progress towards completing the
task?
Return your response in the following format
Thought: <reasoning for why you are taking the next action <next action call>
Assigned!
40

---

Published as a conference paper at ICLR 2026
Here is an example:
Thought: Since there are no objects found I should explore a room I have not explored yet. Explore[<room name>] Assigned!
(eot_tag) Beot_tagYasssitantt.tag
41

---

Published as a conference paper at ICLR 2026
F.3 PROMPTS FOR MEMORY DESIGN EXPERIMENT
Summary for Section 6.1 Prompt
summary: |-
You are helpful assistant designed to summarize episodic task execution traces of an embodied agent.
You will be given a full trace of the agent's actions, thoughts, and results as attempts to follow a human instruction.
Please output compact memory paragraph including: Instruction: Copy exactly the instruction from the trace. This is the sentence just before the first Thought appears. (Try to understand user's intention well.)
Plaa:: Briefly summarize the key high-level steps the agent performed.
Guidelines:
Use 2 to 3 short sentences.
Do not list low-level micro actions.
Ignore repeated failures unless they affected the outcome.
Do not invent any details not present in the trace.
Use past tense and third-person style.
[Example 1]
Input:
pypputttaac_exxmplle
Output:
poutput_exapple
[Example]
Input:
(input_trace) trace)
Output:
42

---

Published as a conference paper at ICLR 2026
Summary for Graph Reformulation
extract knowledge: I-
Analyze the following graph and interpret as natural language.
FOCUS on the user's personalized knowledge and what it is and how we can use it understand the
Provide only simple text description without any formatting JSON. Also provide only the output with any additional explanation. For example:
[Example 1]
Graph:
{f
"knowledge_type" _ttpe
wwwplledde_tppe_exaappa
"nodes": (nodes example),
"edges": (edges_est [eammpee
}}
Output: Oouppurdesscrpptioneeaapppa
Graph: {graph}
Output:
43

---

Published as a conference paper at ICLR 2026
Summary for Knowledge Classifier
classifier: |-
You are given an instruction.
Classify the type of personalized knowledge in the instruction. Return ONLY the knowledge type, nothing else.
The knowledge type should be either "object or Luser_pattern".
Abject_semanticn Personalized knowledge that the user assigns specific objects or groups of objects.
-Examples: gift, heirloom, favorite item, part of set, belongs to someone, has special meaning
oser_pattern: Personalized knowledge that the user assigns to pattern sequences or behavioral routines.
Exxmmpes: prefer to keep, often do, like to, setup for later, routine, habit, pattern of behavior
Rules:
If the instruction only assigns personal meaning to an object
"object_ serentics
If the instruction describes habit, preference, or routine "user_pattern"
[Example 1]
Instruction: finstruction iintruction_example
Output: (output_ wwwwleddee_tyee_eexaaeee
Instruction: {instruction]
Output:
44

---

Published as a conference paper at ICLR 2026
Summary for Graph Encoder
encoder: graph_encoder: |-
You are given an instruction and its knowledge type.
Generate a semantic graph JSON representing the personalized knowledge.
Do not output plain text additional explanations.
Knowledge Type: (knowledge_uee _yppee_______________
object_ semantics: Personalized knowledge that the user assigns specific objects or groups of objects.
user petter:: Personalized knowledge that the user assigns to pattern sequences or behavioral routines.
For Pattern nodes) For user lueeuuueetteeeeeee Location nodes
Create Knowledge node Object nodes (NO
patrrprtttertttett Create Knowledge node + Pattern nodes + -bbject/
Follow this format:
{{
"knowledge_type" "(knowledge_type""
"nodes": [...]
"edges": [...] }1)
45

---

Published as a conference paper at ICLR 2026
### Schema
Node Types
User
Knowledge
Pattern
Object
Location
Edge Types
Hierarchical Edge
User - Knowledge (relation: owns)
-Knolledge - Pattern (relation: entails)
Pattern Object (relation: target)
Pattern - Location (relation: target)
Knowledge Object (relation: alias of target_object object)
Knowledge Location (relation: target_location)
Temporal Edge
Pattern, Pattern (relation: before)
### Examples
[Example 1]
Instruction: finstruction examppl
Knowledge Type: shoowledg_ttpp_eexmlee
Output: ggraaph_xamppe
Instruction: (instruction)
Knowledge Type: (knowledge_typee
Output:
46


Reviews:
[{"review_id": "dGPdu0Hx4M", "reviewer": "Reviewer_KG3h", "summary": "This paper investigates the challenges LLM-powered embodied agents face in performing personalized assistance tasks, which require leveraging user-specific knowledge from past interactions (i.e., episodic memory). The authors introduce MEMENTO, a novel two-stage evaluation framework designed to quantify an agent's memory utilization capabilities. This framework assesses performance along two key dimensions: 'object semantics' (recalling personal meaning of objects) and 'user patterns' (recalling behavioral routines). Through experiments with a suite of modern LLMs (e.g., GPT-4o, Llama-3.1) , the authors find that while agents can recall simple object semantics, they significantly struggle to apply sequential user patterns. They identify two critical bottlenecks: 'information overload' from large retrieved memory sets and 'coordination failures' when handling multiple memories. Based on these findings, the paper proposes a hierarchical knowledge graph-based user-profile memory module that separates structured personalized knowledge from raw episodic memory, demonstrating substantial performance improvements.", "questions": "- What if object semantics and user patterns were stored as structured text templates rather than knowledge graphs? Would this achieve similar benefits with lower complexity?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The two-stage design (memory acquisition vs. utilization) provides a principled way to isolate memory effects from other capabilities, with identical goals but varying instructions across stages, enabling precise measurement of memory utilization through metrics.\n\n- The dataset construction process is rigorous with multiple quality controls.\n\n- The experimental analysis is comprehensive with insightful qualitative analysis", "weaknesses": "- The paper introduces the personalized object rearrangement task as a Partially Observable Markov Decision Process in Section 3.1. The actual agent implementation, however, is a hierarchical controller that uses an LLM as a high-level policy planner in a ReAct-style prompting format. The POMDP formalism and its associated equations feel disconnected from the practical, prompt-based system that is actually built and evaluated. The formalism is not explicitly used to derive the agent architecture or learning algorithm (as it's a zero-shot planner). This creates a minor mismatch between the stated mathematical grounding and the implementation.\n\n- The simplification of the full embodied agent problem means the episodic memory ($h_{acq}$) stored is \"clean.\" It's unclear how performance would be affected if the memory traces themselves were noisy due to perception errors. The proposed KG solution, which relies on clean, symbolic entity names, would also be challenging to implement without this oracle. \n\n- Missing analysis of how memory grows over time (all experiments use fixed 1-2 reference episodes)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LBs5tlYxaz", "reviewer": "Reviewer_f2sq", "summary": "The paper investigates how LLM-powered embodied agents handle personalization via memory, focusing on two dimensions: recognizing objects with user-specific semantics and recalling sequences of user behavioral patterns. The authors propose an evaluation framework named MEMENTO, including both single-memory and joint-memory tasks, and find that while agents can use simple object semantic memories, they struggle to incorporate sequential memory of user routines into planning. A deeper analysis identifies two main bottlenecks: information overload when too many memories are retrieved, and coordination failures when multiple memories must be integrated. To address these, the paper explores memory architectures and proposes a hierarchical knowledge-graph-based user-profile memory module that manages personalized knowledge separately.", "questions": "- For the joint memory, how many personalizations can be jointly used for some decent performance? Is there any quantitative analysis of the number of required personalizations with the corresponding performance of the downstream tasks?\n- Can the analysis be extended to scenarios where personalization indicators (e.g., \"That's my favoriate ...\" or \"It's my ...\") are not given? Addressing personalized information obtained from a set of videos might be a more practical scenario.\n- In Figure 6, some models improve when taking more retrieved samples. Why is this the case? As the current personalization addresses the mapping from an expression to a target object or a set of user patterns, where a single sample might be sufficient for their description.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper provides a systematic analysis of the memory utilization of LLM-powered agents for personalized scenarios, including multiple personal preferences.\n- Underperformance even with the top-k most relevant information for a specific personalization seems interesting.\n- The quantitative comparisons are conducted through diverse open- and closed-sourced LLMs.", "weaknesses": "- It is unclear why the problem addressed in this paper is specifically related to embodied agents. Given that the issues (Sec. 3) and underperformances (Sec. 4 and 5) come from the incapabilities of LLMs, it looks more related to LLMs than embodied agents.\n- As it is explicitly given how some objects and routines are to be described, the problem addressed in this paper is rather related to using the context, which should not necessarily have to be personalization, provided well. In this sense, the term \"personalization\" seems misleading.\n- For the result in Figure 4, the authors argue that the current embodied agents struggle to comprehend user patterns, but I'm not fully convinced because the performance drops may come from the longer horizon of the user patterns than the one of the object semantics. This needs to be distinguished. In addition, as there is no clear performance gaps in object semantics, it is unclear if LLMs indeed suffers from using personalized information.\n  - The result in Figure 3 is somewhat related to this point for original/acquisition to single/joint memory utilization.\n- The evaluation metrics used in this paper seem to be inappropriate. The problem that the authors tackle in this paper is about how well LLMs can use personalized information given in the context, but the metrics used are about how well they complete the entire tasks, potentially including aspects beyond personalized memory utilization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qjcw8e1qbE", "reviewer": "Reviewer_t5nh", "summary": "The paper provides an empirical analysis of memory utilization in embodied agents by evaluating them on the task of room rearrangement. It involves a memory acquisition stage and then a memory utilization stage. The model is evaluated along two dimensions of memory utilization: the ability to recall object semantics, and the ability to recall a user’s behavioral patterns. The work presents various controlled analyses to isolate the effect of memory on downstream performance, and highlights that current LLM-powered embodied agents can be good at retrieving information about object semantics, but struggle with aggregating multi-episodic memory to understand behavior patterns. Finally, they propose a graph-like structure to summarize user pattern behavior, and show that it improves performance across single-memory and multi-memory tasks, highlighting the need for better memory structure to enable personalized embodied agents.", "questions": "- What happens if gold memory is not included? Can it work from noisy memory, could be useful analysis to know.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The work addresses an interesting problem of personalization of embodied agents. Eventually, household agents would more likely than not exist around a user, and would require specializing to their needs. Memory is an interesting and useful component; the analysis done in this work can be useful guidance for future work. \n- The paper is very well written and makes it easy to follow and understand the results and the analysis presented. \n- I liked the idea of the graph-based hierarchical memory design to embed personal user behaviors; it seems to show clear improvement, at least in the current setup.", "weaknesses": "- In Section 5.1, an analysis is conducted, which ablates the value of k, which is the number of retrieved memories. It seems to show that as the value increases how the performance of various agents decreases, credit to information overload. \n  - However, consider this: the overload is also indicative of the LLM’s lack of a longer context, which, once it goes beyond a certain number of memories, is unable to grasp and summarize this information, and hence resorts to general semantic knowledge, instead of personalized one.\n  - If the above were possible, perhaps a better way of measuring information overload could be to provide a more concise version of relevant information, and only increase the amount of direct clues about the user’s personal behavior, without increasing the information in-context too much.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)