Embodied Agents Meet Personalization: Exploring Memory Utilization for
  Personalized Assistance
Authors: name: Taeyoon Kwon, name: Dongwook Choi, name: Sunghwan Kim, name: Hyojun Kim, name: Seungjun Moon, name: Beong-woo Kwak, name: Kuan-Hao Huang, name: Jinyoung Yeo

Embodied agents empowered by large language models (LLMs) have shown strong
performance in household object rearrangement tasks. However, these tasks
primarily focus on single-turn interactions with simplified instructions, which
do not truly reflect the challenges of providing meaningful assistance to
users. To provide personalized assistance, embodied agents must understand the
unique semantics that users assign to the physical world (e.g., favorite cup,
breakfast routine) by leveraging prior interaction history to interpret
dynamic, real-world instructions. Yet, the effectiveness of embodied agents in
utilizing memory for personalized assistance remains largely underexplored. To
address this gap, we present MEMENTO, a personalized embodied agent evaluation
framework designed to comprehensively assess memory utilization capabilities to
provide personalized assistance. Our framework consists of a two-stage memory
evaluation process design that enables quantifying the impact of memory
utilization on task performance. This process enables the evaluation of agents'
understanding of personalized knowledge in object rearrangement tasks by
focusing on its role in goal interpretation: (1) the ability to identify target
objects based on personal meaning (object semantics), and (2) the ability to
infer object-location configurations from consistent user patterns, such as
routines (user patterns). Our experiments across various LLMs reveal
significant limitations in memory utilization, with even frontier models like
GPT-4o experiencing a 30.5% performance drop when required to reference
multiple memories, particularly in tasks involving user patterns. These
findings, along with our detailed analyses and case studies, provide valuable
insights for future research in developing more effective personalized embodied
agents. Project website: https://connoriginal.github.io/MEMENTO