Ella: Embodied Social Agents with Lifelong Memory
Authors: name: Hongxin Zhang, name: Zheyuan Zhang, name: Zeyuan Wang, name: Zunzhe Zhang, name: Lixing Fang, name: Qinhong Zhou, name: Chuang Gan

We introduce Ella, an embodied social agent capable of lifelong learning
within a community in a 3D open world, where agents accumulate experiences and
acquire knowledge through everyday visual observations and social interactions.
At the core of Ella's capabilities is a structured, long-term multimodal memory
system that stores, updates, and retrieves information effectively. It consists
of a name-centric semantic memory for organizing acquired knowledge and a
spatiotemporal episodic memory for capturing multimodal experiences. By
integrating this lifelong memory system with foundation models, Ella retrieves
relevant information for decision-making, plans daily activities, builds social
relationships, and evolves autonomously while coexisting with other intelligent
beings in the open world. We conduct capability-oriented evaluations in a
dynamic 3D open world where 15 agents engage in social activities for days and
are assessed with a suite of unseen controlled evaluations. Experimental
results show that Ella can influence, lead, and cooperate with other agents
well to achieve goals, showcasing its ability to learn effectively through
observation and social interaction. Our findings highlight the transformative
potential of combining structured memory systems with foundation models for
advancing embodied intelligence. More videos can be found at
https://umass-embodied-agi.github.io/Ella/.