LEGENT: Open Platform for Embodied Agents
Authors: name: Zhili Cheng; name: Zhitong Wang; name: Jinyi Hu; name: Shengding Hu; name: An Liu; name: Yuge Tu; name: Pengkai Li; name: Lei Shi; name: Zhiyuan Liu; name: Maosong Sun
Published: 2024-04-28 16:50:12

Abstract: Despite advancements in Large Language Models (LLMs) and Large Multimodal
Models (LMMs), their integration into language-grounded, human-like embodied
agents remains incomplete, hindering complex real-life task performance in
physical environments. Existing integrations often feature limited open
sourcing, challenging collective progress in this field. We introduce LEGENT,
an open, scalable platform for developing embodied agents using LLMs and LMMs.
LEGENT offers a dual approach: a rich, interactive 3D environment with
communicable and actionable agents, paired with a user-friendly interface, and
a sophisticated data generation pipeline utilizing advanced algorithms to
exploit supervision from simulated worlds at scale. In our experiments, an
embryonic vision-language-action model trained on LEGENT-generated data
surpasses GPT-4V in embodied tasks, showcasing promising generalization
capabilities.