Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in
  Long-Horizon Tasks
Authors: name: Zaijing Li; name: Yuquan Xie; name: Rui Shao; name: Gongwei Chen; name: Dongmei Jiang; name: Liqiang Nie
Published: 2024-08-07 08:16:32

Abstract: Building a general-purpose agent is a long-standing vision in the field of
artificial intelligence. Existing agents have made remarkable progress in many
domains, yet they still struggle to complete long-horizon tasks in an open
world. We attribute this to the lack of necessary world knowledge and
multimodal experience that can guide agents through a variety of long-horizon
tasks. In this paper, we propose a Hybrid Multimodal Memory module to address
the above challenges. It 1) transforms knowledge into Hierarchical Directed
Knowledge Graph that allows agents to explicitly represent and learn world
knowledge, and 2) summarises historical information into Abstracted Multimodal
Experience Pool that provide agents with rich references for in-context
learning. On top of the Hybrid Multimodal Memory module, a multimodal agent,
Optimus-1, is constructed with dedicated Knowledge-guided Planner and
Experience-Driven Reflector, contributing to a better planning and reflection
in the face of long-horizon tasks in Minecraft. Extensive experimental results
show that Optimus-1 significantly outperforms all existing agents on
challenging long-horizon task benchmarks, and exhibits near human-level
performance on many tasks. In addition, we introduce various Multimodal Large
Language Models (MLLMs) as the backbone of Optimus-1. Experimental results show
that Optimus-1 exhibits strong generalization with the help of the Hybrid
Multimodal Memory module, outperforming the GPT-4V baseline on many tasks.