LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
Authors: name: Shilong Liu; name: Hao Cheng; name: Haotian Liu; name: Hao Zhang; name: Feng Li; name: Tianhe Ren; name: Xueyan Zou; name: Jianwei Yang; name: Hang Su; name: Jun Zhu; name: Lei Zhang; name: Jianfeng Gao; name: Chunyuan Li
Published: 2023-11-09 15:22:26

Abstract: LLaVA-Plus is a general-purpose multimodal assistant that expands the
capabilities of large multimodal models. It maintains a skill repository of
pre-trained vision and vision-language models and can activate relevant tools
based on users' inputs to fulfill real-world tasks. LLaVA-Plus is trained on
multimodal instruction-following data to acquire the ability to use tools,
covering visual understanding, generation, external knowledge retrieval, and
compositions. Empirical results show that LLaVA-Plus outperforms LLaVA in
existing capabilities and exhibits new ones. It is distinct in that the image
query is directly grounded and actively engaged throughout the entire human-AI
interaction sessions, significantly improving tool use performance and enabling
new scenarios.