LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
Authors: name: Shilong Liu, name: Hao Cheng, name: Haotian Liu, name: Hao Zhang, name: Feng Li, name: Tianhe Ren, name: Xueyan Zou, name: Jianwei Yang, name: Hang Su, name: Jun Zhu, name: Lei Zhang, name: Jianfeng Gao, name: Chunyuan Li

LLaVA-Plus is a general-purpose multimodal assistant that expands the
capabilities of large multimodal models. It maintains a skill repository of
pre-trained vision and vision-language models and can activate relevant tools
based on users' inputs to fulfill real-world tasks. LLaVA-Plus is trained on
multimodal instruction-following data to acquire the ability to use tools,
covering visual understanding, generation, external knowledge retrieval, and
compositions. Empirical results show that LLaVA-Plus outperforms LLaVA in
existing capabilities and exhibits new ones. It is distinct in that the image
query is directly grounded and actively engaged throughout the entire human-AI
interaction sessions, significantly improving tool use performance and enabling
new scenarios.