LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models
Authors: name: Hao Zhang; name: Hongyang Li; name: Feng Li; name: Tianhe Ren; name: Xueyan Zou; name: Shilong Liu; name: Shijia Huang; name: Jianfeng Gao; name: Lei Zhang; name: Chunyuan Li; name: Jianwei Yang
Published: 2023-12-05 18:29:31

Abstract: With the recent significant advancements in large multi-modal models (LMMs),
the importance of their grounding capability in visual chat is increasingly
recognized. Despite recent efforts to enable LMMs to support grounding, their
capabilities for grounding and chat are usually separate, and their chat
performance drops dramatically when asked to ground. The problem is the lack of
a dataset for grounded visual chat (GVC). Existing grounding datasets only
contain short captions. To address this issue, we have created GVC data that
allows for the combination of grounding and chat capabilities. To better
evaluate the GVC capabilities, we have introduced a benchmark called
Grounding-Bench. Additionally, we have proposed a model design that can support
GVC and various types of visual prompts by connecting segmentation models with
language models. Experimental results demonstrate that our model outperforms
other LMMs on Grounding-Bench. Furthermore, our model achieves competitive
performance on classic grounding benchmarks like RefCOCO/+/g and Flickr30K
Entities. Our code will be released at
https://github.com/UX-Decoder/LLaVA-Grounding .