Exploring Multi-Grained Concept Annotations for Multimodal Large
  Language Models
Authors: name: Xiao Xu; name: Tianhao Niu; name: Yuxi Xie; name: Libo Qin; name: Wanxiang Che; name: Min-Yen Kan
Published: 2024-12-08 13:45:44

Abstract: Multimodal Large Language Models (MLLMs) excel in vision--language tasks by
pre-training solely on coarse-grained concept annotations (e.g., image
captions). We hypothesize that integrating fine-grained concept annotations
(e.g., object labels and object regions) will further improve performance, as
both data granularities complement each other in terms of breadth and depth in
concept representation. We introduce a new dataset featuring Multimodal
Multi-Grained Concept annotations (MMGiC) for MLLMs. In constructing MMGiC, we
explore the impact of different data recipes on multimodal comprehension and
generation. Our analyses reveal that multi-grained concept annotations
integrate and complement each other, under our structured template and a
general MLLM framework. We clearly explore and demonstrate the potential of
MMGiC to help MLLMs better locate and learn concepts, aligning vision and
language at multiple granularities. We further validate our hypothesis by
investigating the fair comparison and effective collaboration between MMGiC and
image--caption data on 12 multimodal comprehension and generation benchmarks,
e.g., their appropriate combination achieve 3.95% and 2.34% absolute
improvements over image--caption data alone on POPE and SEED-Bench. Code, data
and models will be available at https://github.com/LooperXX/MMGiC.