DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal
  Perception
Authors: name: Xiaotong Li; name: Fan Zhang; name: Haiwen Diao; name: Yueze Wang; name: Xinlong Wang; name: Ling-Yu Duan
Published: 2024-07-11 08:48:06

Abstract: Existing Multimodal Large Language Models (MLLMs) increasingly emphasize
complex understanding of various visual elements, including multiple objects,
text information, and spatial relations. Their development for comprehensive
visual perception hinges on the availability of high-quality image-text
datasets that offer diverse visual elements and throughout image descriptions.
However, the scarcity of such hyper-detailed datasets currently hinders
progress within the MLLM community. The bottleneck stems from the limited
perceptual capabilities of current caption engines, which fall short in
providing complete and accurate annotations. To facilitate the cutting-edge
research of MLLMs on comprehensive vision perception, we thereby propose
Perceptual Fusion, using a low-budget but highly effective caption engine for
complete and accurate image descriptions. Specifically, Perceptual Fusion
integrates diverse perception experts as image priors to provide explicit
information on visual elements and adopts an efficient MLLM as a centric pivot
to mimic advanced MLLMs' perception abilities. We carefully select 1M highly
representative images from uncurated LAION dataset and generate dense
descriptions using our engine, dubbed DenseFusion-1M. Extensive experiments
validate that our engine outperforms its counterparts, where the resulting
dataset significantly improves the perception and cognition abilities of
existing MLLMs across diverse vision-language benchmarks, especially with
high-resolution images as inputs. The dataset and code are publicly available
at https://github.com/baaivision/DenseFusion.