FINECAPTION: Compositional Image Captioning Focusing on Wherever You
  Want at Any Granularity
Authors: name: Hang Hua; name: Qing Liu; name: Lingzhi Zhang; name: Jing Shi; name: Zhifei Zhang; name: Yilin Wang; name: Jianming Zhang; name: Jiebo Luo
Published: 2024-11-23 02:20:32

Abstract: The advent of large Vision-Language Models (VLMs) has significantly advanced
multimodal tasks, enabling more sophisticated and accurate reasoning across
various applications, including image and video captioning, visual question
answering, and cross-modal retrieval. Despite their superior capabilities, VLMs
struggle with fine-grained image regional composition information perception.
Specifically, they have difficulty accurately aligning the segmentation masks
with the corresponding semantics and precisely describing the compositional
aspects of the referred regions.
  However, compositionality - the ability to understand and generate novel
combinations of known visual and textual components - is critical for
facilitating coherent reasoning and understanding across modalities by VLMs. To
address this issue, we propose FINECAPTION, a novel VLM that can recognize
arbitrary masks as referential inputs and process high-resolution images for
compositional image captioning at different granularity levels. To support this
endeavor, we introduce COMPOSITIONCAP, a new dataset for multi-grained region
compositional image captioning, which introduces the task of compositional
attribute-aware regional image captioning.
  Empirical results demonstrate the effectiveness of our proposed model
compared to other state-of-the-art VLMs. Additionally, we analyze the
capabilities of current VLMs in recognizing various visual prompts for
compositional region image captioning, highlighting areas for improvement in
VLM design and training.