Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting
  Region Captions
Authors: name: Yu-Guan Hsieh; name: Cheng-Yu Hsieh; name: Shih-Ying Yeh; name: Louis Béthune; name: Hadi Pour Ansari; name: Pavan Kumar Anasosalu Vasu; name: Chun-Liang Li; name: Ranjay Krishna; name: Oncel Tuzel; name: Marco Cuturi
Published: 2024-07-09 09:55:04

Abstract: Humans describe complex scenes with compositionality, using simple text
descriptions enriched with links and relationships. While vision-language
research has aimed to develop models with compositional understanding
capabilities, this is not reflected yet in existing datasets which, for the
most part, still use plain text to describe images. In this work, we propose a
new annotation strategy, graph-based captioning (GBC) that describes an image
using a labelled graph structure, with nodes of various types. The nodes in GBC
are created using, in a first stage, object detection and dense captioning
tools nested recursively to uncover and describe entity nodes, further linked
together in a second stage by highlighting, using new types of nodes,
compositions and relations among entities. Since all GBC nodes hold plain text
descriptions, GBC retains the flexibility found in natural language, but can
also encode hierarchical information in its edges. We demonstrate that GBC can
be produced automatically, using off-the-shelf multimodal LLMs and
open-vocabulary detection models, by building a new dataset, GBC10M, gathering
GBC annotations for about 10M images of the CC12M dataset. We use GBC10M to
showcase the wealth of node captions uncovered by GBC, as measured with CLIP
training. We show that using GBC nodes' annotations -- notably those stored in
composition and relation nodes -- results in significant performance boost on
downstream models when compared to other dataset formats. To further explore
the opportunities provided by GBC, we also propose a new attention mechanism
that can leverage the entire GBC graph, with encouraging experimental results
that show the extra benefits of incorporating the graph structure. Our datasets
are released at https://huggingface.co/graph-based-captions.