Interleaved Scene Graph for Interleaved Text-and-Image Generation
  Assessment
Authors: name: Dongping Chen; name: Ruoxi Chen; name: Shu Pu; name: Zhaoyi Liu; name: Yanru Wu; name: Caixi Chen; name: Benlin Liu; name: Yue Huang; name: Yao Wan; name: Pan Zhou; name: Ranjay Krishna
Published: 2024-11-26 07:55:57

Abstract: Many real-world user queries (e.g. "How do to make egg fried rice?") could
benefit from systems capable of generating responses with both textual steps
with accompanying images, similar to a cookbook. Models designed to generate
interleaved text and images face challenges in ensuring consistency within and
across these modalities. To address these challenges, we present ISG, a
comprehensive evaluation framework for interleaved text-and-image generation.
ISG leverages a scene graph structure to capture relationships between text and
image blocks, evaluating responses on four levels of granularity: holistic,
structural, block-level, and image-specific. This multi-tiered evaluation
allows for a nuanced assessment of consistency, coherence, and accuracy, and
provides interpretable question-answer feedback. In conjunction with ISG, we
introduce a benchmark, ISG-Bench, encompassing 1,150 samples across 8
categories and 21 subcategories. This benchmark dataset includes complex
language-vision dependencies and golden answers to evaluate models effectively
on vision-centric tasks such as style transfer, a challenging area for current
models. Using ISG-Bench, we demonstrate that recent unified vision-language
models perform poorly on generating interleaved content. While compositional
approaches that combine separate language and image models show a 111%
improvement over unified models at the holistic level, their performance
remains suboptimal at both block and image levels. To facilitate future work,
we develop ISG-Agent, a baseline agent employing a "plan-execute-refine"
pipeline to invoke tools, achieving a 122% performance improvement.