GATE OpenING: A Comprehensive Benchmark for Judging Open-ended
  Interleaved Image-Text Generation
Authors: name: Pengfei Zhou; name: Xiaopeng Peng; name: Jiajun Song; name: Chuanhao Li; name: Zhaopan Xu; name: Yue Yang; name: Ziyao Guo; name: Hao Zhang; name: Yuqi Lin; name: Yefei He; name: Lirui Zhao; name: Shuo Liu; name: Tianhua Li; name: Yuxuan Xie; name: Xiaojun Chang; name: Yu Qiao; name: Wenqi Shao; name: Kaipeng Zhang
Published: 2024-11-27 16:39:04

Abstract: Multimodal Large Language Models (MLLMs) have made significant strides in
visual understanding and generation tasks. However, generating interleaved
image-text content remains a challenge, which requires integrated multimodal
understanding and generation abilities. While the progress in unified models
offers new solutions, existing benchmarks are insufficient for evaluating these
methods due to data size and diversity limitations. To bridge this gap, we
introduce GATE OpenING (OpenING), a comprehensive benchmark comprising 5,400
high-quality human-annotated instances across 56 real-world tasks. OpenING
covers diverse daily scenarios such as travel guide, design, and brainstorming,
offering a robust platform for challenging interleaved generation methods. In
addition, we present IntJudge, a judge model for evaluating open-ended
multimodal generation methods. Trained with a novel data pipeline, our IntJudge
achieves an agreement rate of 82. 42% with human judgments, outperforming
GPT-based evaluators by 11.34%. Extensive experiments on OpenING reveal that
current interleaved generation methods still have substantial room for
improvement. Key findings on interleaved image-text generation are further
presented to guide the development of next-generation models. The OpenING is
open-sourced at https://opening.github.io.