MMCOMPOSITION: Revisiting the Compositionality of Pre-trained
  Vision-Language Models
Authors: name: Hang Hua, name: Yunlong Tang, name: Ziyun Zeng, name: Liangliang Cao, name: Zhengyuan Yang, name: Hangfeng He, name: Chenliang Xu, name: Jiebo Luo

The advent of large Vision-Language Models (VLMs) has significantly advanced
multimodal understanding, enabling more sophisticated and accurate integration
of visual and textual information across various tasks, including image and
video captioning, visual question answering, and cross-modal retrieval. Despite
VLMs' superior capabilities, researchers lack a comprehensive understanding of
their compositionality -- the ability to understand and produce novel
combinations of known visual and textual components. Prior benchmarks provide
only a relatively rough compositionality evaluation from the perspectives of
objects, relations, and attributes while neglecting deeper reasoning about
object interactions, counting, and complex compositions. However,
compositionality is a critical ability that facilitates coherent reasoning and
understanding across modalities for VLMs. To address this limitation, we
propose MMCOMPOSITION, a novel human-annotated benchmark for comprehensively
and accurately evaluating VLMs' compositionality. Our proposed benchmark serves
as a complement to these earlier works. With MMCOMPOSITION, we can quantify and
explore the compositionality of the mainstream VLMs. Surprisingly, we find
GPT-4o's compositionality inferior to the best open-source model, and we
analyze the underlying reasons. Our experimental analysis reveals the
limitations of VLMs in fine-grained compositional perception and reasoning, and
points to areas for improvement in VLM design and training. Resources available
at: https://hanghuacs.github.io/MMComposition/