T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video
  Generation
Authors: name: Kaiyue Sun; name: Kaiyi Huang; name: Xian Liu; name: Yue Wu; name: Zihan Xu; name: Zhenguo Li; name: Xihui Liu
Published: 2024-07-19 17:58:36

Abstract: Text-to-video (T2V) generation models have advanced significantly, yet their
ability to compose different objects, attributes, actions, and motions into a
video remains unexplored. Previous text-to-video benchmarks also neglect this
important ability for evaluation. In this work, we conduct the first systematic
study on compositional text-to-video generation. We propose T2V-CompBench, the
first benchmark tailored for compositional text-to-video generation.
T2V-CompBench encompasses diverse aspects of compositionality, including
consistent attribute binding, dynamic attribute binding, spatial relationships,
motion binding, action binding, object interactions, and generative numeracy.
We further carefully design evaluation metrics of MLLM-based metrics,
detection-based metrics, and tracking-based metrics, which can better reflect
the compositional text-to-video generation quality of seven proposed categories
with 700 text prompts. The effectiveness of the proposed metrics is verified by
correlation with human evaluations. We also benchmark various text-to-video
generative models and conduct in-depth analysis across different models and
different compositional categories. We find that compositional text-to-video
generation is highly challenging for current models, and we hope that our
attempt will shed light on future research in this direction.