VBench++: Comprehensive and Versatile Benchmark Suite for Video
  Generative Models
Authors: name: Ziqi Huang; name: Fan Zhang; name: Xiaojie Xu; name: Yinan He; name: Jiashuo Yu; name: Ziyue Dong; name: Qianli Ma; name: Nattapol Chanpaisit; name: Chenyang Si; name: Yuming Jiang; name: Yaohui Wang; name: Xinyuan Chen; name: Ying-Cong Chen; name: Limin Wang; name: Dahua Lin; name: Yu Qiao; name: Ziwei Liu
Published: 2024-11-20 17:54:41

Abstract: Video generation has witnessed significant advancements, yet evaluating these
models remains a challenge. A comprehensive evaluation benchmark for video
generation is indispensable for two reasons: 1) Existing metrics do not fully
align with human perceptions; 2) An ideal evaluation system should provide
insights to inform future developments of video generation. To this end, we
present VBench, a comprehensive benchmark suite that dissects "video generation
quality" into specific, hierarchical, and disentangled dimensions, each with
tailored prompts and evaluation methods. VBench has several appealing
properties: 1) Comprehensive Dimensions: VBench comprises 16 dimensions in
video generation (e.g., subject identity inconsistency, motion smoothness,
temporal flickering, and spatial relationship, etc). The evaluation metrics
with fine-grained levels reveal individual models' strengths and weaknesses. 2)
Human Alignment: We also provide a dataset of human preference annotations to
validate our benchmarks' alignment with human perception, for each evaluation
dimension respectively. 3) Valuable Insights: We look into current models'
ability across various evaluation dimensions, and various content types. We
also investigate the gaps between video and image generation models. 4)
Versatile Benchmarking: VBench++ supports evaluating text-to-video and
image-to-video. We introduce a high-quality Image Suite with an adaptive aspect
ratio to enable fair evaluations across different image-to-video generation
settings. Beyond assessing technical quality, VBench++ evaluates the
trustworthiness of video generative models, providing a more holistic view of
model performance. 5) Full Open-Sourcing: We fully open-source VBench++ and
continually add new video generation models to our leaderboard to drive forward
the field of video generation.