VBench++: Comprehensive and Versatile Benchmark Suite for Video
  Generative Models
Authors: name: Ziqi Huang, name: Fan Zhang, name: Xiaojie Xu, name: Yinan He, name: Jiashuo Yu, name: Ziyue Dong, name: Qianli Ma, name: Nattapol Chanpaisit, name: Chenyang Si, name: Yuming Jiang, name: Yaohui Wang, name: Xinyuan Chen, name: Ying-Cong Chen, name: Limin Wang, name: Dahua Lin, name: Yu Qiao, name: Ziwei Liu

Video generation has witnessed significant advancements, yet evaluating these
models remains a challenge. A comprehensive evaluation benchmark for video
generation is indispensable for two reasons: 1) Existing metrics do not fully
align with human perceptions; 2) An ideal evaluation system should provide
insights to inform future developments of video generation. To this end, we
present VBench, a comprehensive benchmark suite that dissects "video generation
quality" into specific, hierarchical, and disentangled dimensions, each with
tailored prompts and evaluation methods. VBench has several appealing
properties: 1) Comprehensive Dimensions: VBench comprises 16 dimensions in
video generation (e.g., subject identity inconsistency, motion smoothness,
temporal flickering, and spatial relationship, etc). The evaluation metrics
with fine-grained levels reveal individual models' strengths and weaknesses. 2)
Human Alignment: We also provide a dataset of human preference annotations to
validate our benchmarks' alignment with human perception, for each evaluation
dimension respectively. 3) Valuable Insights: We look into current models'
ability across various evaluation dimensions, and various content types. We
also investigate the gaps between video and image generation models. 4)
Versatile Benchmarking: VBench++ supports evaluating text-to-video and
image-to-video. We introduce a high-quality Image Suite with an adaptive aspect
ratio to enable fair evaluations across different image-to-video generation
settings. Beyond assessing technical quality, VBench++ evaluates the
trustworthiness of video generative models, providing a more holistic view of
model performance. 5) Full Open-Sourcing: We fully open-source VBench++ and
continually add new video generation models to our leaderboard to drive forward
the field of video generation.