EvalCrafter: Benchmarking and Evaluating Large Video Generation Models
Authors: name: Yaofang Liu; name: Xiaodong Cun; name: Xuebo Liu; name: Xintao Wang; name: Yong Zhang; name: Haoxin Chen; name: Yang Liu; name: Tieyong Zeng; name: Raymond Chan; name: Ying Shan
Published: 2023-10-17 17:50:46

Abstract: The vision and language generative models have been overgrown in recent
years. For video generation, various open-sourced models and public-available
services are released for generating high-visual quality videos. However, these
methods often use a few academic metrics, for example, FVD or IS, to evaluate
the performance. We argue that it is hard to judge the large conditional
generative models from the simple metrics since these models are often trained
on very large datasets with multi-aspect abilities. Thus, we propose a new
framework and pipeline to exhaustively evaluate the performance of the
generated videos. To achieve this, we first conduct a new prompt list for
text-to-video generation by analyzing the real-world prompt list with the help
of the large language model. Then, we evaluate the state-of-the-art video
generative models on our carefully designed benchmarks, in terms of visual
qualities, content qualities, motion qualities, and text-caption alignment with
around 18 objective metrics. To obtain the final leaderboard of the models, we
also fit a series of coefficients to align the objective metrics to the users'
opinions. Based on the proposed opinion alignment method, our final score shows
a higher correlation than simply averaging the metrics, showing the
effectiveness of the proposed evaluation method.