ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of
  Text-to-Time-lapse Video Generation
Authors: name: Shenghai Yuan; name: Jinfa Huang; name: Yongqi Xu; name: Yaoyang Liu; name: Shaofeng Zhang; name: Yujun Shi; name: Ruijie Zhu; name: Xinhua Cheng; name: Jiebo Luo; name: Li Yuan
Published: 2024-06-26 17:50:47

Abstract: We propose a novel text-to-video (T2V) generation benchmark,
ChronoMagic-Bench, to evaluate the temporal and metamorphic capabilities of the
T2V models (e.g. Sora and Lumiere) in time-lapse video generation. In contrast
to existing benchmarks that focus on the visual quality and textual relevance
of generated videos, ChronoMagic-Bench focuses on the model's ability to
generate time-lapse videos with significant metamorphic amplitude and temporal
coherence. The benchmark probes T2V models for their physics, biology, and
chemistry capabilities, in a free-form text query. For these purposes,
ChronoMagic-Bench introduces 1,649 prompts and real-world videos as references,
categorized into four major types of time-lapse videos: biological,
human-created, meteorological, and physical phenomena, which are further
divided into 75 subcategories. This categorization comprehensively evaluates
the model's capacity to handle diverse and complex transformations. To
accurately align human preference with the benchmark, we introduce two new
automatic metrics, MTScore and CHScore, to evaluate the videos' metamorphic
attributes and temporal coherence. MTScore measures the metamorphic amplitude,
reflecting the degree of change over time, while CHScore assesses the temporal
coherence, ensuring the generated videos maintain logical progression and
continuity. Based on the ChronoMagic-Bench, we conduct comprehensive manual
evaluations of ten representative T2V models, revealing their strengths and
weaknesses across different categories of prompts, and providing a thorough
evaluation framework that addresses current gaps in video generation research.
Moreover, we create a large-scale ChronoMagic-Pro dataset, containing 460k
high-quality pairs of 720p time-lapse videos and detailed captions ensuring
high physical pertinence and large metamorphic amplitude.