MiraData: A Large-Scale Video Dataset with Long Durations and Structured
  Captions
Authors: name: Xuan Ju; name: Yiming Gao; name: Zhaoyang Zhang; name: Ziyang Yuan; name: Xintao Wang; name: Ailing Zeng; name: Yu Xiong; name: Qiang Xu; name: Ying Shan
Published: 2024-07-08 19:58:59

Abstract: Sora's high-motion intensity and long consistent videos have significantly
impacted the field of video generation, attracting unprecedented attention.
However, existing publicly available datasets are inadequate for generating
Sora-like videos, as they mainly contain short videos with low motion intensity
and brief captions. To address these issues, we propose MiraData, a
high-quality video dataset that surpasses previous ones in video duration,
caption detail, motion strength, and visual quality. We curate MiraData from
diverse, manually selected sources and meticulously process the data to obtain
semantically consistent clips. GPT-4V is employed to annotate structured
captions, providing detailed descriptions from four different perspectives
along with a summarized dense caption. To better assess temporal consistency
and motion intensity in video generation, we introduce MiraBench, which
enhances existing benchmarks by adding 3D consistency and tracking-based motion
strength metrics. MiraBench includes 150 evaluation prompts and 17 metrics
covering temporal consistency, motion strength, 3D consistency, visual quality,
text-video alignment, and distribution similarity. To demonstrate the utility
and effectiveness of MiraData, we conduct experiments using our DiT-based video
generation model, MiraDiT. The experimental results on MiraBench demonstrate
the superiority of MiraData, especially in motion strength.