VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video
  Generation
Authors: name: Wenhao Wang, name: Yi Yang

Text-to-video generative models convert textual prompts into dynamic visual
content, offering wide-ranging applications in film production, gaming, and
education. However, their real-world performance often falls short of user
expectations. One key reason is that these models have not been trained on
videos related to some topics users want to create. In this paper, we propose
VideoUFO, the first Video dataset specifically curated to align with Users'
FOcus in real-world scenarios. Beyond this, our VideoUFO also features: (1)
minimal (0.29%) overlap with existing video datasets, and (2) videos
searched exclusively via YouTube's official API under the Creative Commons
license. These two attributes provide future researchers with greater freedom
to broaden their training sources. The VideoUFO comprises over 1.09 million
video clips, each paired with both a brief and a detailed caption
(description). Specifically, through clustering, we first identify 1,291
user-focused topics from the million-scale real text-to-video prompt dataset,
VidProM. Then, we use these topics to retrieve videos from YouTube, split the
retrieved videos into clips, and generate both brief and detailed captions for
each clip. After verifying the clips with specified topics, we are left with
about 1.09 million video clips. Our experiments reveal that (1) current 16
text-to-video models do not achieve consistent performance across all
user-focused topics; and (2) a simple model trained on VideoUFO outperforms
others on worst-performing topics. The dataset is publicly available at
https://huggingface.co/datasets/WenhaoWang/VideoUFO under the CC BY 4.0
License.