EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video
  Generation
Authors: name: Xiaofeng Wang; name: Kang Zhao; name: Feng Liu; name: Jiayu Wang; name: Guosheng Zhao; name: Xiaoyi Bao; name: Zheng Zhu; name: Yingya Zhang; name: Xingang Wang
Published: 2024-11-13 07:05:40

Abstract: Video generation has emerged as a promising tool for world simulation,
leveraging visual data to replicate real-world environments. Within this
context, egocentric video generation, which centers on the human perspective,
holds significant potential for enhancing applications in virtual reality,
augmented reality, and gaming. However, the generation of egocentric videos
presents substantial challenges due to the dynamic nature of egocentric
viewpoints, the intricate diversity of actions, and the complex variety of
scenes encountered. Existing datasets are inadequate for addressing these
challenges effectively. To bridge this gap, we present EgoVid-5M, the first
high-quality dataset specifically curated for egocentric video generation.
EgoVid-5M encompasses 5 million egocentric video clips and is enriched with
detailed action annotations, including fine-grained kinematic control and
high-level textual descriptions. To ensure the integrity and usability of the
dataset, we implement a sophisticated data cleaning pipeline designed to
maintain frame consistency, action coherence, and motion smoothness under
egocentric conditions. Furthermore, we introduce EgoDreamer, which is capable
of generating egocentric videos driven simultaneously by action descriptions
and kinematic control signals. The EgoVid-5M dataset, associated action
annotations, and all data cleansing metadata will be released for the
advancement of research in egocentric video generation.