Motion Control for Enhanced Complex Action Video Generation
Authors: name: Qiang Zhou; name: Shaofeng Zhang; name: Nianzu Yang; name: Ye Qian; name: Hao Li
Published: 2024-11-13 04:20:45

Abstract: Existing text-to-video (T2V) models often struggle with generating videos
with sufficiently pronounced or complex actions. A key limitation lies in the
text prompt's inability to precisely convey intricate motion details. To
address this, we propose a novel framework, MVideo, designed to produce
long-duration videos with precise, fluid actions. MVideo overcomes the
limitations of text prompts by incorporating mask sequences as an additional
motion condition input, providing a clearer, more accurate representation of
intended actions. Leveraging foundational vision models such as GroundingDINO
and SAM2, MVideo automatically generates mask sequences, enhancing both
efficiency and robustness. Our results demonstrate that, after training, MVideo
effectively aligns text prompts with motion conditions to produce videos that
simultaneously meet both criteria. This dual control mechanism allows for more
dynamic video generation by enabling alterations to either the text prompt or
motion condition independently, or both in tandem. Furthermore, MVideo supports
motion condition editing and composition, facilitating the generation of videos
with more complex actions. MVideo thus advances T2V motion generation, setting
a strong benchmark for improved action depiction in current video diffusion
models. Our project page is available at https://mvideo-v1.github.io/.