Move-in-2D: 2D-Conditioned Human Motion Generation
Authors: name: Hsin-Ping Huang; name: Yang Zhou; name: Jui-Hsien Wang; name: Difan Liu; name: Feng Liu; name: Ming-Hsuan Yang; name: Zhan Xu
Published: 2024-12-17 18:58:07

Abstract: Generating realistic human videos remains a challenging task, with the most
effective methods currently relying on a human motion sequence as a control
signal. Existing approaches often use existing motion extracted from other
videos, which restricts applications to specific motion types and global scene
matching. We propose Move-in-2D, a novel approach to generate human motion
sequences conditioned on a scene image, allowing for diverse motion that adapts
to different scenes. Our approach utilizes a diffusion model that accepts both
a scene image and text prompt as inputs, producing a motion sequence tailored
to the scene. To train this model, we collect a large-scale video dataset
featuring single-human activities, annotating each video with the corresponding
human motion as the target output. Experiments demonstrate that our method
effectively predicts human motion that aligns with the scene image after
projection. Furthermore, we show that the generated motion sequence improves
human motion quality in video synthesis tasks.