DisPose: Disentangling Pose Guidance for Controllable Human Image
  Animation
Authors: name: Hongxiang Li; name: Yaowei Li; name: Yuhang Yang; name: Junjie Cao; name: Zhihong Zhu; name: Xuxin Cheng; name: Long Chen
Published: 2024-12-12 15:15:59

Abstract: Controllable human image animation aims to generate videos from reference
images using driving videos. Due to the limited control signals provided by
sparse guidance (e.g., skeleton pose), recent works have attempted to introduce
additional dense conditions (e.g., depth map) to ensure motion alignment.
However, such strict dense guidance impairs the quality of the generated video
when the body shape of the reference character differs significantly from that
of the driving video. In this paper, we present DisPose to mine more
generalizable and effective control signals without additional dense input,
which disentangles the sparse skeleton pose in human image animation into
motion field guidance and keypoint correspondence. Specifically, we generate a
dense motion field from a sparse motion field and the reference image, which
provides region-level dense guidance while maintaining the generalization of
the sparse pose control. We also extract diffusion features corresponding to
pose keypoints from the reference image, and then these point features are
transferred to the target pose to provide distinct identity information. To
seamlessly integrate into existing models, we propose a plug-and-play hybrid
ControlNet that improves the quality and consistency of generated videos while
freezing the existing model parameters. Extensive qualitative and quantitative
experiments demonstrate the superiority of DisPose compared to current methods.
Code:
https://github.com/lihxxx/DisPose{https://github.com/lihxxx/DisPose}.