DisCo: Disentangled Control for Referring Human Dance Generation in Real
  World
Authors: name: Tan Wang; name: Linjie Li; name: Kevin Lin; name: Chung-Ching Lin; name: Zhengyuan Yang; name: Hanwang Zhang; name: Zicheng Liu; name: Lijuan Wang
Published: 2023-06-30 17:37:48

Abstract: Generative AI has made significant strides in computer vision, particularly
in image/video synthesis conditioned on text descriptions. Despite the
advancements, it remains challenging especially in the generation of
human-centric content such as dance synthesis. Existing dance synthesis methods
struggle with the gap between synthesized content and real-world dance
scenarios. In this paper, we define a new problem setting: Referring Human
Dance Generation, which focuses on real-world dance scenarios with three
important properties: (i) Faithfulness: the synthesis should retain the
appearance of both human subject foreground and background from the reference
image, and precisely follow the target pose; (ii) Generalizability: the model
should generalize to unseen human subjects, backgrounds, and poses; (iii)
Compositionality: it should allow for composition of seen/unseen subjects,
backgrounds, and poses from different sources. To address these challenges, we
introduce a novel approach, DISCO, which includes a novel model architecture
with disentangled control to improve the faithfulness and compositionality of
dance synthesis, and an effective human attribute pre-training for better
generalizability to unseen humans. Extensive qualitative and quantitative
results demonstrate that DISCO can generate high-quality human dance images and
videos with diverse appearances and flexible motions. Code, demo, video and
visualization are available at: https://disco-dance.github.io/.