DanceTogether! Identity-Preserving Multi-Person Interactive Video
  Generation
Authors: name: Junhao Chen; name: Mingjin Chen; name: Jianjin Xu; name: Xiang Li; name: Junting Dong; name: Mingze Sun; name: Puhua Jiang; name: Hongxiang Li; name: Yuhang Yang; name: Hao Zhao; name: Xiaoxiao Long; name: Ruqi Huang
Published: 2025-05-23 16:37:14

Abstract: Controllable video generation (CVG) has advanced rapidly, yet current systems
falter when more than one actor must move, interact, and exchange positions
under noisy control signals. We address this gap with DanceTogether, the first
end-to-end diffusion framework that turns a single reference image plus
independent pose-mask streams into long, photorealistic videos while strictly
preserving every identity. A novel MaskPoseAdapter binds "who" and "how" at
every denoising step by fusing robust tracking masks with semantically rich-but
noisy-pose heat-maps, eliminating the identity drift and appearance bleeding
that plague frame-wise pipelines. To train and evaluate at scale, we introduce
(i) PairFS-4K, 26 hours of dual-skater footage with 7,000+ distinct IDs, (ii)
HumanRob-300, a one-hour humanoid-robot interaction set for rapid cross-domain
transfer, and (iii) TogetherVideoBench, a three-track benchmark centered on the
DanceTogEval-100 test suite covering dance, boxing, wrestling, yoga, and figure
skating. On TogetherVideoBench, DanceTogether outperforms the prior arts by a
significant margin. Moreover, we show that a one-hour fine-tune yields
convincing human-robot videos, underscoring broad generalization to embodied-AI
and HRI tasks. Extensive ablations confirm that persistent identity-action
binding is critical to these gains. Together, our model, datasets, and
benchmark lift CVG from single-subject choreography to compositionally
controllable, multi-actor interaction, opening new avenues for digital
production, simulation, and embodied intelligence. Our video demos and code are
available at https://DanceTog.github.io/.