HuMo: Human-Centric Video Generation via Collaborative Multi-Modal
  Conditioning
Authors: name: Liyang Chen, name: Tianxiang Ma, name: Jiawei Liu, name: Bingchuan Li, name: Zhuowei Chen, name: Lijie Liu, name: Xu He, name: Gen Li, name: Qian He, name: Zhiyong Wu

Human-Centric Video Generation (HCVG) methods seek to synthesize human videos
from multimodal inputs, including text, image, and audio. Existing methods
struggle to effectively coordinate these heterogeneous modalities due to two
challenges: the scarcity of training data with paired triplet conditions and
the difficulty of collaborating the sub-tasks of subject preservation and
audio-visual sync with multimodal inputs. In this work, we present HuMo, a
unified HCVG framework for collaborative multimodal control. For the first
challenge, we construct a high-quality dataset with diverse and paired text,
reference images, and audio. For the second challenge, we propose a two-stage
progressive multimodal training paradigm with task-specific strategies. For the
subject preservation task, to maintain the prompt following and visual
generation abilities of the foundation model, we adopt the minimal-invasive
image injection strategy. For the audio-visual sync task, besides the commonly
adopted audio cross-attention layer, we propose a focus-by-predicting strategy
that implicitly guides the model to associate audio with facial regions. For
joint learning of controllabilities across multimodal inputs, building on
previously acquired capabilities, we progressively incorporate the audio-visual
sync task. During inference, for flexible and fine-grained multimodal control,
we design a time-adaptive Classifier-Free Guidance strategy that dynamically
adjusts guidance weights across denoising steps. Extensive experimental results
demonstrate that HuMo surpasses specialized state-of-the-art methods in
sub-tasks, establishing a unified framework for collaborative
multimodal-conditioned HCVG. Project Page:
https://phantom-video.github.io/HuMo.