StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation
Authors: name: Shuyuan Tu, name: Yueming Pan, name: Yinming Huang, name: Xintong Han, name: Zhen Xing, name: Qi Dai, name: Chong Luo, name: Zuxuan Wu, name: Yu-Gang Jiang

Current diffusion models for audio-driven avatar video generation struggle to
synthesize long videos with natural audio synchronization and identity
consistency. This paper presents StableAvatar, the first end-to-end video
diffusion transformer that synthesizes infinite-length high-quality videos
without post-processing. Conditioned on a reference image and audio,
StableAvatar integrates tailored training and inference modules to enable
infinite-length video generation. We observe that the main reason preventing
existing models from generating long videos lies in their audio modeling. They
typically rely on third-party off-the-shelf extractors to obtain audio
embeddings, which are then directly injected into the diffusion model via
cross-attention. Since current diffusion backbones lack any audio-related
priors, this approach causes severe latent distribution error accumulation
across video clips, leading the latent distribution of subsequent segments to
drift away from the optimal distribution gradually. To address this,
StableAvatar introduces a novel Time-step-aware Audio Adapter that prevents
error accumulation via time-step-aware modulation. During inference, we propose
a novel Audio Native Guidance Mechanism to further enhance the audio
synchronization by leveraging the diffusion's own evolving joint audio-latent
prediction as a dynamic guidance signal. To enhance the smoothness of the
infinite-length videos, we introduce a Dynamic Weighted Sliding-window Strategy
that fuses latent over time. Experiments on benchmarks show the effectiveness
of StableAvatar both qualitatively and quantitatively.