X-Dyna: Expressive Dynamic Human Image Animation
Authors: name: Di Chang, name: Hongyi Xu, name: You Xie, name: Yipeng Gao, name: Zhengfei Kuang, name: Shengqu Cai, name: Chenxu Zhang, name: Guoxian Song, name: Chao Wang, name: Yichun Shi, name: Zeyuan Chen, name: Shijie Zhou, name: Linjie Luo, name: Gordon Wetzstein, name: Mohammad Soleymani

We introduce X-Dyna, a novel zero-shot, diffusion-based pipeline for
animating a single human image using facial expressions and body movements
derived from a driving video, that generates realistic, context-aware dynamics
for both the subject and the surrounding environment. Building on prior
approaches centered on human pose control, X-Dyna addresses key shortcomings
causing the loss of dynamic details, enhancing the lifelike qualities of human
video animations. At the core of our approach is the Dynamics-Adapter, a
lightweight module that effectively integrates reference appearance context
into the spatial attentions of the diffusion backbone while preserving the
capacity of motion modules in synthesizing fluid and intricate dynamic details.
Beyond body pose control, we connect a local control module with our model to
capture identity-disentangled facial expressions, facilitating accurate
expression transfer for enhanced realism in animated scenes. Together, these
components form a unified framework capable of learning physical human motion
and natural scene dynamics from a diverse blend of human and scene videos.
Comprehensive qualitative and quantitative evaluations demonstrate that X-Dyna
outperforms state-of-the-art methods, creating highly lifelike and expressive
animations. The code is available at https://github.com/bytedance/X-Dyna.