FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking
  Portrait
Authors: name: Taekyung Ki; name: Dongchan Min; name: Gyoungsu Chae
Published: 2024-12-02 02:50:07

Abstract: With the rapid advancement of diffusion-based generative models, portrait
image animation has achieved remarkable results. However, it still faces
challenges in temporally consistent video generation and fast sampling due to
its iterative sampling nature. This paper presents FLOAT, an audio-driven
talking portrait video generation method based on flow matching generative
model. We shift the generative modeling from the pixel-based latent space to a
learned motion latent space, enabling efficient design of temporally consistent
motion. To achieve this, we introduce a transformer-based vector field
predictor with a simple yet effective frame-wise conditioning mechanism.
Additionally, our method supports speech-driven emotion enhancement, enabling a
natural incorporation of expressive motions. Extensive experiments demonstrate
that our method outperforms state-of-the-art audio-driven talking portrait
methods in terms of visual quality, motion fidelity, and efficiency.