GenCA: A Text-conditioned Generative Model for Realistic and Drivable
  Codec Avatars
Authors: name: Keqiang Sun; name: Amin Jourabloo; name: Riddhish Bhalodia; name: Moustafa Meshry; name: Yu Rong; name: Zhengyu Yang; name: Thu Nguyen-Phuoc; name: Christian Haene; name: Jiu Xu; name: Sam Johnson; name: Hongsheng Li; name: Sofien Bouaziz
Published: 2024-08-24 21:25:22

Abstract: Photo-realistic and controllable 3D avatars are crucial for various
applications such as virtual and mixed reality (VR/MR), telepresence, gaming,
and film production. Traditional methods for avatar creation often involve
time-consuming scanning and reconstruction processes for each avatar, which
limits their scalability. Furthermore, these methods do not offer the
flexibility to sample new identities or modify existing ones. On the other
hand, by learning a strong prior from data, generative models provide a
promising alternative to traditional reconstruction methods, easing the time
constraints for both data capture and processing. Additionally, generative
methods enable downstream applications beyond reconstruction, such as editing
and stylization. Nonetheless, the research on generative 3D avatars is still in
its infancy, and therefore current methods still have limitations such as
creating static avatars, lacking photo-realism, having incomplete facial
details, or having limited drivability. To address this, we propose a
text-conditioned generative model that can generate photo-realistic facial
avatars of diverse identities, with more complete details like hair, eyes and
mouth interior, and which can be driven through a powerful non-parametric
latent expression space. Specifically, we integrate the generative and editing
capabilities of latent diffusion models with a strong prior model for avatar
expression driving.
  Our model can generate and control high-fidelity avatars, even those
out-of-distribution. We also highlight its potential for downstream
applications, including avatar editing and single-shot avatar reconstruction.