DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D
  Diffusion
Authors: name: Yukun Huang; name: Jianan Wang; name: Ailing Zeng; name: Zheng-Jun Zha; name: Lei Zhang; name: Xihui Liu
Published: 2024-09-25 17:59:45

Abstract: Leveraging pretrained 2D diffusion models and score distillation sampling
(SDS), recent methods have shown promising results for text-to-3D avatar
generation. However, generating high-quality 3D avatars capable of expressive
animation remains challenging. In this work, we present DreamWaltz-G, a novel
learning framework for animatable 3D avatar generation from text. The core of
this framework lies in Skeleton-guided Score Distillation and Hybrid 3D
Gaussian Avatar representation. Specifically, the proposed skeleton-guided
score distillation integrates skeleton controls from 3D human templates into 2D
diffusion models, enhancing the consistency of SDS supervision in terms of view
and human pose. This facilitates the generation of high-quality avatars,
mitigating issues such as multiple faces, extra limbs, and blurring. The
proposed hybrid 3D Gaussian avatar representation builds on the efficient 3D
Gaussians, combining neural implicit fields and parameterized 3D meshes to
enable real-time rendering, stable SDS optimization, and expressive animation.
Extensive experiments demonstrate that DreamWaltz-G is highly effective in
generating and animating 3D avatars, outperforming existing methods in both
visual quality and animation expressiveness. Our framework further supports
diverse applications, including human video reenactment and multi-subject scene
composition.