DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaption
  by Combining 3D GANs and Diffusion Priors
Authors: name: Biwen Lei; name: Kai Yu; name: Mengyang Feng; name: Miaomiao Cui; name: Xuansong Xie
Published: 2023-12-28 05:46:26

Abstract: Text-guided domain adaption and generation of 3D-aware portraits find many
applications in various fields. However, due to the lack of training data and
the challenges in handling the high variety of geometry and appearance, the
existing methods for these tasks suffer from issues like inflexibility,
instability, and low fidelity. In this paper, we propose a novel framework
DiffusionGAN3D, which boosts text-guided 3D domain adaption and generation by
combining 3D GANs and diffusion priors. Specifically, we integrate the
pre-trained 3D generative models (e.g., EG3D) and text-to-image diffusion
models. The former provides a strong foundation for stable and high-quality
avatar generation from text. And the diffusion models in turn offer powerful
priors and guide the 3D generator finetuning with informative direction to
achieve flexible and efficient text-guided domain adaption. To enhance the
diversity in domain adaption and the generation capability in text-to-avatar,
we introduce the relative distance loss and case-specific learnable triplane
respectively. Besides, we design a progressive texture refinement module to
improve the texture quality for both tasks above. Extensive experiments
demonstrate that the proposed framework achieves excellent results in both
domain adaption and text-to-avatar tasks, outperforming existing methods in
terms of generation quality and efficiency. The project homepage is at
https://younglbw.github.io/DiffusionGAN3D-homepage/.