StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D
Authors: name: Pengsheng Guo; name: Hans Hao; name: Adam Caccavale; name: Zhongzheng Ren; name: Edward Zhang; name: Qi Shan; name: Aditya Sankar; name: Alexander G. Schwing; name: Alex Colburn; name: Fangchang Ma
Published: 2023-12-02 02:27:58

Abstract: In the realm of text-to-3D generation, utilizing 2D diffusion models through
score distillation sampling (SDS) frequently leads to issues such as blurred
appearances and multi-faced geometry, primarily due to the intrinsically noisy
nature of the SDS loss. Our analysis identifies the core of these challenges as
the interaction among noise levels in the 2D diffusion process, the
architecture of the diffusion network, and the 3D model representation. To
overcome these limitations, we present StableDreamer, a methodology
incorporating three advances. First, inspired by InstructNeRF2NeRF, we
formalize the equivalence of the SDS generative prior and a simple supervised
L2 reconstruction loss. This finding provides a novel tool to debug SDS, which
we use to show the impact of time-annealing noise levels on reducing
multi-faced geometries. Second, our analysis shows that while image-space
diffusion contributes to geometric precision, latent-space diffusion is crucial
for vivid color rendition. Based on this observation, StableDreamer introduces
a two-stage training strategy that effectively combines these aspects,
resulting in high-fidelity 3D models. Third, we adopt an anisotropic 3D
Gaussians representation, replacing Neural Radiance Fields (NeRFs), to enhance
the overall quality, reduce memory usage during training, and accelerate
rendering speeds, and better capture semi-transparent objects. StableDreamer
reduces multi-face geometries, generates fine details, and converges stably.