Progressive Rendering Distillation: Adapting Stable Diffusion for
  Instant Text-to-Mesh Generation without 3D Data
Authors: name: Zhiyuan Ma, name: Xinyue Liang, name: Rongyuan Wu, name: Xiangyu Zhu, name: Zhen Lei, name: Lei Zhang

It is highly desirable to obtain a model that can generate high-quality 3D
meshes from text prompts in just seconds. While recent attempts have adapted
pre-trained text-to-image diffusion models, such as Stable Diffusion (SD), into
generators of 3D representations (e.g., Triplane), they often suffer from poor
quality due to the lack of sufficient high-quality 3D training data. Aiming at
overcoming the data shortage, we propose a novel training scheme, termed as
Progressive Rendering Distillation (PRD), eliminating the need for 3D
ground-truths by distilling multi-view diffusion models and adapting SD into a
native 3D generator. In each iteration of training, PRD uses the U-Net to
progressively denoise the latent from random noise for a few steps, and in each
step it decodes the denoised latent into 3D output. Multi-view diffusion
models, including MVDream and RichDreamer, are used in joint with SD to distill
text-consistent textures and geometries into the 3D outputs through score
distillation. Since PRD supports training without 3D ground-truths, we can
easily scale up the training data and improve generation quality for
challenging text prompts with creative concepts. Meanwhile, PRD can accelerate
the inference speed of the generation model in just a few steps. With PRD, we
train a Triplane generator, namely TriplaneTurbo, which adds only 2.5%
trainable parameters to adapt SD for Triplane generation. TriplaneTurbo
outperforms previous text-to-3D generators in both efficiency and quality.
Specifically, it can produce high-quality 3D meshes in 1.2 seconds and
generalize well for challenging text input. The code is available at
https://github.com/theEricMa/TriplaneTurbo.