SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now systematically compare VSD with SDS in both theory and practice.
Note that we have two variables of time: one is the diffusion time  $t \in [0, T]$  and the other is the gradient flow time  $\tau$ , corresponding to the optimization iteration for each  $\theta$ .
Table 1: Design space of text-to-3D via 2D diffusion. We highlight the contributions of this paper that improve the fidelity, diversity and ability to generate complex scenes by *, † and ‡ respectively.
Method	DreamFusion [34]	Magic3D [20]	Fantasia3D [4]	Ours
NeRF Representation
Resolution*	64	64	-	512
Backbone	mipNeRF360 [2]	Instant NGP [32]	-	Instant NGP [32]
Initialization‡	Object	Object	-	Object / Scene initialization
NeRF Training
Base model	Imagen [41]	eDiff-I [1]	-	Stable Diffusion [38]
Number of particles†	1	1	-	1~4
Distillation objective*†	SDS (Eq. (3))	SDS (Eq. (3))	-	VSD (Eq. (9))
CFG*	100	100	-	7.5
Time schedule*	U(0.02, 0.98)	U(0.02, 0.98)	-	U(0.02, 0.98) → U(0.02, 0.5)
Mesh Representation
Initialization	-	From NeRF	Handcrafted	From NeRF
Texture and geometry	-	Entangled	Disentangled	Disentangled
Mesh Training
Distillation objective*†	-	SDS (Eq. (3))	SDS (Eq. (3))	VSD (Eq. (9))
CFG*	-	100	100	7.5
SDS as a special case of VSD. Theoretically, comparing the update rules of SDS (Eq. (3)) and VSD (Eq. (9)), SDS is a special case of VSD by using a single-point Dirac distribution  $\mu (\theta |y)\approx \delta (\theta -\theta^{(1)})$  as the variational distribution (see Appendix C.3 for derivation). In particular, VSD not only employs potentially multiple particles but also learns a parametric score function  $\epsilon_{\phi}$  even for a single particle (i.e.,  $n = 1$ ). Empirically, the learned neural network may potentially offer superior generalization ability over the Dirac distribution in SDS, thus it may provide more accurate updating directions in low-density regions. Moreover, by using LoRA, VSD can additionally exploit the text prompt  $y$  in the estimation  $\epsilon_{\phi}(\pmb {x}_t,t,c,y)$ , while the Gaussian noise  $\epsilon$  used in SDS cannot leverage the information from  $y$ . Thus, VSD may provide samples which are more aligned with the prompt  $y$ .
VSD is friendly to CFG. As VSD aims to sample  $\theta$  from the optimal  $\mu^{*}$  defined by the pretrained model  $\epsilon_{\mathrm{pretrain}}$ , the effects by tuning the CFG in  $\epsilon_{\mathrm{pretrain}}$  for 3D sampling by VSD are quite similar to 2D sampling by the traditional ancestral sampling methods [14, 27]. Therefore, VSD can tune CFG as flexibly as the classic text-to-image methods, and we use the same setting of CFG (e.g. 7.5) as the common text-to-image generation task for the best performance. To the best of our knowledge, this for the first time addresses the problem in previous SDS [34, 20, 4, 29] that it usually requires a large CFG (i.e., 100).
VSD vs. SDS in 2D experiments that isolate 3D representations. To directly compare SDS and VSD, we consider a special case of the rendering function  $g(\theta)$  to decouple the optimization algorithm from 3D representations. In particular, we set  $g(\theta, c) \equiv \theta$  for any  $c$ . Then the rendered image  $x = g(\theta, c) = \theta$  is the same 2D image as  $\theta$ . In such a case, optimizing the parameter  $\theta$  is equivalent to generating an image in 2D space, thereby independent of the 3D representation. We show the results of different sampling methods in Fig. 3. SDS exhibits failure under both small and large CFG weights. Particularly with the default CFG weight (i.e., 100) used in SDS, the 2D samples share the same issues previously observed in text-to-3D such as over-saturation and oversmoothing [34]. In contrast, VSD demonstrates flexibility in accommodating various CFG weights and produces realistic samples using a normal CFG weight (i.e., 7.5), behaving similarly to ancestral sampling from diffusion models. See more details and analysis in Appendix G.
As other 3D factors are isolated in this comparison, these theoretical and empirical results suggest that the aforementioned practical issues of SDS [34] stem from the oversimplified variational distribution and large CFG employed by SDS. Such results strongly motivate us to employ VSD for text-to-3D generation, where it still substantially and consistently outperforms SDS (see evidence in Sec 5).
4 ProlificDreamer
We further present a clear design space for text-to-3D in Sec. 4.1 and systematically study other elements orthogonal to the distillation algorithm in Sec. 4.2. Combining all improvements highlighted in Tab. 1, we arrive at ProlificDreamer, an advanced text-to-3D approach.
DreamFusion
A 3D model of an adorable cottage with a thatched roof.
Magic3D
Fantasia3D
A plate piled high with chocolate chip cookies.
Figure 4: Comparison with baselines. Our results have higher fidelity and more details.
4.1 Design Space of Text-to-3D Generation
We adopt the two-stage approach [20], with several improvements in the design space of text-to-3D generation as summarized in Table 1. Specifically, in the first stage, we optimize a high-resolution (e.g., 512) NeRF by VSD to utilize its high flexibility for generating scenes with complex geometry. In the second stage, we use DMTet [45] to extract textured mesh from the NeRF obtained in the first stage, and further fine-tune the textured mesh for high-resolution details. The second stage is optional because both NeRF and mesh have their own advantages in representing 3D content and are preferred in certain cases. Nevertheless, ProlificDreamer can generate both high-fidelity NeRFs and meshes.
4.2 3D Representation and Training
We systematically study other elements orthogonal to the algorithmic formulation. Specifically, we propose a high rendering resolution of  $512 \times 512$  during training and an annealed distilling time schedule to improve the visual quality. We also carefully design a scene initialization, which is crucial for complex scene generation.
High-resolution rendering for NeRF training. We choose Instant NGP [32] for efficient high-resolution rendering and optimize NeRF with up to 512 training resolution using VSD. By applying VSD, we obtain high-fidelity NeRFs with resolutions varying from 64 to 512.
Scene initialization for NeRF training. We initialize the density for NeRF as  $\sigma_{\mathrm{init}}(\pmb{\mu}) = \lambda_{\sigma}$  ( $1 - \frac{||\pmb{\mu}||_2}{r}$ ), where  $\lambda_{\sigma}$  is the density strength,  $r$  is the density radius, and  $\pmb{\mu}$  is the coordinate. For object-centric scenes, we follow the object-centric initialization used in Magic3D [20] with  $\lambda_{\sigma} = 10$  and  $r = 0.5$ ; For complex scenes, we propose scene initialization by setting  $\lambda_{\sigma} = -10$  to make the density "hollow" and  $r = 2.5$  that encloses the camera. We show in Appendix D that the scene initialization can help to generate high-fidelity complex scenes without other modifications to the existing algorithm. In addition, we can further add a centric object to the complex scene by using object-centric initialization for  $||\pmb{\mu}||_2 < 5/6$  and scene initialization for others, where the hyperparameter  $5/6$  ensures the initial density function is continuous.
Annealed time schedule for score distillation. We utilize a simple two-stage annealing of time step  $t$  in the score distillation objective, suitable for both SDS (Eq. (3)) and VSD (Eq. (9)). For the first several steps we sample time steps  $t \sim \mathcal{U}(0.02, 0.98)$  and then anneal into  $t \sim \mathcal{U}(0.02, 0.50)$ . The key insight is that, essentially, we aim to match the original  $q_0^\mu(\boldsymbol{x}_0|c, y)$  with  $p_0(\boldsymbol{x}_0|y^c)$ . The KL-divergence for larger  $t$  can provide reasonable optimization direction during the early stage of training. During training, while  $\boldsymbol{x}^*$  is approaching the support of  $p_0(\boldsymbol{x}^*|y^c)$ , a smaller  $t$  can narrow the gap between  $p_t(\boldsymbol{x}^*|y^c)$  and  $p_0(\boldsymbol{x}^*|y^c)$ , and provide elaborate details aligning with  $p_0(\boldsymbol{x}^*|y^c)$ .
Mesh representation and fine-tuning. We adopt a coordinate-based hash grid encoder inherited from NeRF stage to represent the mesh texture. We follow Fantasia3D [4] to disentangle the
(1) 64 rendering
(2) + 512 rendering
(3) + Annealed  $t$
Figure 5: Ablation study of proposed improvements for high-fidelity NeRF generation. The prompt is an elephant skull. (1) The common setting [34, 20] adopts 64 rendering resolution and SDS loss. (2) We improve the generated quality by increasing the rendering resolution. (3) Annealed time schedule adds more details to the generated result. (4) VSD makes the results even better with richer details.
(4) + VSD
optimization of geometry and texture by first optimizing the geometry using the normal map and then optimizing the texture. In our initial experiments, we find that optimizing geometry with VSD provides no more details than using SDS. This may be because the mesh resolution is not large enough to represent high-frequency details. Thus, we optimize geometry with SDS for efficiency. But unlike Fantasia3D [4], our texture optimization is supervised by VSD with CFG = 7.5 with the annealed time schedule, which can provide more details than SDS.
5.1 Results of ProlificDreamer
We show the generated results of ProlificDreamer in Fig. 1(a) and 1(b), including high-fidelity mesh and NeRF results. All the results are generated by VSD. For all experiments without mentioned, VSD uses  $n = 1$  particle for a fair comparison with SDS. In Fig. 1(c), we also demonstrate VSD can generate diverse results, showing that different particles in a round are diverse (with  $n = 4$ ).
Object-centric generation. We compare our method with three SOTA baselines, DreamFusion [34], Magic3D [20] and Fantasy3D [4]. All of the baselines are based on SDS. Since none of them is open-sourced, we use the figures from their papers. As shown in Fig. 4, ProlificDreamer generates 3D objects with higher fidelity and more details, which demonstrates the effectiveness of our method.
In appendix, we add user study (Section J) and quantitative results (Section K) of ProlificDreamer against the baselines to demonstrate the effectiveness of our method.
Large scene generation. As shown in Fig. 1(b), our method can generate  $360^{\circ}$  scenes with high-fidelity and fine details. The depth map shows that the scenes have geometry instead of being a  $360^{\circ}$  textured sphere, verifying that with our scene initialization alone we can generate high-fidelity large scenes without much modification to existing components. See more results in Appendix B.
Ablation on NeRF training. Fig. 5 provides the ablation on NeRF training. Starting from the common setting [34, 20] with 64 rendering resolution and SDS loss, we ablate our proposed improvements step by step, including increasing resolution, adding annealed time schedule, and adding VSD all improve the generated results. It demonstrates the effectiveness of our proposed components. We provide more ablation on large scene generation in Appendix D, with a similar conclusion.
Ablation on mesh fine-tuning. We ablate between SDS and VSD on mesh fine-tuning, as shown in Appendix D. Fine-tuning texture with VSD provides higher fidelity than SDS. As the fine-tuned results of textured mesh are highly dependent on the initial NeRF, getting a high-quality NeRF at the first stage is crucial. Note that the provided results of both VSD and SDS in mesh fine-tuning are based on and benefit from the high-fidelity NeRF results in the first stage by our VSD.
Ablation on CFG. We perform ablation to explore how CFG affects generation diversity. We find that smaller CFG encourages more diversity. Our VSD works well with small CFG and provides
considerable diversity, while SDS cannot generate plausible results with small CFG (e.g., 7.5), which limits its ability to generate diverse results. Results and more details are shown in Appendix H.
Diffusion models. Score-based generative model [48, 49] and diffusion models [14] have shown great performance in image synthesis [14, 7]. Recently, large-scale diffusion models have shown great performance in text-to-image synthesis [36, 41, 38], which provides an opportunity to utilize it for zero-shot text-to-3D generation.
Text-to-3D generation. DreamField [19] proposes a text-to-3D method using CLIP [35] guidance. DreamFusion [34] proposes a text-to-3D method using 2D diffusion models. Score Jacobian Chaining (SJC) [55] derives the training objective of text-to-3D using a 2D diffusion model from another theoretical basis. Magic3D [20] extends text-to-3D to a higher resolution with mesh [45] representation. Latent-NeRF [29] optimizes NeRF in latent space. Fantasia3D [4] optimizes a mesh with DMTet [45] from scratch. Although Fantasia3D achieves remarkable zero-shot text-to-3D generation, it requires user-provided shape guidance for the generation of complex geometry. [17] propose score debiasing and prompt debiasing to mitigate multiface problem and is orthogonal to our work. TextMesh [52] is contemporary with us and proposes a different pipeline for high-fidelity text-to-3D generation. 3DFuse [44] proposes to incorporate 3D awareness into 2D diffusion for better 3D consistency in text-to-3D generation. In addition, adjusting time schedule has also been discussed in previous works [12, 55]. However, previous works either require carefully devising the schedule [12] or perform inferior to the simple random time schedule [55]. Instead, our 2-stage annealing schedule is easy to train and achieves better generation quality than the random time schedule.
Text-driven large scene generation. Text2Room [16] generates indoor rooms from a given prompt. However, it uses additional monocular depth estimation models as prior, and we do not use any additional models. Our method generates the wild text prompt and uses only a text-to-image diffusion model. Set-the-scene [5] is a contemporary work with us aimed at large scene generation with a different pipeline. Overall, ProlificDreamer uses an advanced optimization algorithm, i.e., VSD with our proposed two-stage annealed time schedule, which has a significant advantage over the previous SDS / SJC (see Appendix C.3 for details). As a result, ProlificDreamer achieves high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops) and photo-realistic mesh results.
In this work, we systematically study the problem of text-to-3D generation. In terms of the algorithmic formulation, we propose variational score distillation (VSD), a principled particle-based variational framework that treats the 3D parameter as a random variable and infers its distribution. VSD naturally generalizes SDS in the variational formulation and addresses the practical issues of SDS observed before. With other orthogonal improvements to 3D representations, our overall approach, ProlificDreamer, can generate high-fidelity NeRF and photo-realistic textured meshes.
Limitations and broader impact. Although ProlificDreamer achieves remarkable text-to-3D results, the generation takes hours of time, which is much slower than image generation by a diffusion model. Although large scene generation can be achieved with our scene initialization, the camera poses during training are regardless of the scene structure, which may be improved by devising an adaptive camera pose range according to the scene structure for better-generated details. Moreover, due to the limited expressiveness of the base 2D model, the generation for complex prompts may fail, and the generated samples may have multi-face Janus problem [57] in some cases because of the poor text-image alignment for the view-dependent prompts. In addition, content creation by generative models may cause harmful social impacts on the labor market. Also, like other generative models, our method may be utilized to generate fake and malicious contents, which needs more caution.
This work was supported by NSF of China Projects (Nos. 62061136001, 61620106010, 62076145, U19B2034, U1811461, U19A2081, 6197222); Beijing Outstanding Young Scientist Program NO.
BJWZYJH012019100020098; a grant from Tsinghua Institute for Guo Qiang; the High Performance Computing Center, Tsinghua University; and the Research Funds of Renmin University of China (22XNKJ13). J.Z was also supported by the XPlorer Prize. C. Li was also sponsored by Beijing Nova Program (No. 20220484044).
[1] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers, 2023.
[2] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mipnerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[3] Changyou Chen, Ruiyi Zhang, Wenlin Wang, Bai Li, and Liquin Chen. A unified particle-optimization framework for scalable bayesian sampling. arXiv preprint arXiv:1805.11659, 2018.
[4] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. *Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.* arXiv preprint arXiv:2303.13873, 2023.
[5] Dana Cohen-Bar, Elad Richardson, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Setthe-scene: Global-local training for generating controllable nef scenes. arXiv preprint arXiv:2303.13450, 2023.
[6] Gianluca Detommaso, Tiangang Cui, Youssef Marzouk, Alessio Spantini, and Robert Scheicl. A stein variational newton method. Advances in Neural Information Processing Systems, 31, 2018.
[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[8] Lauro Langosco di Langosco, Vincent Fortuin, and Heiko Strathmann. Neural variational gradient descent. arXiv preprint arXiv:2107.10731, 2021.
[9] Hanze Dong, Xi Wang, Yong Lin, and Tong Zhang. Particle-based variational inference with preconditioned functional gradient flow. arXiv preprint arXiv:2211.13954, 2022.
[10] Yihao Feng, Dilin Wang, and Qiang Liu. Learning to draw samples with amortized stein variational gradient descent. arXiv preprint arXiv:1707.06626, 2017.
[11] Clement Fuji Tsang, Maria Shugrina, Jean Francois Lafleche, Towaki Takikawa, Jiehan Wang, Charles Loop, Wenzheng Chen, Krishna Murthy Jatavallabhula, Edward Smith, Artem Rozant-sev, Or Perel, Tianchang Shen, Jun Gao, Sanja Fidler, Gavriel State, Jason Gorski, Tommy Xiang, Jianing Li, Michael Li, and Rev Lebaredian. Kaolin: A pytorch library for accelerating 3d deep learning research. https://github.com/NVIDIAGames/kaolin, 2022.
[12] Alexandros Graikos, Nikolay Malkin, Nebojsa Joic, and Dimitris Samaras. Diffusion models as plug-and-play priors. arXiv preprint arXiv:2206.09012, 2022.
[13] Yuan-Chen Guo, Ying-Tian Liu, Chen Wang, Zi-Xin Zou, Guan Luo, Chia-Hao Chen, Yan-Pei Cao, and Song-Hai Zhang. threestudio: A unified framework for 3d content generation. https://github.com/threestudio-project/threestudio, 2023.
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[15] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
[51] Jiaxiang Tang. Stable-dreamfusion: Text-to-3d with stable-diffusion, 2022. https://github.com/ashawkey/stable-dreamfusion.
[52] Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. arXiv preprint arXiv:2304.12439, 2023.
[53] Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
[54] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/diffusers, 2022.
[55] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:2212.00774, 2022.
[56] Ziyu Wang, Tongzheng Ren, Jun Zhu, and Bo Zhang. Function space particle optimization for bayesian neural networks. arXiv preprint arXiv:1902.09754, 2019.
[57] Wikipedia contributors. Janus — Wikipedia, the free encyclopedia. https://en.wikipedia.org/w/index.php?title=Janus, 2023.
[58] Jingbo Zhang, Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao. Text2nerf: Text-driven 3d scene generation with neural radiance fields, 2023.
A Additional Experiment Results of 3D Textured Meshes
A delicious croissant.
A sliced loaf of fresh bread.
A small saguaro cactus planted in a clay pot.
A blue tulip.
A plate of fried chicken and waffles with maple syrup on them.
A rabbit, animated movie character, high detail 3d model.
Figure 6: More results of ProlificDreamer of 3D textured meshes.
A cauldron full of gold coins.
A lionfish.
We provide more results of ProlificDreamer of 3D textured meshes in Fig. 6 and Fig. 7.
A car made out of sushi.
A DSLR photo of an imperial state crown of England.
A rotary telephone carved out of wood.
A marble bust of a mouse.
A Matte painting of a castle made of cheesecake surrounded by a moat made of ice cream.
A plush dragon toy.
A praying mantis wearing roller.
Figure 7: More results of ProlificDreamer of 3D textured meshes.
Figure 8: Results of high quality NeRF from ProlificDreamer. Compared with DreamFusion, our ProlificDreamer generates better NeRF results in terms of fidelity and details, which demonstrates the effectiveness of our VSD against SDS.
B Additional Experiment Results of 3D NeRF
Here, we provide more generated results in Fig. 8 of high quality NeRF. We compare with DreamFusion, as DreamFusion also uses NeRF representation. It can be seen from the figure that ProlificDreamer generates better NeRF results in terms of fidelity and details, which demonstrates the effectiveness of our VSD against SDS.
In Fig. 9, we provide more results of large scenes with  $360^{\circ}$  environment using our scene initialization. In Fig. 10, we provide more results of multiple particle experiments.
C Theory of Variational Score Distillation
C.1 Particle-based Variational Inference
Particle-based variational inference (ParVI) [23, 6, 3, 56, 25, 9] aims to draw (particle) samples from a desired distribution by minimizing the KL-divergence between particle samples and the desired distribution, and is widely-used in Bayesian inference [23, 10, 21]. Specifically, denote  $\mathcal{P}(\Theta)$  as the set containing all the distributions on a support space  $\Theta$  with Euclidean distance, and  $\mathbb{W}_2(\Theta) := \{\mu \in \mathcal{P}(\Theta) : \exists \theta_0 \in \Theta \text{ s.t. } \mathbb{E}_{\mu(\theta)}[||\theta - \theta_0||^2] < +\infty\}$  as the 2-Wasserstein space [53] equipped with the Wasserstein 2-distance [53]. Given a desired distribution  $\mu^* \in \mathcal{P}(\Theta)$ , ParVI
Figure 9: Result of high quality NeRF from ProlificDreamer, a large scene with  $360^{\circ}$  environment using our scene initialization. The prompt here is Small lavender isometric room, soft lighting, unreal engine render, voxels.
A car made out of sushi.
Figure 10: More results of multi-particle experiments in NeRF.