SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We initialize one or several 3D structures  $\{\theta^{(i)}\}_{i = 1}^n$ , a noise prediction model  $\epsilon_{\phi}$  parameterized by  $\phi$ . At each iteration, we sample a camera pose  $c$  from a pre-defined distribution as previous works [34, 20]. Then we render 2D image from 3D structures at pose  $c$  with differentiable rendering  $\pmb{x}_0 = \pmb{g}(\theta, c)$ . To optimize 3D parameters  $\theta$ , we compute the gradient direction of 2D image and then back propagate to the parameter of NeRF, using VSD as Eq. (9). To model the score of the variational distribution, we train an additional diffusion model  $\epsilon_{\phi}$  parameterized by LoRA. We optimize Eq. (8) to train LoRA after optimization of 3D parameters, using the rendered image  $\pmb{x}_0 = \pmb{g}(\theta, c)$  and pose  $c$ . Note that, at each iteration, we perform differentiable rendering only once but use the rendered image twice for both computing gradient direction with VSD and training LoRA. Thus the computation cost will not increase much compared to SDS.
F More Details on Implementation and Hyper-Parameters
Training details. We use v-prediction [42] to train our additional diffusion model  $\epsilon_{\phi}$ . The camera pose  $c$  is fed into a 2-layer MLP and then added to timestep embedding at each U-Net block. NeRF/mesh and LoRA batch sizes are set to 1 owing to the computation limit. A larger batch size of
Figure 13: Ablation on how number of particles affects the results. The diversity of the generated results is slightly larger as the number of particles increases. The quality of generated results is not affected much by the number of particles. The prompt is A high quality photo of an ice cream sundae.
NeRF/mesh may improve the generated quality and we leave it in future work. The learning rate of LoRA is 0.0001 and the learning rate of hash grid encoder is 0.01. We render in RGB color space for high-resolution synthesis, unlike [29, 55] that render in latent space. In addition, we set  $\omega(t) = \sigma_t^2$ . For most experiments, we only use  $n = 1$  particle for VSD to reduce the computation time (and we only use a batch size of 1, due to the computation resource limits). For NeRF rendering, we sample 96 points along each ray, with 64 samples in coarse stage and 32 samples in fine stage. We choose a single-layer MLP to decode the color and volumetric density from the hash grid encoder as previous work [20].
For object-centric scenes, we set the camera radius as  $\mathcal{U}(1.0, 1.5)$ . The bounding box size is set as 1.0. For large scene generation, we enlarge the range of camera radius to  $\mathcal{U}(0.1, 2.3)$  for better details
(1) 64 rendering.
(2) 128 rendering.
(3) 256 rendering.
Figure 14: Ablation study on rendering resolution during NeRF training. Training with higher resolution produces better results with finer details. However, our VSD still provides competitive results under a lower resolution (128 or 256), which is more computationally efficient. The prompt here is an elephant skull.
(4) 512 rendering.
and geometry consistency. We enlarge the bounding box size to 5.0. For large scenes with a centric object, we set the range of camera radius as  $\mathcal{U}(1.0,2.0)$ .
DreamFusion [34] introduces an explicit shading model based on normal vectors during the training process to enhance the geometry. We currently disable the shadings and it reduces the computational cost and memory consumption. We leave incorporating the shading model into our method as future work.
**Optimization.** We optimize  $25k$  steps for each particle with AdamW [26] optimizer in NeRF stage. We optimize  $15k$  steps for geometry fine-tuning and  $30k$  steps for texture fine-tuning. At each stage, for the first  $5k$  steps we sample time steps  $t \sim \mathcal{U}(0.02, 0.98)$  and then directly change into  $t \sim \mathcal{U}(0.02, 0.50)$ . For large scene generation, we delay the annealing time to  $10k$  steps since large scene generation requires more iterations to converge. The NeRF training stage consumes  $17/17/18/27\mathrm{GB}$  GPU memory with  $64/128/256/512$  rendering resolution and batch size of 1. The Mesh fine-tuning stage consumes around 17GB GPU memory with 512 rendering resolution and batch size of 1. The whole optimization process takes around several hours per particle on a single NVIDIA A100 GPU. We believe adding more GPUs in parallel will accelerate the generation process, and we leave it for future work.
Licenses Here we provide the URL, citations and licenses of the open-sourced assets we use in this work.
Table 2: URL, citations and licenses of the open-sourced assets we use in this work.
URL	Citation	License
https://github.com/ashawkey/stable-dreamfusion	[51]	Apache License 2.0
https://github.com/threestudio-project/threestudio	[13]	Apache License 2.0
https://github.com/Stability-AI/stablediffusion	[38]	MIT License
https://github.com/NVIDIAGameWorks/kaolin	[11]	Apache License 2.0
https://github.com/huggingface/diffusers	[54]	Apache License 2.0
G 2D Experiments of Variational Score Distillation
Here we describe the details of the experiments on 2D images with Variation Score Distillation in the main text of Fig. 3. Here we set the number of particles as 8. We train a smaller U-Net from scratch to estimate the variational score since the distribution of several 2D images is much simpler than the distribution of images rendered from different poses of 3D structures. The optimization takes around 6000 steps. Additional U-Net is optimized 1 step on every optimization step of the particle images. The learning rate of particle images is 0.03 and the learning rate of U-Net is 0.0001. The batch size of particles and U-Net are both set as 8. Here we do not use annealed time schedule for both VSD and SDS for a fair comparison.
To further demonstrate the effectiveness of our VSD, we increase the number of particles to a large number of 2048. The optimization takes around  $20k$  steps. The batch size of U-Net is 192 and the batch size of particles is set as 16. The results are shown in Figure 15. It can be seen from the figure that our VSD generates plausible results even under a large number of particles, which demonstrates the scalability of VSD. Although, due to the high optimization cost of 3D representation, the number of particles in 3D experiments is relatively small, we demonstrate that VSD has the potential to scale up to more particles.
Figure 15: Selected samples from 2048 particles in a 2D experiment of VSD. Our VSD generates plausible results even under a large number of particles, which demonstrates the scalability of VSD. The prompt is an astronaut is riding a horse.
To match the 3D experiment setting, we also provide 2D experiments with LoRA and add annealed time schedule. The results are shown in Figure 16. The number of particles is set as 6 and  $\mathrm{CFG} = 7.5$ . Our VSD provides high-fidelity and diverse results in this setting. We also set the number of particles as 2048 and the results are shown in Figure 17.
Figure 16: 2D experiments of VSD (CFG= 7.5) results with LoRA and annealed time schedule. The number of particles is 6. Our VSD provides high-fidelity and diverse results in this setting. The prompt is an astronaut is riding a horse.
We also provide the results of SDS for comparison in Fig. 18. Compared to our VSD, the results of SDS are smoother and lack details.
Figure 17: Selected samples of 2D experiments of VSD (CFG= 7.5) results with LoRA. The number of particles is 2048.
Figure 18: SDS result in 2D.
Moreover, we visualize VSD/SDS training phase of 2D in Fig. 19. Since the gradient is not directly readable, we visualize  $x + \Delta x$ , which is the updated results if current sample optimizes via this gradient direction. As shown in Fig. 19, SDS tends to provide over-saturated and over-smooth gradient while VSD provides more natural-looking gradients with more details. As a consequence, VSD provides better final results.
H How will CFG weights affect diversity?
In this section, we explore how CFG affects the diversity of generated results. For VSD, we set the number of particles as 4 and run experiments with different CFG. For SDS, we run 4 times
Figure 19: Gradient visualization of VSD and SDS during the training phase in 2D experiment. SDS tends to provide over-saturated and over-smooth gradient while VSD provides more natural-looking gradients with more details.
of generation with different random seeds. The results are shown in Figure 20. As shown in the figure, smaller CFG provides more diversity. We conjecture that this is because the distribution of smaller guidance weights has more diverse modes. However, when the CFG becomes too small (e.g.,  $\mathrm{CFG} = 1$ ), it cannot provide enough guidance strength to generate plausible results. Therefore, for all our 3D experiments shown in the results, we set  $\mathrm{CFG} = 7.5$  as a trade-off between diversity and optimization stability. Note that SDS could not work well in such small CFG weights. Instead, our VSD provides a trade-off option between CFG weight and diversity, and it can generate more diverse results by simply setting a smaller CFG.
I Limitations and Discussions
Although ProlificDreamer achieves remarkable text-to-3D results, the generation takes hours (especially for high-resolution NeRF training), which is much slower than the vanilla image generation by a diffusion model. Speeding up the text-to-3D generation is another critical problem, and we leave it in future work.
Besides, our proposed scene initialization demonstrates its effectiveness in generating expansive scenes, yet there are still limitations, particularly with respect to camera positioning. Our current model sets the camera with a fixed view toward the scene's center, which serves object-centric scenes effectively but may be suboptimal for scenes with intricate geometry and detailed textures. Furthermore, despite our efforts to produce outcomes with a rich structure, occasional failures occur and the geometry reverts to a simplistic textured sphere. In order to address these limitations, future research could focus on developing improved camera poses capable of capturing and rendering scenes in finer detail. Furthermore, our present model relies solely on the text-to-image diffusion model without the assistance of other models. The integration of off-the-shelf depth estimation models [37], as utilized in other studies [16, 58], could potentially enhance the accuracy and detail of the scene generation process.
In addition, the correspondence between text prompts and generated results is sometimes insufficient, especially for complex prompts. We conjecture that it is because the ability to generate from complex prompts is limited by the text encoder of Stable Diffusion. In addition, the multi-face Janus problem [57] also exists in our case. Nevertheless, we believe our proposed VSD and other contributions are orthogonal to these problems, and the generation quality can be further improved by introducing more techniques, such as using a more powerful text-to-image diffusion model that understands view-dependent prompts better [34, 41], or a diffusion model with more 3D priors [24].
J User Study
For completeness, we follow previous works [20, 4] and conduct a user study by comparing ProlificDreamer with DreamFusion [34], Magic3D [20] and Fantasy3D [4] under 15 prompts, 5 prompts for each baseline. Since none of the baselines have released their codes, we can only use
(c) VSD with single particle and different seeds provide slightly less diversity.
Figure 20: Ablation on how CFG weight affects the randomness. Smaller CFG provides more diversity. But too small CFG provides less optimization stability. The prompt is A high quality photo of an ice cream sundae.
the figures from their papers, which limits the number of results from baselines for us to compare. So we currently only compare under 15 prompts. The volunteers are shown the generated results of our ProlificDreamer and baselines and asked to choose the better one in terms of fidelity, details and vividness. We collect results from 109 volunteers, yielding 1635 pairwise comparisons. The results are shown in Table 3. Our method outperforms all of the baselines.
Table 3: Results of user study. The percentage of user preference  $(\uparrow)$  is reported in the table.
Name	DreamFusion	Magic3D	Fantasia3D
Prefer baseline	6.87	5.50	9.73
Prefer ProlificDreamer (Ours)	94.13	94.50	90.27
K Quantitative Results
Table 4: 3D sample quality by SDS or VSD, 100 prompts.
Method	SDS	VSD (n=1)
3D-FID (↓)	118.92	107.02
Table 5: 3D sample quality by SDS or VSD, 25 prompts.
Method	SDS	VSD (n=1)	VSD (n=4)
3D-FID (↓)	191.82	186.87	185.88
In this section, we add some quantitative results to demonstrate the effectiveness of VSD.
Experiment Setting For 3D experiments, we compute the FID score between rendered images by SDS/VSD and 2D sampled images by ancestral sampling, named as 3D-FID. Specifically, we select 100 prompts from previous works including DreamFusion, Magic3D and Fantasy3D. For each prompt, we use VSD (number of particles  $n = 1$ , CFG=7.5) or SDS (CFG=100) to optimize one 3D object and render 10 images uniformly from the circumference at an angle of  $30^{\circ}$  above the horizon, and collect 1k images in total. To isolatedly compare VSD with SDS, we run with the default setting of the stage-1 NeRF training of ProlificDreamer (i.e., both VSD and SDS are in 512 resolution and use annealed  $t$ ).
In Table 4, We compute the FID score between the 1k samples from the 100 prompts and a 50k reference batch, which is sampled by 50-step DPM-Solver++ [28] with 500 images per prompt. In Table 5, due to the time and computation resource limits, we compare the results for VSD with  $n = 4$  under only 25 randomly-selected prompts from the aforementioned 100 prompts, and compare SDS, VSD ( $n = 1$ ) and VSD ( $n = 4$ ) with the 50-step DPM-Solver++ under these 25 prompts. For VSD ( $n = 4$ ), as we can get 4 particles (3D objects) per prompt, we randomly select one particle per prompt and render the corresponding 10 images of the selected particle for fair comparison. For 2D experiments in Table 6, we follow the common setting of evaluating text-to-image models by computing FID on MSCOCO2014 validation set. Specifically, we randomly select 1k prompts and sample one image per prompt by either 50-step DPM-Solver++, SDS (CFG=100), VSD ( $n = 4$ , CFG=7.5) or VSD ( $n = 8$ , CFG=7.5) to collect 1k samples for each method, and then compute the FID between the obtained samples and the entire COCO validation set. For VSD, as we can get  $n$  images per prompt, we randomly select one image per prompt for fair comparison.
Results VSD with  $n = 1$  still outperforms SDS in 3D (both with 512 resolution and annealed  $t$ ), as shown in Table 4 and Table 5, which demonstrates the effectiveness of VSD.
Using more particles is slightly better. Due to the limitation of time and computation resources, we only compare  $n = 1$  and  $n = 4$  in 3D experiments, and  $n = 4$  with  $n = 8$  in the 2D experiments. As shown in Table 5, VSD with 4 particles slightly outperforms VSD with 1 particles in the 3D setting; and as shown in Table 6, VSD with 8 particles slightly outperforms VSD with 4 particles in the 2D setting.
Table 6: 2D sample quality by different samplers, 1000 prompts.
Method	SDS	VSD (n=4)	VSD (n=8)	DPM++
FID (↓)	90.09	68.02	66.68	47.91
VSD outperforms SDS in 2D. As shown in Table 6, the FID by VSD is much better than SDS. As the 2D setting isolates the sampling algorithm from the 3D representations, we can directly compare different sampling algorithms, finding that VSD can get better sample quality than SDS (though still worse than SOTA diffusion samplers, it can generalize to 3D cases).
L Why using SDS in stage-2 for the geometry optimization of mesh?
VSD can also be used to generate geometry. To validate this, we provide an ablation example in Fig. 21 (3a),(3b). As shown in the figure, VSD can obtain reasonable geometry. Although the some part of the geometry from VSD is with more details than SDS (including the tail of the horse), on the whole, the result from VSD is similar with SDS. We conjecture that this is because currently the triangle size of the mesh is relatively large and can't represent very fine details. Thus, for efficiency, we use SDS instead of VSD for mesh geometry optimization. We believe that VSD can be used to obtain high quality mesh if more advanced mesh representation is available.
Moreover, despite that we use SDS to optimize the geometry in stage-2, VSD is still crucial in stage-1 and stage-3, in which VSD significantly improves the generated quality.
(1) Generate NeRF with VSD.
(2) Get coarse mesh with Marching Tets.
(3a) Mesh optimization with SDS.
Figure 21: Pipeline along with ablation study of VSD for geometry optimization. In (3a) and (3b), we show that VSD can also be used for mesh optimization of geometry. We also show intermediate results of our pipeline in this figure along with the samples from LoRA during the training phase in (5).
(3b) Mesh optimization with VSD.
(4) Inherit texture from NeRF.
(5) Finetune texture with VSD.
LoRA sample during stage (5).
M More Comparisons with Baselines
Here, we provide more comparisons with baselines in Fig. 22, Fig. 23, Fig. 24, Fig. 25, Fig. 26 and Fig. 27. Since none of the baselines have released their codes, we can only directly copy the figures from the corresponding papers. Some baselines are missing given a specific prompt because the prompt is not included in the corresponding papers. To demonstrate geometry, some baselines choose textureless shading [34, 20], while the other [4] prefers the normal map. For ProlificDreamer, we uniformly show the normal map for consistency. As shown in the figures, our method achieves better results in terms of fidelity and details.
DreamFusion
A plate of fried chicken and waffles with maple syrup on them.
Figure 22: More results of ProlificDreamer compared with baselines.
A sliced loaf of fresh bread.
Ours
A blue tulip.
Ours
A cauldron full of gold coins.
Ours
A lionfish.
Figure 23: More results of ProlificDreamer compared with baselines.
A praying mantis wearing roller.
Ours
Figure 24: More results of ProlificDreamer compared with baselines.
Fantasia 3D
DreamFusion
Figure 25: More results of ProlificDreamer compared with baselines.
A delicious croissant.
Magic 3D
DreamFusion
A rabbit, animated movie character, high detail 3d model.
Figure 26: More results of ProlificDreamer compared with baselines.
Magic 3D
Magic 3D
Figure 27: More results of ProlificDreamer compared with baselines.