ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation
Authors: Zhengyi Wang
Cheng Lu
Yikai Wang
Fan Bao
Chongxuan Li
Hang Su
Jun Zhu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: diffusion models
text to 3D
TL;DR: We achieve photo-realistic text-to-3D results by only using 2D diffusion models, with our variational score distillation training objective.

Abstract: Score distillation sampling (SDS) has shown great promise in text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-saturation, over-smoothing, and low-diversity problems. In this work, we propose to model the 3D parameter as a random variable instead of a constant as in SDS and present *variational score distillation* (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed *ProlificDreamer*, can generate high rendering resolution (i.e., 512$\times$512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic.

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation
Zhengyi Wang 1.3, Cheng Lu"!, Yikai Wang', Fan Bao 1:3
Chongxuan Li Hang Su'r4, Jun Zhu 1,3,4
'Dept. of Comp, Sci. & Tech., BNRist Center, Tsinghua- Bosch Joint ML Center, Tsinghua University; ?Golling School of Artificial Intelligence, Renmin University of China, Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China ShhnngShu, Beijing, China; *Pazhou Laboratory (Huangpu), Guangzhou, China (wang-zy21, bf bfialls tsinghua. edu. en; lucheng. 1et.:gm115mmmmmcmmmm com; yikaiwwoutlook. com; chongxuanl aeteuannnirrrrs fruc. edu. cn; suhangss@tssinghua. edu. cn buszj@tsinghe edu.cn
Abstract
Score distillation sampling (SDS) has shown great promise in text-to-3D gen- eration by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-saturation, over-smoothing, and low-diversity problems. In this work, propose model the 3D parameter random variable instead of constant as SDS and present variational score distillation (VSD), a principled particle-based variational framework explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffu- sion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3I) such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well ex- plored. Our overall approach, dubbed ProlificDreammer, can generate high rendering resolution (i.e., 512 x 512) and high-fidelity NeRF with rich structure and com- plex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic. Project page: https:/ //mp. cs. tsinghua. edu. www.eeew.......eeeeeeee cn/prolitee iiiifiicddrreerrr
Introduction
3D content and technologies enable us to visualize, comprehend, and interact with complex objects and environments that are reflective of our real-life experiences. Their pivotal role extends across wide array of domains, encompassing architecture, animation, gaming, and the rapidly evolving fields of virtual and augmented reality. In spite of the extensive applications, the production of premium 3D content often remains formidable task. necessitates a significant investment of time and effort, even when undertaken by professional designers. This challenge has prompted the development of content based on textual descriptions, these innovative methods present promising way towards streamlining the 3D content creation process. Furthermore, they stand to make this process more accessible, potentially encouraging significant paradigm shift in the aforementioned fields.
text-to-31) methods [19 [31] 34 20 29 [6]. By automating the generation
'Equal contribution;
Corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Michelangelo style statue of dog reading news cellphone.
pincopple.
chimpanzee dressed Henry VIII king of Englond.
An elephant skull.
model house Tudor style.
A tarantula, highly detailed.
A snail on a leaf.
An astronout is riding horse.
(a) ProlificDreamer generate meticulously detailed and photo- photo-realistic 3D textured meshes.
red fire hydront
DSLR photo table with dim
A Notte pointing of costle made of cheesecale surrounded by a moot mode of crean.
DSLR photo Space Shuttle.
spraying water.
Inside of smart realistic detailed photo, 4k.
A DSLR photo of homburger inside restaurant.
(b) ProlificDreamer can generate high rendering resolution (i.e.. 512 512) and high-fidelin NeRF with rich structures and complex effects. Besides, the hottom results show that PromifieDreamer can generate complex scenes with 360 views because of initialization (see Sec.
A hotdog in tutu skirt. (c) ProlificDreamee generate diverse and semantically correct 3D scenes given the same text.
highly detailed sand castle.
Figure 1: Text-to-3I samples generated by ProlificDreamer from scratch. Our base model is Stable Diffusion and we do not employ any other assistant model or user-provided shape guidance (sce Table Dp. See our accompanying videos in our project page for better visual quality.

---

Diffusion models [46 14 49
have significantly advanced eext-to-image synthesis [36 41 38 I.
particularly when trained on large-scale datasets 43]. Inspired by these developments, DreamFu- sion [4 employs pretrained, large-scale text-to-image diffusion model for the generation of 3DD content from text in the wild, cireumventing the need for any 3D data. DreamFusion introduces the Score Distillation Sampling (SDS) algorithm to optimize a single 3D representation such that the image rendered from any view maintains a high likelihood as evaluated by the diffusion model, given the text. Despite its wide application [20] 4 29 [55], empirical observations [44] indicate that SDS often suffers from over-saturation, over-smoothing, and low-diversity problems, which have yet to be thoroughly explained or adequately addressed. Additionally, orthogonal elements in the design space for text-to-3D such as rendering resolution and distillation time schedule, have not been fully explored, suggesting a significant potential for further improvement. In this paper, present systematic study of all these elements to obtain elaborate 3D representations.
We first present Variational Score Distillation (VSD), which treats the corresponding 3D scene given a textual prompt as a random variable instead of single point as in SDS 34 VSD optimizes distribution of 3D scenes such that the distribution induced on images rendered from all views aligns as closely as possible, in terms of KL divergence, with the one defined by the pretrained 2D diffusion model (see Sec. 3.T. Under this variational formulation, VSD naturally characterizes the phenomenon that multiple 3D scenes can potentially align with one prompt. To solve it efficiently, VSD adopts particle-based variational inference [23 3 9], and maintains a set of 3D parameters as particles to represent the 3D distribution. We derive a novel gradient- based update rule for the particles via the Wasserstein gradient flow (see Sec. 3.2) and guarantee that the particles will be samples from the desired distribution when the optimization converges (see Theorem Our update requires estimating the score function of the distribution on diffused rendered images, which can be
efficiently and effectively implemented by a low-rank adaptation (LoRA) [8] 40] of the pretrained diffusion model. The final algorithm alternatively updates the particles and score function.
We show that SDS is special case of VSD, by using a single-point Dirac distribution as the variational distribution (see Sec. 3.3). D. This insight explains the restricted diversity and fidelity of the generated 3D scenes by SDS. Moreover, even with a single particle, VSD can learn a parametric score model, potentially offering superior generalization over SDS. We also empirically compare SDS and VSD in 2D space by using an identity rendering function that isolates other 3D factors. Similar to ancestral sampling from diffusion models, VSD is able produce realistic samples using a normal CFG weight (i.e,, 7.5). In contrast, SDS exhibits inferior results, sharing the same issues previously observed in text-to-3D, such over-saturation and over-smoothing
We further systematically study other elements orthogonal to the algorithm for text-to-3D and present a clear design space in Sec. Specifically, we propose high rendering resolution of 512 x 512 during training and an annealed distilling time schedule to improve the visual quality. We also propose scene initialization, which is crucial for complex scene generation. Comprehensive ablations in Sec. demoosttatt the effectiveness of all the aforementioned elements particularly for VSD. Our overall approach can generate high-fidelity and diverse 3D results. We term it ProlificDreamer""
As shown in Fig. and Sec. 5 ProlifiDDreamer can generate 512 512 rendering resolution and high-fidelity Neural Radiance Fields (NeRF) with rich structure and complex effects (e.g., smoke and drops). Besides, for the first time, DreaifieDreamer can successfully construct complex scenes with multiple objects in 360 views given the textual prompt. Further, initialized from the generated NeRF, Dreamfic can generate meticulously detailed and photo-realistic 3D textured meshes.
2 Background
We present preliminaries on diffusion models, score distillation sampling, and 3D representations.
involves forward process (4: ( to gradually add noise to a data point Dy ~~ qo (x())) and reverse process {pt (Pthefooa] to denoise/generate data. The forward process is defined by (a( == (on of I) and (ax))) == f (x (oo))n(( (wr)) )aro. where 0t, o, > O are hyperparameters satisfying on R 1, og Ri 0, on R 0, 0 R 1; and the reverse process is defined by denoising from p1 (m1) = (0, I) with parameterized noise prediction
Diffusion models. A diffusion model 46 14, 49
AA prolific dreamer is someone who experiences vivid dreams quite regularly [50] which corresponds to the high-fidelity and diverse results of our method.
3

---

network co(x,, t) to predict the noise added to a clean data which is trained by minimizing (Di = Beg (=)))())))) (())) (( + o,e) - =
where (t) a time-dependent weighting function. After training, we have ps. Re St and thus we can draw samples from Po R qu- Moreover, the noise prediction network can be used for approximating the score function of both qi and Pæ by Va, log ((x))) x Vx, (r = -c,( t)/o,.
One of the most successful applications of diffusion models is text-to-image generation 41 36, 38) where the noise prediction model E,(2,, 1, v. conditioned on text prompt y. In practice, classifier- free guidance (CFG [55) is a key technique for trading off the quality and diversity of the samples, which modifies the model by ég (2t, 1, y) == (1 (1+ &) L, y) sc s (24,1, f, Ø), where is a special "empty" text prompt representing for the unconditional case, and > 0 is the guidance scale. A larger guidance scale usually improves the text-image alignment but reduces diversity.
Text-to-3D generation by score distillation sampling (SDS) [34]. SDS is an optimization method by distilling pretrained diffusion models, also known as Score Jacobian Chaining (SJC) [55]. is with great promise. Given pretrained text-to-image diffusion model ( with the noise prediction network Epectrain (zy, t. y), SDS optimizes a single 3D representation with parameter 0 E O, where @ is the space of O with the Euclidean metric. Given a camera parameter e with distribution p(c) and a differentiable rendering mapping g(:, c) : Rd,, denote yf as the "view-dependent prompt" [34] (i.e., text prompt with view information), of (c) as the distribution at time t of the forward diffusion process starting from the rendered image g c) with the camera c and 3D parameter O. SDS optimizes the parameter 0 by solving
widely used in text-to-3D generation [34 55 20 29 55
min Sas (0) EE,e (or//a))(tt)D) (gl (x, ((all))))) DEG
(2)
where t ~ 4(0.02, 0.98), E ~~ NO, I). and 2 = (0, c) + dre. Its gradient is approximated by sDL Er,,,c (t Epretrain (x,,, t,y
Ög(6,c) c)
(3)
00
Notwithstanding this progress, empirical observations 34 show that SDS often suffers from over- saturation, over-smoothing, and low-diversity issues, which have yet to be thoroughly explained or adequately addressed.
3D representations. We employ NeRF 30 32] (Neural Radiance Fields) and textured mesh 455 as two popular and important types of 3D representations. In particular, NeRF represents 3D objects using multilayer perceptron (MLP) that takes coordinates a 3D space as input and outputs the corresponding color and density. Here, 0 corresponds to the parameters of the MLP. Given camera pose c, the rendering process g(,,c e) defined as casting rays from pixels and computing the weighted sum of the color of the sampling points along each ray to composite the color of each pixel. NeRF is flexible for optimization and is capable representing extremely complex scenes. Textured mesh 45 represents the geometry of 3D object with triangle meshes and the texture with color on the mesh surface. Here the 3D parameter consists of the parameters to represent the coordinates of triangle meshes and parameters of the texture. The rendering process g(8, c) given camera pose is defined by casting rays from pixels and computing the intersections between rays and mesh surfaces to obtain the color of each pixel. The textured mesh allows high-resolution and fast rendering with differentiable rasterization.
3 Variational Score Distillation
We now present Variational Score Distillation (VSD) (see Sec. 3.1 that learns to sample from a distribution of the 3D scenes. By using 3D parameter partieles to represent the target 3D distribution, we derive principled gradient-based update rule for the particles via the Wasserstein gradient flow (see Sec. 3.2p. We further show that SDS is special case of VSD and constructs an experiment in 2D space to study the optimization algorithm isolated from the 3D representations, explaining the practical issues of SDS both theoretically and empirically (see Sec. 3.3)
3.1 Sampling from 3D Distribution as Variational Inference
In principle, given a valid text prompt y, there exists probabilistic distribution of all possible 3D representations. Under 3D representation (c.g., NeRF) parameterized by 0, such distribution can

---

Back Propugation
A highly detailed sand castle.
Text to Image Diftusion
Pronin (.
c,
Sample
LoRt
Render 2D Imsage
dd Noise
Nelt//Mesh
I Back Propagation
Particles of NeRFiMesh
mit E2 ae <d
Add Noise
Overview of VSD. The 3D representation is differentiably rendered at random pose c. The rendered image sent to the pretrained diffusion and the score the variational distribution (estimated by LoRA) to compute the gradient of VSD. LoRA is also updated on the rendered image.
Figure 2:
be modeled as a probabilistic density (((ly). Denote (argle; c, y) as the (implicit) distribution of the rendered image 2o := 9(00 c) given the camera with the rendering function g(:, and denote pa y²) as the marginal distribution of & = defined by the pretrained text-to-image diffusion model with the view-dependent prompt y². To obtain 3D representations of high visual quality, we propose to optimize the distribution to align the rendered images its samples with the pretrained diffusion model in all views by solving
min DKL (16 (zule, v) || Po(zuly ()")
(4)
This is typical variational inference problem that uses the variational distribution di" (1olc, y) to approximate (distill) the target distribution (o
Directly solving problem (4) is hard because is rather complex and the high-density regions of Po may be extremely sparse high dimension (7]. Inspired by the success of diffusion models [4 49
we construct a series of optimization problems with different diffused distributions indexed by t. As increases T, the optimization problem becomes easier because the diffused distributions get closer to the standard Gaussian. We simultaneously solve an ensemble of these problems (termed as variational score distillation or VSD) as follows:
μ arg min R,,,, [(rr/on (() (ul' (meeee, v) (xxelt))))
(5)
where (x c. v) I g6 (30o c, y) (2 )dao and (a = po (o Pro (ax (aro dlao are the corresponding noisy distributions at time with the Gaussian transition Pio (at labo) = (wx of I). and (t) is a time-dependent weighting function.
Compared with SDS that optimizes for the single point O, VSD optimizes for the whole distribution μ., from which we sample 0. Notably, prove that introducing the additional KL-divergence for t> 0 VSD does not affect the global optimum of the original problem (4), as shown below.
Theorem (Global optimum VSD, proof Appendix C.4 For each > O, we have
DKL (g" (c, (r()) + d (2olc, y) = (o
(6)
3.2 Update Rule for Variational Score Distillation
To solve problem (5) a direct way be train another parameterized generative model for μt, but it may bring much computation cost and optimization complexity. Inspired by previous particle-based variational inference (3
methods, maintain 3D parameter (0) i= as particles and derive a novel update rule for them. Intuitively, we use {0] to "represent" the current distribution and
g(0) will be samples from the optimal distribution μ² if the optimization converges. Such optimization can be realized by simulating ODE w.r.t. 0, as shown in the following theorem.
Theorem 2 (Wasserstein gradient flow of VSD, proof in Appendix C.. Starting from an initial distribution noo denote the Wasserstein gradient flow minimizing problem (5) in the distribution (function) space at each time 2 0 as (17),20 with Pocc = μ². Then can sample 0,, from μty by
We optimize up particles due to the computation resource limit. See details in Appendix D.3
5

---

(a) SDS
CCGG 7.5)
(b) SDS
(CFG 100)
(c) Ancestral sampling 27
(CFG = 7.5)
(d) VSD (CFG = 7.5, ours)
Figure 3: Samples of different methods space, Similarly to ancestral sampling, VSD generates realistic images with common CFG weight of 7.5 and outperforms SDS significantly. The prompts from left right are hamburger, horse, and monster truck, respectively. See details in Appendix G
firstly sampling Po (oo(ll))) and then simulating the following ODE:
d0, dr
- [u(t)) -oxVa log (e 10) - -o. log of (ma c. u))
Og Og(Or, c)
00,
score of real images
score of noisy rendered images
where of is the corresponding noisy distribution at diffusion time w.r.t. at ODE time T.
According to Theorem we can simulate the ODE in Eq. 7 for large enough T to approximately sample from the desired distribution μr" The ODE involves the score function of noisy real im- ages and that of noisy rendered images each time r. The score function of noisy real images -a, Vn, log (*=l))) can be approximated by the pretrained diffusion model Epretmin ( L, y²). The score function of noisy rendered images -a, V., log / (.+)c,,) y) is estimated by another noise prediction network (2,, I, c, y), which is trained on the rendered images by {0(0) with the standard diffusion objective (see Eq. (I):
min E1- (oo (cs(arg(0 c)) t,,,,,,,, c
(8)
practice, we parameterize 6. by either a small U-Net 33 or a LoRA (Low-rank adaptation [8] 40)
of the pretrained model Epretrin (x,, t, ) and add additional camera parameter to the condition embeddings in the network. In most cases, we find that using LoRA can greatly improve the fidelity of the obtained samples (e.g., results in Fig. T.. We believe that it is because LoRA is designed for efficient few-shot fine-tuning and can leverage the prior information in Cpretrin (the information of both images and text corresponding y).
Note that at each ODE time т, we need to ensure e, matches the current distribution q Thus, we optimize Eg and g alternately, and each particle g(3) updated by ((3) < g(+) - 17 VeLvD ( (0(1).
where 1) > 0 is the step size (learning rate). According to Theorem 2 the corresponding gradient is
VoLvsD(O) Ex,,,c w(t) (Epretrain (3i, t, t,y") (x @g(0.c)
(9)
00
where x; co We show the approach of VSD in Fig. (sce pseudo code in Appendix
E
3.3 Comparison with SDS
We now systematically compare VSD with SDS in both theory and practice.
'Note that we have two variables of time: one is the diffusion time t E []] and the other is the gradient flow time corresponding the optimization iteration for each 0.

---

Table I: Design space of text-to-3D via 2D diffusion. We highlight the contributions of this paper that improve the fidelity, diversity and ability to generate complex scenes by f and respectively.
Method
DreamFusion
34
Magic3D 20
Fanfasia3D
4
Ours
NeRF Representation
Resolution
64
512
Backbone
MipNeRF330
Instant NGP 32
Instant NGP 22
Initialization
Object
Object
Object/ Scene initialization
NeRF Training
Base model
Imagen
eDiff-
Stable Diffusion 38
Number of particles
1~4
Distillation objective*t
SDS (Eq.
3
SDS (Eq. 100
VSD (F4.
CFG
100
7.5
Time schedule"
(4(00.2,0.98)
24(0.02, 0.98)
(0.02, 0.98) (4.002
Mesh Representation
Initialization
From NeRF Entangled
Handcrafted Disentangled
From NeRF
Texture and goometry
Disentangled
Mesh Training
Distillation objective**
SDS (Eg.
SDS (Eq.
VSD (Egq 9
CFG"
100
100
7.5
SDS special case of VSD. Theoretically, comparing the update rules of SDS (Eq. (3) and VSD (Eq. (2)). SDS is special case of VSD by using a single-point Dirac distribution μ((0z) = 6(0-0(1)
the variational distribution (see Appendix C.3 for derivation). In particular, VSD only employs potentially multiple particles but also learns a parametric score function even for single particle (i.e., n 1). Empirically, the learned neural network may potentially offer superior generalization ability over the Dirac distribution in SDS, thus it may provide more accurate updating directions in low-density regions. Moreover, by using LoRA, VSD can additionally exploit the text prompt y the estimation (2t,,, L, G, y). while the Gaussian noise used in SDS cannot leverage the information from y. Thus, VSD may provide samples which are more aligned with the prompt y-
VSD is friendly to CFG. As VSD aims to sample 0 from the optimal μa" defined by the pretrained model Epecrain the effects by tuning the CFG Epretrain for 3D sampling by VSD are quite similar to 2D sampling by the traditional ancestral sampling methods [14] 27]. Therefore, VSD can tune CFG as flexibly the classic text-to to-ttoogeeeee methods, and we use the same setting CFG (e.g. 7.5) as the common text-to-imape generation task for the best performance, To the best of our knowledge, this for the first time addresses the problem in previous SDS CFG (i.e., 100).
29 that usually requires large
VSD vs. SDS in 2D experiments that isolate 3D representations. To directly compare SDS and VSD, we consider a special case of the rendering function g(0) to decouple the optimization algorithm from 3D representations. In particular, we set g(0, c) = 0 for any c. Then the rendered image = g(, c) is the same 2D image as 0. In such case, optimizing the parameter is equivalent to generating an image in 2D space, thereby independent of the 3D representation. We show the results of different sampling methods in Fig. 3 SDS exhibits failure under both small and large CFG weights. Particularly with the default CFG weight (i.e., 100) used SDS, the 2D samples share the same issues previously observed in text-to-3D such as sver-saturation and over- smoothing 34.. contrast, VSD demonstrates flexibility in accommodating various CFG weights and produces realistic samples using normal CFG weight (i.e., 7.5), behaving similarly to ancestral sampling from diffusion models. See more details and analysis Appendix G
As other 3D factors are isolated in this comparison, these theoretical and empirical results suggest that the aforementioned practical issues of SDS [34] stem from the oversimplified variational distribution and large CFG employed by SDS. Such results strongly motivate us to employ VSD for text-to-3D generation, where still substantially and consistently outperforms SDS (see evidence in Sec
4 ProlifiDDreamer
We further present clear design space for text-to-3D in Sec. 4.1 and systematically study other elements orthogonal to the distillation algorithm in Sec. 4.2 4.2
Combining all improvements highlighted in Tab. [ we arrive at PromificDreamer, an advanced text-to-3D approach.
7

---

DreamFushi
Magic3D
Fantasia3D
Ours
A 3D model of an adorable cottage with thatched roof.
plate piled high with chocolate chip cookies. Figure 4: Comparison with baselines. Our results have higher fidelity and more details.
4.1 Design Space of Text-to-3D Generation
We adopt the two-stage approach 20 with several improvements the design space of text-to-3D generation as summarized in Table Specifically, in the first stage, we optimize a high-resolution (c.g., 512) NeRF by VSD utilize high flexibility for generating scenes with complex geometry. In the second stage, we use DMTet 45 to extract textured mesh from the NeRF obtained in the first stage, and further fine-tune the textured mesh for high- high-resolution details. The second stage is optional because both NeRF and mesh have their own advantages in representing 3D content and are preferred certain cases. Nevertheless, PremificDreamer can generate both high-fidelity NeRFs and meshes.
4.2 3D Representation and Training
We systematically study other elements orthogonal to the algorithmic formulation. Specifically, we propose high rendering resolution of 512 x 512 during training and an annealed distilling time schedule to improve the visual quality. We also carefully design a scene initialization, which is crucial for complex scene generation.
High-resolution rendering for NeRF training. We choose Instant NGP 32 for efficient high- resolution rendering and optimize NeRF with up to 512 training resolution using VSD. By applying VSD, we obtain high-fidelity NeRFs with resolutions varying from 64 to 512.
Scene initialization for NeRF training. We initialize the density for NeRF as (1)) = Aa (1 [ where X is the density strength, r is the density radius, and is the coordinate. For object- centric scenes, we follow the object-centric initialization used in Magic3D 20) with 2g 10 and 0.5; For complex scenes, we propose scene initialization by setting X -10 to make the density "hollow" and r 2.5 that encloses the camera. We show in Appendix [ that the help to generate high-fidelity complex scenes without other modifications
scene initialization to the existing algorithm. In addition, we can further add a centric object to the complex scene by using object-centric initialization for ||/||2 5/6 and scene initialization for others, where the hyperparameter 5/6 ensures the initial density function is continuous.
Annealed time schedule for score distillation. We utilize simple two-stage annealing of time step t in the score distillation objective, suitable for both SDS (Eq. 33) and VSD (Eq. ()). For the first several steps sample time steps (0.022 0.98) and then anneal into ~ (0.022 0.50). The key insight is that, essentially, we aim to match the original of (zolc, y) with po Po( The KL-divergence for larger can provide reasonable optimization direction during the early stage of training. During training, while x" is approaching the support of po (x" (g a smaller t can narrow the gap between pr (x=""yy" and po lv²), and provide elaborate details aligning with po na("" (a= ly").
Mesh representation and fine-tuning. We adopt a coordinate-basse hash grid encoder inherited from NeRF stage to represent the mesh texture. We follow Fantasia3D 4 to disentangle the

---

(I) 64 rendering
(2) + 512 rendering
(3) + Annealed t
(4) + VSD
Figure 5: Ablation study of proposed improvements for high-fide -iddfiidllity NeRF generation. The prompt is an elephant skull. (1) The common setting [34] [0) adopts 64 rendering resolution and SDS loss. (2) We improve the generated quality by increasing the rendering resolution. (3) Annealed time schedule adds more details to the generated result. (4) VSD makes the results even better with richer details.
optimization of geometry and texture by first optimizing the geometry using the normal map and then optimizing the texture. our initial experiments, we find that optimizing geometry with VSD provides more details than using SDS. This may be because the mesh resolution is not large enough to represent nigh-frequency details. Thus, we optimize geometry with SDS for efficiency. But unlike Fantasia3D [4) our texture optimization is supervised by VSD with CFG 7.5 with the annealed time schedule, which can provide more details than SDS.
Experiments
5.1 Results of ProlificDreamer
We show the generated results of ProlificDreamee in Fig. [(a] and [(b) including high-fidelity mesh and NeRF results. All the results are generated by VSD. For all experiments without mentioned, VSD uses 1 particle for fair comparison with SDS. In Fig. [[c]
we also demonstrate VSD can generate diverse results, showing that different particles in a round are diverse (with n = 4).
Object-centric generation. We compare our method with three SOTA baselines, DreamFusion 34 Magic3D (20 and Fantasia3D [ All of the baselines are based on SDS. Since none of them open-sourced, we use the figures from their papers. As shown in Fig. 4 ProlificDreamer generates 3D objects with higher fidelity and more details, which demonstrates the effectiveness of our method.
In appendix, we add user study (Section JT and quantitative results (Section K of ProaificDreamer against the baselines to demonstrate the effectiveness of our method.
Large scene generation. As shown in Fig. l(b) our method can generate 360 scenes with high- fidelity and fine details. The depth map shows that the scenes have geometry instead of being 360 textured sphere, verifying that with our scene initialization alone we can generate high-fidelity large scenes without much modification existing components. See more results in Appendix B
5.2 Ablation Study
Ablation on NeRF training. Fig. 5 provides the ablation on NeRF training. Starting from the common setting (34 20 with 64 rendering resolution and SDS loss, we ablate our proposed improve- ments step by step, including increasing resolution, adding annealed time schedule, and adding VSD all improve the generated results. It demonstrates the effectiveness our proposed components. We provide more ablation on large scene generation in Appendix D, with a similar conclusion.
Ablation on mesh fine-tuning. We ablate between SDS and VSD on mesh fine-tuning, shown in Appendix D Fine-tuning texture with VSD provides higher fidelity than SDS. As the fine-tuned results of textured mesh highly dependent the initial NeRF, getting high-qually NeRF the first stage is crucial. Note that the provided results of both VSD and SDS in mesh fine-tuning are based on and benefit from the high-fidelity NeRF results in the first stage by our VSD.
blation on CFG. We perform ablation to explore how CFG affects generation diversity. We find that smaller CFG encourages more diversity. Our VSD works well with small CFG and provides

---

considerable diversity, while SDS cannot generate plausible results with small CFG (e.g., 7.5), which limits its ability generate diverse results. Results and more details are shown in Appendix
6 Related Works
Diffusion models.
Score-based generative model 49] and diffusion models 14 have shown
great performance in image synthesis [14] 71. Recently, large-scale diffusion models have shown
great performance in text-to-image synthesis [36 38 which provides an opportunity to utilize
for zero-shot text-to-3D generation.
Text-to-3D generation.
DreamField [19] proposes a text-to-3D method using CLIP 35 guidance. DreamFusion 3 proposes a text-to-3D method using 2D diffusion models. Score Jacobian Chaining (SJC) 155 derives the training objective of text-to-3D using a 2D diffusion model from another theo- retical basis. Magic3D 20 extends text-to-3D to higher resolution with mesh 45 representation. Latent-NeRF [29] optimizes NeRF latent space. Fantasia3D 4 optimizes a mesh with DMTet [45] from scratch. Although Fantasia3D achieves remarkable zero-shot text-to-3D generation, it requires user-provided shape guidance for the generation of complex geometry. 17 propose score debiasing and prompt debiasing to mitigate multiface problem and is orthogonal our work. TextMesh (52) contemporary with us and proposes a different pipeline for high-fidelity text-to-3I) generation. 3DFuse [44] proposes incorporate 3D awareness into 2D diffusion for better 3D consistency in text-to-3D generation. In addition, adjusting time schedule has also been discussed in previous works 12 [55] However, previous works either require carefully devising the schedule [22 or perform inferior to the simple random time schedule [55]. Instead, our 2-stage annealing schedule is easy to train and achieves better generation quality than the random time schedule.
Text-driven large scene generation. Text2Room [66 generates indoor rooms from a given prompt. However, it additional monocular depth estimation models prior, and we do not any additional models. Our method generates the wild text prompt and uses only a text-to-image diffusion model. Set-the-scene [5] is contemporary work with us aimed large scene generation with different pipeline. Overall, ProlificDreamer uses an advanced optimization algorithm, i.e, VSD with our proposed two-stage annealed time schedule, which has a significant advantage over the previous SDS SJC (see Appendix C.3 for details). As result, ProlificDreamer achieves high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops) and photo-realistic mesh results.
7 Conclusion
In this work, we systematically study the problem of text-to-3D generation. In terms the algorithmic formulation, we propose variational score distillation (VSD), principled particle-based variational framework that treats the 3D parameter as a random variable and infers its distribution. VSD naturally generalizes SDS in the variational formulation and addresses the practical issues of SDS observed before. With other orthogonal improvements to 3D representations, our overall approach, ProlificDreamer, can generate high-fidelity NeRF and photo-realistic textured meshes.
Limitations and broader impact. Although ProlifiDDreamer achieves remarkable text-to-3D results, the generation takes hours of time, which is much slower than image generation by diffusion model. Although large scene generation can be achieved with our scene initialization, the camera poses during training are regardless of the scene structure, which may be improved by devising adaptive camera pose range according to the scene structure for better -etter-generate details. Moreover, due to the limited expressiveness of the base 2D model, the generation for complex prompts may fail, and the generated samples may have multi-face Janus problem [57] in some cases because the poor text-image alignment for the view-dependent prompts. In addition, content creation by generative models may cause harmful social impacts on the labor market. Also, like other generative models, method may be utilized to generate fake and malicious contents, which needs more caution.
Acknowledgement
This work was supported by NSF of China Projects (Nos. 62061 36001, 61620106010, 62076145, U19B2034, U1811461, A19A2081, 6197222); Beijing Outstanding Young Scientist Program NO.
10

---

BJJWZYYH012211100020999 grant from Tsinghua Institute for Guo Qiang; the High Performance Computing Center, Tsinghua University; and the Research Funds of Renmin University of China (22XNKJ13). J.Z was also supported by the XPlorer Prize. C. Li was also sponsored by Beijing Nova Program (No. 20220484044).
References
[1] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu, ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers, 2023.
[2] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVP Conference Compuler Vision and Patlern Recognition, pages 5470-5479, 2022.
[3] Changyou Chen, Ruiyi Zhang, Wenlin Wang, Bai Li, and Liqun Chen. unified particle- optimization framework for scalable bayesian sampling. arXiv preprint arXiv: 2018.
[4] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303. 2023.
[5] Dana Cohen-Bar, Elad Richardson, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Set- the-scene: Global-local training for generating controllable nerf scenes. arXiv preprint arXiv:2303.13450, 2023.
[6] Gianluca Detommaso, Tiangang Cui, Youssef Marzouk, Alessio Spantini, and Robert Scheichl. stein variational newton method. Advances in Neural Information Processing Systems, 31, 2018.
[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[8] Lauro Langosco Langosco, Vincent Fortuin, and Heiko Strathmann. Neural variational gradient descent. arXiv preprint arXiv:2107.1073/ 2021.
[9] Hanze Dong, Xi Wang, Yong Lin, and Tong Zhang. Particle-based variational inference with preconditioned functional gradient flow. arXiv preprint arXiv:2211.13954. 2022.
[10] Yihao Feng, Dilin Wang, and Qiang Liu. Learning to draw samples with amortized stein variational gradient descent. arXiv preprint wxXiv:7777.06620 2017.
[11) Clement Fuji Tsang, Maria Shugrina, Jean Francois Lafleche, Towaki Takikawa, Jiehan Wang, Charles Loop, Wenzheng Chen, Krishna Murthy Jatavallabhula, Edward Smith, Artem Rozant- sev, Or Perel, Tianchang Shen, Jun Gao, Sanja Fidler, Gavriel State, Jason Gorski, Tommy Xiang, Jianing Li, Michael Li, and Rev Lebaredian. Kaolin: pytorch library for accelerating 3d deep learning research. https::/ggis //tt:////ttttbt. wow.///ViDD.Gaaaaoooossseeeeeeee
2022.
[12] Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models plug-and-play priors. arXiv preprint arXiv:2206.09M 09012. 2022.
[13] Yuan-Chen Guo, Ying-Tian Liu, Chen Wang, Zi-Xin Zou, Guan Luo, Chia-Hao Chen, Yan- Pei Cao, and Hon Zhang. threestudio: A unified framework for 3d content generation.
https: thub. com/ wwop/hhreestuiiooo htwpai//oppooocce//eeetsttttaati 2023.
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances
in Neural Information Processing Systems, -8551, 2020.
[15] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint a7X1v 2022.

---

[16] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models, arXiv preprint arXiv:2303.1198) 2023.
[17] Susung Hong, Donghoon Ahn, and Seungryong Kim. Debiasing scores and prompts of 2d diffusion for robust text-to-3d 33 generation. arXiv preprint arXiv:2303.1541. 2023.
[18] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
2021.
[19] Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Picter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields, 2022.
[20] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Y Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint orXiv:2211.10440 2022.
[21] Chang Liu, Jingwei Zhuo, Pengyu Cheng, Ruiyi Zhang, and Jun Zhu. Understanding and accelerating particle-based variational inference. In International Conference Machine Learning, pages 4082-4092. PMLR, 2019.
[22] Chang Liu, Jingwei Zhuo, Pengyu Cheng, Ruiyi Zhang, Jun Zhu, and Lawrence Carin. Ac- celerated first-order methods on the wasserstein space for bayesian inference. stat, 1050:4, 2018.
[23] Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29, 2016.
[24] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- -:: Zero-shot one image 3d object, 2023.
(25) Chang Liua and Jun Zhub. Geometry in sampling methods: A review manifold meme and particle-based variational inference methods. Advancements in Bayesian Methods and Implementationss 47:239, 2022.
[26] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint 1711v:171..051000 2017.
[27] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprins 20X.0:2206.00927, 2022.
[28] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models, 2023.
[29] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or Or. Latent-nerf for chape-guided generation of 3d shapes and textures. arXiv preprint wwXiv:2211.07600 2022.
[30] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[31] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 Conference Papers, pages 1-8, 2022.
[32] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1- 15, 2022.
[33] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint 1rXiv:1810.0000 2018.
12

---

[34] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. Xrviv preprint (etv:2099.4988. 2022.
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[36] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[37] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1623 1637, 2020.
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[39] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234 241. Springer, 2015.
[40] Simo Ryu. Low-rank adaptation for fast text-to-image diffusion fine-tuning. https: //github com/ homm/cloonofssin/oora 2023.
[41] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic sot-to--mmae diffusion models with deep language understanding. arXiv preprint arXiv:2205. 2022.
[42] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models, 2022.
[43] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
[44] Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin- Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint arXiv:2303. 077337, 2023.
[45] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yo Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[46] Jascha Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages
PMLR, 2015.
[47] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances neural information processing systems, 32, 2019.
[48] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garneut, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[49] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations, 2021.
[50] Harold Stewart. The experiencing of the dream and the transference. The International Journal of Psycho-Analysis, 54:345, 1973.
13

---

[51]
Jiaxiang Tang.
Text-to-3d with stable-diffusion, 2022.
[52] Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. arXiv preprint arXiv:2300.12439, 2023.
[53] Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
[54] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-arr diffusion models. https
//githu.. com hugg ace/ /ifffusers 2022.
[55] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:22/2.00774. 2022.
[56] Ziyu Wang, Tongzheng Ren, Jun Zhu, and Bo Zhang. Function space particle optimization for bayesian neural networks. arXiv preprint arXiv:1902.09754 2019.
[57] Wikipedia contributors. Janus Wikipedia, the free encyclopedia. https://en. www.enwwwiiiiiiiie org/w/inden wttpieex..pppptiiteeeeee php?title=Janus 2023.
[58] Jingbo Zhang, Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao. Text2nerf: Text-driven 3d scene generation with neural radiance fields, 2023.
14

---

A Additional Experiment Results of 3D Textured Meshes
A delicious croissant.
A sliced loaf of fresh bread.
small saguaro cactus planted in a clay pot.
A blue tulip.
plate of fried chicken and waffles with maple syrup on them.
A cauldron full of gold coins.
rabbit, animated movie character, high detail 3d model.
lionfish.
Figure More results of ProlificDreamer of 3D textured meshes.
We provide more results of ProlificDreamer of 3D textured meshes in Fig. and Fig
15

---

A DSLR photo of an imperial state crown of England.
A car made out of sushi.
rotary telephone carved out of wood.
A marble bust of mouse.
A Matte painting of castle made of cheesecake surrounded by a moat made of ice cream.
A typewriter.
A plush dragon toy.
A praying mantis wearing roller.
Figure More results of PromificDreamer of 3D textured meshes.
16

---

Ows
ream Fusion
An old vintage car.
A baby bunny sitting top of stack of pancakes.
banana peeling itself.
A delicious croissant
A blue standing large basket of rainbow macarons.
Figure 8: Results of high quality NeRF from ProlificDreamee Compared with DreamFusion, ProlificDreamer generates better NeRF results in terms fidelity and details, which demonstrates the effectiveness of our VSD against SDS.
B Additional Experiment Results of 3D NeRF
Here, we provide more generated results in Fig. of high quality NeRF. We compare with Dream- Fusion, DreamFusion also uses NeRF representation. can be seen from the figure that Pro- lificDreamer generates better NeRF results in terms of fidelity and details, which demonstrates the effectiveness of VSD against SDS.
In Fig. 9
we provide more results large scenes with 360 environment using our scene initialization.
In Fig
10 provide more results of multiple particle experiments.
C Theory of Variational Score Distillation
C.1 Particle-based Variational Inference
aims to draw (particle) samples from a desired distribution by minimizing the KL-divergence between particle samples and the desired distribution, and is widely-used in Bayesian inference 23 T0 21 Specifically, denote P(O) as the set containing all the distributions on a support space with Euclidean distance, and (2( {μ P(0)= 30, θ Ey(o) [||@ 0u +oo} the 2- 2asssersteiin space [53) equipped with the Wasserstein 2-distance
Particle-based variational inference (ParVI) (23 6, 56 25
53
Given desired distribution μ² P()), Pa
17

---

Figure 9: Result of high quality NeRF from ProlificDreamer, large scene with 360 environment using our scene initialization. The prompt here is Small lavender isometric room, soft lighting, unreal engine render, voxels.
car made out of sushi.
A tarantula, highly detailed.
Figure 10: More results of multi cunti-particle experiments in NeRF
starts with some (n) particles (0)*) y sampled from an initial distribution pi E P(O) and update these particles with vector field v-(-) on θ by dofa) v,(( )d7 at each time 2 O, such that the distribution μr, of {o()) ) converges to μ² as T oo and n + cC. Typically, the measure My follows the "steepest descending curves" of functional on W2(O), which is known as the Wasserstein gradient flow. A common setting to define the functional je, as the KL-divergence DRL(UT 1x)), and then μr, follows 0-p² = - Vo Vu log ( and the corresponding vector field is v (07) = Vo log H.. However, directly computing the vector field is intractable because it is non-trivial to compute the time-dependent score function Vp log μ, (0). To tackle this issue, traditional PI methods either restrict the functional gradient within RKHS and leverage analytical kernels to approximate the vector field 23 3 22), or learn neural network to estimate the vector
field [8; but all of these methods are hard to scale up to high-dimensiona data such as images.
C.2 VSD as Particle-based Variational Inference
In this section, we formally propose the theoretical guarantee of VSD.
Denote (O) as the 2-Wasserstein space 53 of the probability distributions defined on the parameter space θ. For any distribution E (2( with corresponding random variable 0 ~ u(Oly), denote the implicit distribution for 2o := g0, c) as 4 (xolc, y). and let qf (æx(c, y) = [ (xolc, ()
be the marginal distribution of the random variable diffusion process. We want to optimize the distribution ((0lly) in the function space W2(O) by distilling pretrained diffusion model pr (yy))) via minimizing the following functional of μc:
during the
min Eip4] := Er,,c (9W
Ti
(f pe(arle)))
(10)
az
18

---

Compared with the vanilla SDS in Eq. 2 which optimizes the parameter & in the parameter space θ here we aim to optimize the distribution (measure) ((fly) in the function space (2( Inspired by previous Pa techniques [2 3 56 we firstly derive the gradient flow minimizing El E(u] in (2( and the update rule of each particle ~ ju()), as shown in the following theorem.
Theorem 3 (Wasserstein gradient flow of score distillation). Starting from an initial distribution no (0l)) the gradient flow {u+ 2-20 minimizing Elu-] on (2(@) at each time T 2 0 satisfies
(0w() Vo | (0l0))
0g c)
(ow((( (Va, log (m (v Van lo (axele, v))
Ôr
00
and the corresponding update rule for each O ~ pe satisfies
d0, B,, w(t)
0g(0,, c)
of Vo, log Pa
(-or V a log s."h (2ale,
(12)
dr
00,
Reepremmin (.3))
Ree (21,,c))
Theorem 3 shows that by letting the random variable 0 ~ (0- (y) move across the ODE trajectory in Eq. 12. its underlying distribution /4- will move by the direction of the steepest descent that minimizes Elp]] Therefore, to obtain samples (in θ) from μ² = arg min, Elu), we can simulate the
ODE in Eq. (12) by estimating two score functions Vn, log p. (ax (n) and Vn, log oh" (anale, v) at each ODE time T, which corresponds to the VSD objective in Eq.
C.3 Discussions of SDS SJC and VSD
There are three main differences between our proposed VSD and previous SDS 34 SJC [55 (1) VSD can optimize multiple (n 2 1) 3D objects, while SDS / SIC only optimizes a single (n = 1) 3D object. (2) VSD can generate high-fidelity results with the common (e.g.. 7.5) CFG (for both n = 1 and n > 1), while SDS SJC needs extremely large (e.g. 100) CFG. (3) SDS SJC is special case
of VSD by using single-point Dirac distribution /(0ly) Rs 5(0 g()) as the variational distribution.
SDS / SJC as special case of VSD. Here we explain SDS SJC under our variational infer- ence framework. If we directly approximate p(Oly) s 5(0 - 0()) by its empirical distribu- tion with the single sample ((1), then we have qf (#r|c, y) (( (a)g((()) ((())))),) .c), of and thus -o,Va, log of (assce, . y) ( (00) c))/o,- -0 E, which recovers the vanilla SDS / SJC. There- fore, SDS / SJC is a special case of VSD for the underlying distribution ((0ll)) by the empirical distribution with a single point. Such an approximation has no generalization ability for 0 * Q(1), and thus the updating direction by the score function -, Va, log of (ærlcc y) (or equivalently, the Gaussian noise E) may be rather bad at low-density regions, resulting poor samples for the final ((1), Instead, VSD regards g(1) as samples from the underlying distribution (u(lly) and trains neural network es to approximate the corresponding score functions, thus can leverage the generalization ability of neural networks for better approximating the underlying distribution (0ll)). Moreover, by using LoRA, VSD can additionally exploit the text prompt y in the estimation (1t,, f, c, y), while the Gaussian noise E used in SDS cannot leverage the information from y.
SDS / SJC as mode-seeking, VSD as sampling. VSD aims to sample from the optimal μ², while SDS / SJC aims to find the optimal 0* minimizing the objective in Eq. @ As the global optimum 8" in SDS / SJC is the mode of Eq. 2 SDS/ / SJC is also known as performing mode-seeking [34. To demonstrate the differences between sampling and mode-secking. we consider a special case of the rendering function (0) to decouple the optimization algorithm and the 3D representation.
In particular, we set g00 c) = 0 for any c and 0 E Rd. then the rendered image x = g(0, c) = is the same 2D image 0. In such case, we have of μ,, and it is easy to prove that μ² = Po (according to Theorem D For VSD, sampling / from μ is corresponding to the traditional ancestral sampling (27 from Pui while for SDS / SJC, we have VoLsus (0) IE, [w))( [t()))peettainn (x,, t, o Re E Vt) log Pa (z>lv²)) and thus the optimal 6" is the mode of the "averaged likelihood" of Pt for all t. However, it is common that the mode of deep generative models may have poor sample quality [33]. We show in Fig. 3 that under the same CFG (7.5), both VSD and ancestral sampling can generate good samples but the sample quality of SDS is quite poor.
SDS SJC requires large CFG, while VSD is friendly to CFG. As VSD aims to sample from the optimal μº defined by the pretrained model Epectrin. the effects by tuning the CFG in Epretrai for
19

---

3D samples 0 by VSD are quite similar to the effects for the 2D samples by the traditional ancestral sampling [4 17 Therefore, VSD can tune CFG as flexibly as the classic text-to-image methods, and we use the same setting of CFG (e.g. 7.5) as the common text-to- ext-to-imaae generation task for the best performance. To the best of our knowledge, this for the first time addresses the problem in previous that it usually requires a large CFG (i.e., 100). Specifically, SDS (Eq. (3) uses E) while VSD (Eq. ())) uses (Epretrain (Ze, f, y²) - -E c, y)). For example, O. we have VoLSDS(O)) VaLvSD (0) - e ( (2t, 1, y²) - Eo (2r, i, C, y))l Intuitively, to obtain highly detailed samples, the updating direction for 8 needs to be "fine" and "sharp". As SDS only depends Epecrain needs large CFG (= 100) to make sure Eprctrii to be "sharp" enough; however, large CFG, in turn, reduces the diversity of the results and also hurts the quality. Instead, VSD leverages an additional score function eg ( C, y) to give more elaborate direction than the zero-mean Gaussian noise, be rather "fine" and "sharp" due the difference between Epretrain
SDS (34 20 4 29]
(Epretrain (x,, t, 1f)
for the 2D special case of g(,cc)
E (E) (æ t, yr) and
and the updating direction and Eg. We empirically find that VSD can obtain much better quality than SDS both 2D and 31) generation.
C.4 Proof of Main Theorem
C.4.1 Proof of Theorem
1
Proof of Theorem 7 Denote æ as the random variable following db (wolc, y) and ay as the random variable following & (x,1c, y). By the definition of qi' have
x
0:500 + ore,
(13)
where e ~ (N0, I). Therefore, the characteristic functions of qf and 4 satisfy
=33² Put (s) Put (ats) (or(o.((rrs)== = - Fop (ars).
(14)
Similarly, the characteristic functions of pt and po satisfy
o7.2
= (5) (s)
(ots).
(15)
Therefore, have
DKL
(z))c,0) (xxlcc)))) p.(x.l))) = + of = + Pptt
pp
+ PPo5
pmo
* % = Po
(16)
C.4.2 Proof of Theorem
As the gradient flow minimizing E(4] on (2( satisfies
0pr
- V.. (ur Vo
[E[Hr]
(17)
8j17
thus we only need to compute the first variation 6g[m]
We propose the following lemmas for
computing 6E4u] 6y
Lemma 1. For P.q E (IR"), for any E Rdd dDL ll p)
1 [x]] log g(x) - ((x)( +
(18)
8q
Proof.
This is a classic conclusion in particle-based variational inference (e.g., see Sec.3.2. in
3).
Lemma 2. For a fixed e and Xt.
ōgf' (welc. [0] (2)(2x))0)) (
(19)
8p
where v g((,c) c).
20

---

Proof of Lemma 2 By the definition of of we have
of (ax c. y) = oog (wolow)) (m [coo]] = (((l)) (200
(20)
so
jgf (2r1c,, v) [0] goo = =N(
(21)
Lemma 3.
öDRL(hh (2x)c, (xxlcc)) Pe (1)) (0) EE (log (wvle,,)))) logp((>.lv)) 1},
(22)
where a.g(0,c) + o N(E
Proof of Lemma we have
By the chain rule of functional derivative, according to Lemma and Lemma 2
8DL (yt (1cle, v) ll Pe (v0)
8DKL |l Pe),
dut (aell,, v)
[0)
[ara]
(0)d.:
(23)
8p
8at
/ (log df (xele,,)) - logge( + (22e0(22
(24)
(a [log of (x..c, y) log (2)e))))) 1]
(25)
Eg [log of (zelc, v) (ogpo + 1],
(26)
where (ee I), 20 =g(0,c). og + o.e.
Below we provide the proof of Theorem
Proof of Theorem 3 According to Lemma have
[E[p]
I w(() (log di' (xxl,,w) logga(x:l1))) +1]
[0]
Er
(27)
8p
an
where a0g(0,0 c) o,e. So
|E1 V
[0] Bi,,,
w(((t (Vx, log of (ærlc,)) Vx, log (x+ll
Te
Ox,
(28)
ōf
00
(g(
Ey,,cc
orw(l) (Va, log of (xblc, Va, log log
(29)
00
Thus, the measure Mr step during the Wasserstein gradient flow minimizing Elu] W2( satisfies
Öh+ = --V --w,
(30)
0r
8E [PT]]
Vo
pr v
(31)
8p1
(ow(t)) (Va, log of (xzle, v) Va. log log
0g(0, c)
V.
Mr (fly))
80
(32)
By the definition of Fokker-Plance equation, the corresponding process of each particle 0, at time satisfies
d0,
0g(0,
Er,,,
aww(t) (Vx, log Va, log th" (Ix v))
(33)
dr
00
21

---

D Additional Ablation Study
D.1
Ablation Study on Large Scene Generation
Here we perform an ablation study on large scene generation to validate the effectiveness of our proposed improvements. We start from 64 rendering resolution, with SDS loss and our initialization. The results are shown in Figure [T] It can be seen from the figure that, with our scene initialization, the results are with 360 surroundings instead of being object-centric. Increasing rendering resolution is slightly beneficial. Adding annealed time schedule improves the visual quality of the results. Replacing SDS with VSD makes the results more realistic with more details.
SDS 64 Rendering Scene Initialization
+ +512 Rendering
A Anealed Time Schedule
+VSD
Depth
Figure 11: Ablation study on scene generation. With our scene initialization, the results are with 360 surroundings instead of being object-centric. Our annealed time schedule and VSD are both beneficial for the generation quality. The prompt here is Inside of a smart home, realistic detailed photo, 4k.
D.2 Ablation Study on Mesh Fine-tuning
Here we provide an ablation study on mesh fine-tuning. Fine-tuning with textured mesh further improves the quality compared to the NeRF result. Fine-tuning texture with VSD provides higher fidelity than SDS. Note that both VSD and SDS in mesh fine-tuning are based on and benefit from the high-fidelity NeRF results by VSD. And it's crucial get high-quality NeRF with VSD the first stage.
D.3 Ablation on Number of Particles
Here provide ablation study on number of particles. We vary the number of particles in 1, 4, and examine how the number of particles affects the generated results. The CFG of VSD is set as 7.5. The results are shown in Fig. [3] As shown in the figure, the diversity the generated results is slightly larger as the number of particles increases. Meanwhile, the quality of generated results is not affected much by the number particles. Owing to the high computation overhead to optimize computation resources, we now only test at most 8 particles. We
representations and limitations provide experiment with 2048 particles Appendix G demonstrate the scalability of VSD. We leave the experiments of more particles in 3D as future work.
22

---

(1) High fidelity NeRF generated by VSD (ours).
(2) Extract and fine- tune geometry from NeRF.
(3a) Texture fine-tuned by (3b) Texture fine-tuned by SDS.
VSD (ours).
Pipeline of ProlificDreamer along with ablation study of VSD. After generating high- quality NeRF, we extract and finetune a textured mesh. VSD provides high-fidelity texture, while SDS tends to generate smoother results. Note that both VSD and SDS in mesh fine-tuning are based on and benefit from the high-fidelity NeRF results by our VSD. The prompt here is an elephant skull.
Figure 12:
Algorithm 1 Variational Score Distillation
Input: Number of particles n (2 1). Large text-to-image diffusion model Epectrain- Learning rate 771 and 1/2 for 3D structures and diffusion model parameters, respectively. A prompt y/
initialize 3D structures {0(0) J a noise prediction model Eø parameterized by Φ.
2: while not converged do
3: Randomly sample 0 ~ {0(0) ) and a camera pose c.
4: Render the 3D structure pose get a 2D image œo g(0,c).
5: 0 m Er,,c (())) (Cpremmin (zt, t, y-) ca t, c, (g().c)
6: o Φ 72 V Veeee,, c, y) cllB-
7: end while
8: return
D.4 Ablation on Rendering Resolution
Here we provide an ablation study on the rendering resolution during NeRF training with VSD. As shown in Fig. [4] training with a higher resolution produces better results with finer details. In addition, our VSD still provides competitive results under lower training resolution (128 or 256), which is more computationally efficient than the 512 resolution.
E Algorithm for Variational Score Distillation
We provide summarized algorithm of variational score distillation in Algorithm [
We initialize one several 3D structures { a noise prediction model Co. parameterized by d. At each iteration, we sample camera pose from pre-defined distribution as previous works [34 20 Then we render 2D image from 3D structures at pose with differentiable rendering 27g = g(, c). optimize 3D parameters 0, we compute the gradient direction 2D image and then back propagate the parameter of NeRF, using VSD as Eq. (9) To model the score of the variational distribution, we train an additional diffusion model Es parameterized by LoRA. We optimize Eq. (8) to train LoRA after optimization of 3D parameters, using the rendered image 3o gl, c) and pose Note that, at each iteration, we perform differentiable rendering only once but use the rendered image twice for both computing gradient direction with VSD and training LoRA. Thus the computation cost will not increase much compared to SDS.
F More Details on Implementation and Hyper-Parameters
Training details. We use v-prediction 42 to train our additional diffusion model 6g. The camera pose fed into a 2-layer MLP and then added to timestep embedding at each U-Net block. NeRF/mesh and LoRA batch sizes are set 1 owing to the computation limit. A larger batch size of
23

---

1 particle
Seed 1
Seed
Seed 3
Seed 4
Seed 5
Seed
Seed 7
Seed 8
2 particles
Sead
Seed 2
Seed
Sood 4
4 particees
Seed 1
Seed 2
8 particles
Seed
Figure 13: Ablation on how number of particles affects the results. The diversity of the generated results is slightly larger the number of particles increases. The quality of generated results is affected much by the number of particles. The prompt is A high quality photo of an ice cream sundae.
NeRF/mesh may improve the generated quality and we leave it in future work. The learning rate of LoRA is 0.0001 and the learning rate of hash grid encoder is 0.01. We render in RGB color space for high-resolution synthesis, unlike 29 555 that render in latent space. In addition, we set w(t) = of For most experiments, we only use 1 particle for VSD reduce the computation time (and only use a batch size of 1, due to the computation resource limits). For NeRF rendering, we sample 96 points along each ray, with 64 samples in coarse stage and 32 samples in fine stage. We choose single-layer MLP to decode the color and volumetric density from the hash grid encoder as previous work (20).
For object-centric scenes, we set the camera radius as (1110, 1.5). The bounding box size set as 1.0. For large scene generation, enlarge the range of camera radius U(..1, 2.3) for better details

---

(1) 64 rendering.
(2) 128 rendering.
(3) 256 rendering.
(4) 512 rendering.
Figure 14: Ablation study rendering resolution during NeRF training. Training with higher resolution produces better results with finer details. However, our VSD still provides competitive results under lower resolution (128 or 256), which is more computationally efficient. The prompt here is an elephant skull.
and geometry consistency. We enlarge the bounding box size to 5.0. For large scenes with centric object, we set the range of camera radius as 2/(1.0, 2.0).
DreamFusion [34] introduces explicit shading model based normal vectors during the training process to enhance the geometry. We currently disable the shadings and reduces the computational cost and memory consumption. We leave incorporating the shading model into our method future work.
Optimization. We optimize 25k steps for each particle with AdamW (26 optimizer in NeRF stage. We optimize 15k steps for geometry fine-tuning and 30k steps for texture fine-tuning, At each stage, for first 5k steps we sample time steps (0.002 0.98) and then directly change into ( ~ (0.002, 0.50). For large scene generation, we delay the annealing time to 10k: steps since large scene generation requires more iterations to converge. The NeRF training stage consumes 17/17/18/27GB GPU memory with 64/ 12/ /256/ 5122 rendering resolution and batch size of 1. The Mesh fine-tuning stage consumes around 17GB GPU memory with 512 rendering resolution and batch size of 1. The whole optimization process takes around several hours per particle on single NVIDIA A100 GPU. We believe adding more GPUs in parallel will accelerate the generation process, and we leave it for future work.
Licenses Here provide the URL, citations and licenses of the open open-sourced assets we use in this work.
Table 2: URI,, citations and licenses of the open-sourced assets we use in this work.
URL
Citation
License
3T
Apache License 2.0
Apache License 2.0
38
MIT License
Apache License 2.0
http://gthau..cooonnnggggffgfffo
Apache License 2.0
G 2D Experiments of Variational Score Distillation
Here we describe the details of the experiments on 2D images with Variation Score Distillation in the main text of Fig. 3 Here we set the number of particles as 8. We train smaller U-Net from scratch estimate the variational score since the distribution of several 2D images is much simpler than the distribution of images rendered from different poses of 3D structures. The optimization takes around 6000 steps. Additional U-Net is optimized step on every optimization step the particle images. The learning rate of particle images is 0.03 and the learning rate of U-Net is 0.0001. The batch size of particles and U-Net are both set 8. Here we do not use annealed time schedule for both VSD and SDS for fair comparison.
25

---

To further demonstrate the effectiveness of our VSD, we increase the number of particles to large number of 2048. The optimization takes around 20k steps. The batch size of U-Net is 192 and the batch size of particles is set as 16. The results are shown in Figure [5] It can be seen from the figure that our VSD generates plausible results even under a large number particles, which demonstrates the scalability of VSD. Although, due to the high optimization cost of 3D representation, the number of particles 3D experiments is relatively small, we demonstrate that VSD has the potential to scale up to more particles.
Figure 15: Selected samples from 2048 particles in a 2D experiment of VSD. Our VSD generates plausible results even under large number of particles, which demonstrates the scalability of VSD. The prompt is an astronaut is riding horse.
To match the 3D experiment setting, we also provide 2D experiments with LoRA and add annealed time schedule. The results are shown in Figure 16] The number of particles is set as and CFG= 7.5. Our VSD provides high-fidelity and diverse results in this setting. We also set the number of particles as 2048 and the results are shown in Figure 17
Figure 16: 2D experiments of VSD (CFG= 7.5) results with LoRA and annealed time schedule. The number of particles is 6. Our VSD provides high-fidelity and diverse results in this setting. The prompt an astronaut riding a horse.
We also provide the results of SDS for comparison in Fig. 18 Compared to our VSD, the results of SDS are smoother and lack details.
Figure 17:
Selected samples of 2D experiments of VSD (CFG= 7.5)
Figure 18: SDS result in 2D.
results with LoRA. The number of particles 2048.
Moreover, we visualize VSDISDS training phase of 2D in Fig. 19 Since the gradient is not directly readable, visualize + Av. which is the updated results if current sample optimizes via this gradient direction. As shown in Fig. 19 SDS tends to provide over-saturated and over-smooth gradient while VSD provides more notural gradients with more details. As a consequence. VSD provides better final results.
H How will CFG weights affect diversity?
In this section, we explore how CFG affects the diversity of generated results. For VSD, we set the number of particles as 4 and run experiments with different CFG. For SDS, we run times
26

---

40 Ax (0=)
(0=400)
Xg + Ax
Xy + (t=#80))
(t=8850)
0 ittess
300 iters
600 iters 1000 iters
0 iters
300 iters
600 iters 1000 iters
(1) Gradient visualization of SDS.
(2) Gradient visualization of VSD (ours).
Figure 19: Gradient visualization of VSD and SDS during the training phase in 2D experiment. SDS tends provide over-saturated and over-smooth gradient while VSD provides natural-looking gradients with more details.
of generation with different random seeds. The results are shown Figure 20 As shown in the figure, smaller CFG provides more diversity. We conjecture that this is because the distribution of smaller guidance weights has more diverse modes. However, when the CFG becomes too small (c.g.. CFG= 1), cannot provide enough guidance strength to generate plausible results. Therefore, for all our 3D experiments shown in the results, we set CFG = 7.5 as trade-off between diversity and optimization stability. Note that SDS could not work well in such small CFG weights. Instead, our VSD provides trade-off option between CFG weight and diversity, and it can generate more diverse results by simply setting smaller CFG.
I Limitations and Discussions
Although PrelificDreamer achieves remarkable text-to-3D results, the generation takes hours (espe- cially for resouutoooution NeRF training), which much slower than vanilla image generation by a diffusion model. Speeding up the text-to-3D generation is another critical problem, and we leave it future work.
Besides, our proposed scene initialization demonstrates its effectiveness in generating expansive scenes, yet there are still limitations, particularly with respect to camera positioning. Our current model sets the camera with a fixed view toward the scene's ' center, which serves object-centric scenes effectively but may suboptimal for scenes with intricate geometry and detailed textures. Furthermore, despite our efforts to produce outcomes with rich structure, occasional failures occur and the geometry reverts to a simplistic textured sphere. order address these limitations, future research could focus developing improved camera poses capable of capturing and rendering scenes in finer detail. Furthermore, our present model relies solely on the text-to-image diffusion model without the assistance of other models. The integration of off-the-shelf depth estimation models [7, as utilized in other studies [6, 58 could potentially enhance the accuracy and detail of the scene generation process.
In addition, the correspondence between text prompts and generated results sometimes insufficient, especially for complex prompts. We conjecture that it is because the ability to generate from complex prompts limited by the text encoder of Stable Diffusion. Janus problem [57] also exists in our case. Nevertheless, we believe our proposed VSD and other contributions are orthogonal to these problems, and the generation quality can be further improved by introducing more techniques, such using more powerful text-to-image diffusion model that understands view-dependent prompts better 34 or diffusion model with more 3D priors 24
addition, the multi-face
User Study
For completeness, we follow previous works 20 and conduct user study by comparing
ProlificDreamer with DreamFusion (34), Magie3D 20 and Fantasia3D 4 under 15 prompts,
prompts for each baseline. Since none of the basclines have released their codes, we can only use
27

---

VSD CRG =1
VSD CRG =2
VSD CRG =4
VSD
CRG =7.5
VSD CRG =25
VSD CFG 100
Patide
Paricle
Particle
Particle
VSD provides diversity with lower CFG between particles.
SDS CRG =7.5
SDS CFG =100
Seed 1
Seed
Seed 3
Seed
(b) SDS works with only large CFG weights.
VSD CRG =2
VSD CFG
VSD CRG =7.5
Seed 1
Seed 2
Seed 3
Seed 4
(c) VSD with single particle and different seeds provide slightly less diversity.
Figure 20: Ablation on how CFG weight affects the randomness. Smaller CFG provides more diversity. But too small CFG provides less optimization stability. The prompt is A high quality photo of an ice cream sundae.
28

---

the figures from their papers, which limits the number of results from baselines for us to compare. So we currently only compare under 15 prompts. The volunteers are shown the generated results of our ProlificDreamee and baselines and asked to choose the better one in terms of fidelity, details and vividness. We collect results from 109 volunteers, yielding 1635 pairwise comparisons. The results are shown in Table 3
Our method outperforms all of the baselines.
Table 3: Results of user study. The percentage of user preference (1) is reported in the table.
Name
DreamFusion
Magic3D
Fantasia3D
Prefer baseline
6.87
5,50
9.73
Prefer PreaificDreamer (Ours)
94.13
94.50
90.27
K
Quantitative Results
Table 4: 3D sample quality by SDS VSD, 100 prompts.
Method
SDS
VSD (n=1)
3D-FID (L)
118.92
107.02
Table 5: 3D sample quality by SDS or VSD, 25 prompts.
Method
SDS
VSD (n=1)
VSD (n=4)
3D-FID (L) 191.82
186.87
185.88
In this section, we add some quantitative results to demonstrate the effectiveness of VSD.
Experiment Setting For 3D experiments, we compute the FID score between rendered images by SDS/VSD and 2D sampled images by ancestral sampling, named as 3D-FID. Specifically, we select 100 prompts from previous works including DrcamFusion, Magic3D and Fantasia3D. For each prompt, we use VSD (number of particles n=1, CFG=7.5) or SDS (CFG=100) to optimize one 3D object and render 10 images uniformly from the circumference at angle of 30 above the horizon, and collect Ik images in total. To isolatedly compare VSD with SDS, we run with the default setting of the stage-1 1 NeRF training of ProlificDreamer (i.e., both VSD and SDS are in 512 resolution and use annealed t).
We compute the FID score between the 1k samples from the 100 prompts and 50k reference batch, which is sampled by 50-step DPM-Solver++ 28 with 500 images per prompt. In [5 due to the time and computation resource limits, we compare the results for VSD with n=4 under only randomly-selected prompts from the aforementioned 100 prompts, and compare SDS, VSD (n=1) and VSD (n=4) with the 50-step DPM-Solver++ under these 25 prompts. For VSD (n=4), as we can get particles (3D objects) per prompt, we randomly select one particle per prompt and render the corresponding 10 images of the selected particle for fair comparison. For 2D experiments in Table we follow the common setting of evaluating text-to-image models by computing FID on MSCOC02014 validation set. Specifically, per prompt by either 50-step DPM-Solver++, SDS (CFG=100), VSD (n=4, CFG=7.5) or VSD (n=8, CFG=7.5) to collect 1 samples for each method, and then compute the FID between the obtained samples and the entire COCO validation set. For VSD, as we can get n images per prompt, we randomly select one image per prompt for fair comparison.
In Table 4
randomly select 1k prompts and sample one image
Results VSD with n=1 still outperforms SDS in 3D (both with 512 resolution and annealed f), as shown in Table and Table 5 which demonstrates the effectiveness of VSD.
Using more particles is slightly better. Due to the limitation of time and computation resources, we only compare n=1 and n=4 in 3D experiments, and n=4 with n=8 in the 2D experiments. As shown in T5 VSD with partieles slightly outperforms VSD with 1 particles the 3D setting; and as shown in Table 6 VSD with particles slightly outperforms VSD with 4 particles in the 2D setting.
29

---

Table 6: 2D sample quality by different samplers, 1000 prompts.
Method
SDS
VSD (n=4)
VSD (n=8)
DPM++
FID ())
90.09
68.02
66.68
47.91
VSD outperforms SDS 2D. As shown in Table the FID by VSD is much better than SDS. As the 2D setting isolates the sampling algorithm from the 3D representations, we can directly compare different sampling algorithms, finding that VSD can get better sample quality than SDS (though still worse than SOTA diffusion samplers, it can generalize 3D cases).
L Why using SDS in stage-2 for the geometry optimization of mesh?
VSD can also be used to generate geometry. To validate this, we provide an ablation example in Fig. (3)) As shown in the figure. VSD can obtain reasonable geometry. Although the some part of the geometry from VSD is with more details than SDS (including the tail of the horse), on the whole, the result from VSD similar with SDS. We conjecture that this is because currently the triangle size of the mesh is relatively large and can't represent very fine details. Thus, for efficiency, we use SDS instead of VSD for mesh geometry optimization. We believe that VSD can be used to obtain high quality mesh if more advanced mesh representation is available.
Moreover, despite that and stage-3, in which VSD significantly improves the generated quality.
SDS optimize the geometry in stage-2, VSD is still crucial in stage- 1
(1) Generate
(2) Get coarse NeRF with mesh with
(3a) Mesh optimization with SDS.
(3b) Mesh optimization with VSD.
(4) Inherit texture from
(5) Finetune texture with
LoRA sample during stage (5).
VSD.
Marching Tets.
NeRF
VSD.
Figure 21: Pipeline along with ablation study of VSD for geometry optimization. In (3a) and (3b), we show that VSD can also be used for mesh optimization of geometry. We also show intermediate results of our pipeline in this figure along with the samples from LoRA during the training phase in (5).
M More Comparisons with Baselines
Here, provide more comparisons with baselines in Fig. 22 Fig. 23 Fig. 24 Fig. Fig. 26 and
Fig. 27 Since none of the basclines have released their codes, we can only directly copy the figures from the corresponding papers. Some baselines are missing given specific prompt because the prompt is not included in the corresponding papers. To demonstrate geometry, some baselines choose textureless shading 34 [2]] while the other 4 prefers the normal map. For Preaerr uniformly show the normal map for consistency. As shown in the figures, our method achieves better results in terms of fidelity and details.
30

---

Drsaonusion Michelangelo style statue of dog reading news on a cellphone.
Ours
HUbb aut!
DreamFusion A plate of fried chicken and waffles with maple syrup on them.
Ours
Ours A chimpanzee dressed like Henry VIII king of England.
DreamFusion
O urs
DreamFusion
A model of a house in Tudor style.
Figure 22: More results of ProlifieDreamer compared with baselines.
31

---

Ours
Fusion
A sliced loaf of fresh bread.
Ours
DreamFusion
A blue tulip.
Ours
DreamFusion
A cauldron full of gold coins.
Ours
DreamFusion
A lionfish.
Figure 23: More results of ProlifieDreamer compared with baselines.
32

---

Ours
DreamFusion
A praying mantis wearing roller.
Ours
DreamFusion
A rotary telephone carved out of wood.
Ours
DreamFusion
A typewriter.
Ours
DreamFusion
A plush dragon toy.
Figure 24: More results of ProlifiDDreamer compared with baselines.
33

---

Ours
Fantasia 3D
A Matte painting of castle made of cheesecake surrounded by moat made of ice cream.
Ours
Fantasia 3D
A pineapple.
Ours
Magic3D
DreamFusion
A small saguaro cactus planted in clay pot.
Figure 25: More results of ProlifiDDreamer compared with baselines.
34

---

Our
Fantasia 3D
DreamFusion
A delicious croissant.
Ours
Magic 3D
DreamFusion
A rabbit, animated movie character, high detail 3d model.
Figure 26: More results of ProlifiDDreamer compared with baselines.
35

---

Ours
Fantasia 3D
Magic 3D
DreamFusion
A car made out of sushi.
Ours
Fantasia 3D
Magic 3D
DreamFusion
A marble bust of a mouse.
Figure 27: More results of ProlifiDDreamer compared with baselines.
36


Reviews:
[{"review_id": "WuxRQCoYS0", "reviewer": "Reviewer_W2L2", "summary": "The authors introduce the VSD algorithm that generalizes SDS to optimizing a distribution of shapes in terms of KL divergence to the image diffusion model.\nLacking analytic score of the implicit rendered image distribution, they train a surrogate by using LORA finetuning to the base image diffusion model.\nThe resulting algorithm is demonstrated for optimizing an empirical distribution of particles and interpreted in framework of particle VI.\nThey empirically find that a lower CFG is possible when using VSD as opposed to SDS which alleviates artefacts (even in the n=1 case).\n\nAdditionally, a simple NERF initialization for scenes, annealed time schedule and higher resolution rendering are proposed as orthogonal improvements.", "questions": "- On line 247 it is mentioned that for mesh finetuning, VSD was not helpful for geometry. To clarify, the pipeline for the textured mesh then is: 1. VSD for Nerf init, 2. SDS on DMTet geom finetuning, 3. VSD on Texture finetuning? How would this compare to following Fantasia3D and skipping the 1st NERF stage? Can you show the intermediate results during each stage? Does the SDS stage affect the geometry much in stage 2?\n- Is the ice cream ablation in Figure 12 using the textured mesh pipeline?\n- In the appendix, there was some intuition about VSD proving more fine/sharp update direction. What do samples from the LoRA tuned model and the corresponding update direction look like?\n- The paper claims that SDS is one of the main bottlenecks to scaling resolutions to 512 (line 226). Could you elaborate on this, since it seems they are orthogonal?\n- For the 2D experiment, you tried the small U-Net with 2048 particles. Did you also try n=2048 for the LoRA? Are there issues with scaling LoRA?\n", "limitations": "The limitations as expanded upon in the appendix is sufficient. Maybe the authors can expand more about the robustness of the method and sensitivity to hyperparameters which is a common issue in related works.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novel method: Using an auxiliary fine-tuned image diffusion model to approximate the score of an implicit distribution is a very neat idea and could have potential wider applicability in other areas of generative modelling.\n\nPresentation: The motivation behind the method, technical background and mathematical framework are presented well and the writing is clear in these aspects. Details regarding the method are also fairly clear.\n\nStrong qualitative result: The ability to use lower CFG is an interesting finding and the authors give evidence for its effectiveness in producing high quality results. Combined with the other practical additions to the training pipeline, this results in very impressive SOTA visual quality.\n", "weaknesses": "Clearer experiments: There are a lot of different experimental settings which dilutes the evaluation of the core VSD contribution imo.\nThe paper seems to have 4 main experimental settings:\na. Single particle textured mesh objects\nb. Single particle NERFs (object and scene)\nc. Multi-particle NERF\nd. 2D VSD results  \n\nIn terms of multiple particles, these are only demonstrated for 1 2D prompt and 2 NERF prompts (and maybe 1 mesh in the appendix). Can multiple particle results be demonstrated for the prompts used in a/b.\n\nGiven the lack of quantitative metrics, the contributions of VSD, annealing t and higher resolution are only evaluated qualitatively on 1 or 2 prompts. Ablating VSD for more prompts would help strengthen the case for it. Currently it is only done for one of the prompts in setting a.\n\nLack of quantitative metrics: I understand there aren’t great established metrics in this area but currently the comparison is only done by showing a few qualitative examples and a user study with 5 prompts picked per baseline.\nEspecially since one of the claims in the paper is that VSD allows diverse generation, quantifying this over a representative set of prompts would greatly strengthen the evaluation section.\nR-precision has been reported in previous papers and given the motivation of matching in KL, a conditional FID to the image diffusion model could also be considered. \nIf computation is a bottleneck, then these could have also been obtained for a coarser version/2D case.\n\nClarity of explaining VSD results: Although VSD was motivated from the stand point of enabling diverse generations, the empirical benefits it brings in the single particle case is surprising and remains not well explained.\nHence, some of the discussions in section 3.3 (about “superior generalization” and regarding CFG on lines 185,186) seem too vague.\nI would recommend the authors could change the wording to make it more clear which results are empirical findings that may not have theoretical justification yet and further to write any intuition/speculation about the reason more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review and suggestions. Below we provide a point-to-point response to all comments. We hope you will find our response satisfactory and raise your score accordingly.\n\n\n***Q1: Can multiple particle results be demonstrated for the prompts used in a/b？***\n\n**A**: Yes. We provide more multi-particle results in the ***response pdf, Fig.5***, and will add more multi-particle results in the final version.\n\n***Q2: Comparing VSD and SDS for more prompts.***\n\n**A**: We provide more qualitative results comparing VSD with SDS in the ***response pdf, Fig.1***. We also add quantitative results in ***common response Q1***, including both 2D experiments (evaluated on 1000 prompts) and 3D experiments (evaluated on 100 prompts). In all experiments, VSD significantly outperforms SDS under the same setting, showing the effectiveness of VSD itself.\n\n***Q3: Lack of quantitative metrics.***\n\n**A**: Quantatively evaluating the results of text-to-3D is challenging. There is no well-estabilished method for this. Since FID score is the most common evaluation metric in 2D generative models, which evaluates both the quality and diversity of the samples, we add quantitative results of FID scores (see details in ***common response Q1***), including both 2D (1000 prompts) and 3D (100 prompts) experiments. In particular, FID scores involve the variance of samples, and thus the better FID scores of VSD suggest that VSD improves the diversity.\n\n\n***Q4: Improve the writing clarity of explaining VSD results.***\n\n**A**: We will clarify the discussions in Sec.3.3 about the single-particle VSD cases, following the suggestions. In particular, we will clarify the discussions about the empirical findings and theoretical justifications separately, and highlight the intuition of the reason for the benifits by VSD with single particle.\n\n***Q5: Pipeline and mesh finetuning intermediate results? Does SDS affect much in stage2?***\n\n**A**: Yes, the whole pipeline is: 1. VSD for NeRF, 2. SDS on geom finetuning, 3. VSD on texture finetuning. We show the intermediate results during each stage in the ***response pdf, Fig.3***. We find that converting NeRF to mesh causes lots of detail lost, and using both SDS and VSD can improve the geom results (we provide a detailed explanation for why using SDS in geom stage in ***common response Q2***) and using VSD can further improve the mesh texture.\n\n***Q6: Compare to Fantasia3D if skipping the 1st NeRF stage?***\n\n**A**: The 1st NeRF stage is crucial for the final results because it does not need any handcrafted initialization for each prompt (as in Fantasia3D), and using VSD in 1st stage can greatly improve both the geometry and texture quality used for initializing stage2 and stage3, since the NeRF quality by VSD is superior to SDS. So we adopt the 1st stage for better performance.\n\n\nGiven the NeRF initialization, as shown in ***response pdf, Fig.1***, VSD outperforms SDS significantly in texture optimization. Based on such results, even if skipping the 1st NeRF stage and using a handcrafted initialization as in Fantasia3D, ProlificDreamer still employs VSD to optimize the mesh texture in stage3, which is supposed to obtain a much better texture than SDS (as adopted in Fantasia3D).\n\n***Q7: Is the ice cream ablation in Figure 12 using the textured mesh pipeline?***\n\n**A**: No. It's the NeRF result from stage-1.\n\n***Q8: Visualize samples from the LoRA and the corresponding VSD update direction?***\n\n**A**: We provide a sample from the LoRA in ***response pdf, Fig.3***, showing that LoRA samples are consistent to 3D object when optimization converges. Moreover, we visualize VSD/SDS training phase of 2D in ***response pdf, Fig.2***. Since the gradient is not directly readable, we visualize $x+\\Delta x$, which is the updated results if current sample optimizes via this gradient direction. As shown in Fig.2, SDS tends to provide over-saturated and over-smooth gradient while VSD provides more natural-looking gradients with more details. As a consequence, VSD provides better final results.\n\n***Q9: Elaborate why SDS is one of the main bottlenecks to scaling resolutions to 512?***\n\n**A**: We apologize for the potential misunderstanding for the unclear statement. We clarify that SDS is one of the main bottlenecks to generate high-fidelity NeRF, which is orthogonal to the resolution, and the quality by SDS is still poor if simply scaling resolutions to 512. Instead, VSD can provide more details. We provide more examples in ***response pdf, Fig.1.(1)*** to show that 512-res SDS is still much worse than 512-res VSD. We will make it clearer in final version.\n\n***Q10: Did you also try n=2048 for the LoRA in 2D? Are there issues with scaling LoRA?***\n\n**A**: Following the suggestion, we conduct a new experiment to scale up LoRA in 2D experiments, and find that LoRA obtains better visual quality than U-Net with a large number of particles (e.g. n=2048), so it does not have scaling issues. We will add these results in the final version.\n\nOnce again, thank you for your constructive feedback and for considering our paper for acceptance. We'll revise our paper according to your suggestions. We kindly request that you consider raising the score accordingly if we addressed your concerns."}, {"review_id": "Bx8wIhj9sU", "reviewer": "Reviewer_wz2D", "summary": "This paper proposes Variational Score Distillation (VSD), which is a new generalization of Score Distillation Sampling (SDS) that is based on ", "questions": "1. In Figure 3, the VSD results generate slightly more 'oversaturated' results in comparison to ancestral sampling. Similarly, the mesh results in the paper generally seem to still suffer from over saturation issues (although some of this is likely just due to not modeling accurate physically based rendering). Is there any explanation for the perceived differences between VSD and ancestral sampling and why they lead to these qualitative differences?\n\n2. For the SDS experiments (in Figure 17 in the appendix, for example), what are the experimental settings? Do they use the same batch size = 1 with all of the same hyper parameters as the VSD? What hyper parameter differences exist between this and DreamFusion, Magic3D?\n\n3. How does the SDS NeRF results from this paper (without VSD) compare to DreamFusion and Magic3D in terms of perceived quality? What are the factors that cause this quality difference?", "limitations": "The limitations are addressed, although I have a lot of questions about the actual efficacy of VSD with respect to quality improvements. It would also be nice to have a broader statement on the societal impacts (even if it's somewhere in the appendix, unless I just didn't see it) since text-to-3D models are something that can bring potentially harmful impacts to the labor market around content creation. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The biggest strength of this paper is likely in its practical applicability; notably, I find it impressive that the paper is able to produce good results with a batch size = 1. Many prior works need larger batch sizes, and especially mesh generation (as in Magic3D) suffer if they do not have large batch sizes which significantly hurts performance. This is significant, as it makes text-to-3D generation much more feasible without access to a large GPU cluster / DGX sort of machine. However, it's not unclear if this is really due to VSD (more on this in the weakness section).\n\nThere are also other practical benefits to this algorithm, like the additional diversity in the output that they can offer. This is also significant, as one of the practical needs in deploying a text-to-X system is being able to 'reroll' for better outputs that better align with the user's intent through the text prompt. This is something that VSD uniquely offers.\n\nThe paper is also very well written, and the writing is very clear despite the complexity of what is being described. I do have a couple of more detail-oriented questions that I will ask in the questions section. ", "weaknesses": "I am slightly unconvinced about the effectiveness of VSD when it comes to increased quality. \n\nFirst, as a baseline, the generated results qualitatively do look much better than DreamFusion or Magic3D. The user studies agree, with an overwhelming margin (90+%). \n\nHowever, it is also the case that the generated results in this paper _without_ VSD and with just the higher resolution prior (and annealed t) already looks arguably significantly better than DreamFusion or Magic3D. Without a user study to then compare against this result with and without VSD, it's hard to make the claim that VSD itself is the factor that contributes most significantly to the improved quality. It's not entirely made clear in the paper what exactly is the factor that leads to this significantly quality increase. Empirically, high resolution NeRF training for SDS with Stable Diffusion has been difficult to make work (just even based on open source re-implementations of various papers), _especially_ with a small batch size of N=1 that they claim in the paper... so I am very curious what leads to the quality increase. But also it makes me question whether VSD really adds much in terms of quality without more examples or a user study. (Figure 5 does show an example where there seems to be perceived difference, but in my own experience, this is a minor difference that could just be caused by a slightly different hyper parameter)\n\nFigure 17, although it's only on a single example, somewhat confirms this since the high CFG weight seems to result in very similar results with the VSD results with low CFG. Although high CFG weight is a factor for over saturation, some of the results with VSD also seems to suffer from over saturation anyways. \n\nRegardless, VSD does offer other benefits like diversity. But it would be amazing to hear from the authors on their thoughts on this quality difference in the SDS baseline. Ideally a user study can compare between their baseline NeRF generation results with SDS against VSD, but I understand if that's difficult. \n\nAt the very least, there should be more examples (beyond the single example they show in Figure 5) that compare between their baseline NeRF results with SDS vs VSD. This would make this paper an extremely solid contribution. (it's also possible I just missed something).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review and suggestions. Below we provide a point-to-point response to all comments. If our response has addressed the concerns and brings new insights to the reviewer, we will highly appreciate it if the reviewer considers raising the score.\n\n***Q1: What exactly is the factor that leads to this significantly quality increase? Directly compare VSD with SDS? Oversaturated in Figure 17?***\n\n**A**: Thanks for the constructive comments. We believe that VSD itself is the main factor to improve quality and add both quantiative and qualitative results to confirm it.\n\nQuantitatively, we fairly compare VSD and SDS in terms of the widely adopted FID score in both 2D and 3D experiments (see details in ***common response Q1***). In particular, we compare VSD with SDS in the same setting of 512 resolution and annealed $t$ in the 3D experiments to eliminate the effects of other factors. Qualitatively, Figure 17 shows that VSD relieves the oversatuated problem of SDS. We provide extra results comparing VSD and SDS in ***response pdf, Fig.1***, including NeRF training stage-1 and mesh texturing stage-3, both with 512 resolution and annealed $t$.\n\nIn all experiments, VSD itself significantly outperforms SDS, showing its effectiveness. We will add all new results in the final version and we believe that the quality of the paper will be significantly improved.\n\n\n***Q2: VSD is slightly more oversaturated than ancestral sampling. Is there any explanation for the perceived differences between VSD and ancestral sampling and why they lead to these qualitative differences?***\n\n**A**: Theoretically, if the variational distributions are sufficiently powerful and the optimization of the variatioinal inference problem is sufficient, then the distribution of samples by VSD will be the same as that by ancestral sampling. However, in practice, we adopt a LoRA model as the variational distribution and optimize it by Adam, which slightly viodate the above conditons and results in the qualitative differences. \n\nNevertheless, we argue that VSD is still prefarable in zero-shot 3D generation because it significantly outperforms SDS (see more details in response to your Q1) and ancestral sampling can be only used in 2D.\n\n\n***Q3: What are the experimental settings for the SDS experiments in Fig.17?***\n\n**A**: We use the same settings as VSD (i.e., same hyperparameters, 512 resolution, annealed $t$). In particular, for the 3D representations, we follow most of the experiment settings and hyperparameters in Magic3D, and the differences are: 1. We use 512 resolution and annealed $t$; 2. We use batch size = 1 (following the re-implementation of stable-dreamfusion). 3. We do not use the shading proposed in DreamFusion because we currently find that the generated texture is better without shading. We will make the detailed settings clearer in the final version, and leave adding shading to ProlificDreamer as future work.\n\n***Q4: How does the SDS NeRF results from this paper (without VSD) compare to DreamFusion and Magic3D in terms of perceived quality? What are the factors that cause this quality difference?***\n\n**A**: Since DreamFusion and Magic3D use a powerful text-to-image base models (Imagen and eDiff-I) than stable-diffusion and not open-sourced, we use the open-sourced re-implementation of them by stable-dreamfusion, and adopt the improvement in the above response of Q3, finding that it improves the performance of SDS in stable-dreamfusion (see Fig.5 in the main text). However, it is still hard to say which is better between our SDS implementation and the official Dreamfusion and Magic3D. Nevertheless, after integrating VSD and the whole pipeline in ProlificDreamer, our generation quality is much better (see user study in appendix K and examples in appendix L) than Dreamfusion and Magic3D.\n\n***Q5: Add a broader statement on the societal impacts?***\n\n**A**: We have included a broader statement in Sec.6. We will follow the suggestion and add a longer version in the appendix. \n\nOnce again, thank you for your constructive feedback and for considering our paper for acceptance. We'll revise our paper according to your suggestions. We kindly request that you consider raising the score accordingly if we addressed your concerns."}, {"review_id": "vG1i6fIpAo", "reviewer": "Reviewer_krEH", "summary": "In this paper, a novel method for generating 3D representations from text prompts is proposed. More specifically, a novel variational score distillation (VSD) based optimisation strategy is proposed that allows for optimising NeRF- and mesh-based 3D representations by utilising pre-trained 2D diffusion models. It improves over the previously in Dreamfusion proposed score distillation sampling (SDS) by optimising a distribution over the 3D parameters instead of a single point Dirac distribution. Further, the authors made additional engineering improvements to the text-to-3d pipeline. The proposed system does not rely on high CFG weights as required for SDS and leads to impressive results that compare favourably against the state-of-the-art. \n", "questions": "- Have the authors investigated whether there is a relationship between number of particles and the quality of the results?\n- Could the authors expand on the limited geometry variance in contrast to the texture variance, and why the use of the VSD is not improving over SDS for the mesh geometry prediction?\n- Could the authors expand on the importance of used camera pose priors, especially in the context of scene generation?\n- Are different text prompts used for different camera poses?", "limitations": "The authors discuss limitations both in the main paper as well as more extensively in the supplementary. I believe the discussion is thorough and good as is. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The authors clearly identify an important shortcoming of current text-to-3D approaches: suffering from over-saturation, over-smoothing, and low-diversity problems. They identify the SDS-based optimisation as the core problem, and propose a theoretically-sound alternative that can be trained with only a small overhead compared to SDS.\n- The shown results clearly look impressive and the obtained scene representations clearly improve over the state-of-the-art results.\n- The manuscript is (mostly) very well written, has a great structure, and a great \"reading flow\".\n- The performed experiments, including the 2D experiments, clearly highlight the authors' contributions and show the effectiveness of the individual components of the proposed methods.\n- The authors provide a clear and correct overview over their core contributions in Table 1 and demonstrate their familiarity with the fast-moving field of text-to-3D generation.\n", "weaknesses": "- Number of particles: I am sceptical if n=1 or n=4 (L. 150) is really enough to capture the distribution. It would be interesting to investigate the relationship of number of particles and the quality of the results; as it seems that the prediction network for the noisy rendered images is trained with all particles, it could be that the quality overall improves with more samples. \n\n- Geometry optimisation and VSD/SDS: Accordingly to L. 247 ff., I believe that the authors explain that the mesh geometry is fine-tuned with SDS, not VSD (see also question below). If understood correctly, this part should express more clearly that it is only referring to the mesh geometry, not the overall geometry optimisation. Further, this seems to be an interesting limitation of the current method. It would be interesting to see a respective ablation figure comparing SDS and VSD for this mesh-geometry prediction stage.\n\n- Geometry Diversity: Figure 1 c.) shows the diversity of samples, and it appears that the geometry diversity is limited compared to the texture diversity. It would be interesting to discuss potential reasons for this.\n\n- Experimental Results: No quantitive metrics are reported and the only user study is \"hidden\" in Table 3 of the supplementary material. While it is true that the task of text-to-3D is very challenging to be measured quantitatively, metrics reported by previous methods such as Dreamfusion could also be reported, and the user study can be shown in the main paper or at least be referenced. \n\n- Camera Pose Priors: It would be very interesting to see how the camera sampling strategies affect the results; in particular for the \"scene generation\" experiments. Have the authors started investigating this dimension of the problem?\n\n- Related work: A good and thorough related work discussion is contained only in the supplementary. I would encourage the authors to add a discussion, if accepted, to the main manuscript. \n\n- Noisy prediction network: The scores for the noisy real and the noisy rendered images are predicted by two separate models (L. 166), and the one for the rendered images is smaller / a low-rank approximation. Is this introducing some form of imbalance or is this not a problem for the optimisation?\n\n- Training of noisy renderings score prediction network: If understood correctly from the supplementary, the network for predicting the scores of the noisy rendered images is trained simultaneously with the NeRF model. It could be interesting to study how different optimisation schemes affect the results.\n\n- Input text prompt: I believe it is not stated whether the same text prompt or different text prompts are used for different camera poses.\n\nTypos:\n- L 178: speical case -> special case\n- L 283: variation score distillation -> variational score distillation\n- L. 284: 3D parameter -> 3D parameters\n- L 637 of supp mat: an addition diffusion model -> an additional diffusion model", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review and suggestions. Below we provide a point-to-point response to all comments. If our response has addressed the concerns and brings new insights to the reviewer, we will highly appreciate it if the reviewer considers raising the score.\n\n***Q1: Is n=1~4 is really enough to capture the distribution? The relationship of number of particles and the quality?***\n\n**A**: Yes, it is enough because we adopt LoRA for the variational distribution. LoRA has a strong image prior and is suitable for few-shot score function learning. We have dicussed and provided corresponding quantitative experiments in ***common response, Q3***. In general, increasing the number of particles slightly increases the generated quality.\n\n***Q2: Why the use of the VSD is not improving over SDS for the mesh geometry prediction?***\n\n**A**: Please refer to **common response, Q2** for the discussion on this topic. We have provided a respective ablation figure comparing SDS and VSD for this mesh-geometry prediction stage, please see **response pdf, Fig.5**.\n\n***Q3: Discuss potential reasons why the geometry diversity is limited compared to the texture diversity (Fig.1.c).***\n\n**A**: The samples by VSD also have diverse geometry for some other prompts, and we provide extra results which demonstrates more geometry diversity in the ***response pdf, Fig.5***. We conjecture that the demonstrated two prompts in Fig.1.c are limited in geometry diversity, due to the image prior in stable-diffusion. In addition, lower the cfg will bring more geometry diversity as shown in Figure 17 in our appendix.\n\n\n***Q4: Quantitative evaluation results?***\n\n**A**: We provide quantitative results in **common response, Q1**, including both 2D experiments (evaluated on 1000 prompts) and 3D experiments (evaluated on 100 prompts). It will be added into the main paper. We will also reference the user study in the main paper.\n\n***Q5: Add related work to the main manuscript.***\n\n**A**: Sure, we will add the related work into the main text.\n\n\n***Q6: Is the smaller / low-rank approximation model for the noisy rendered images introducing some form of imbalance?***\n\n**A**: No, the training procedure by using LoRA is stable and we did not notice any problems for the optimization. We believe that it is because LoRA has a strong image prior and is suitable for few-shot learning (estimating the score functions of noisy rendered images), so the estimated score (update direction) in VSD is meaningful (please see ***response pdf, Fig.2*** for visualization of the updating direction of VSD). Moreover, LoRA only trains a small amount of parameters, and the base architecture is the same as the pretrained stable-diffusion, so the capacity of these two models is balanced.\n\n***Q7: It could be interesting to study how different optimization schemes for the noisy renderings score prediction network affect the results.***\n\n**A**: During the early experiments, we have tried different training schemes and find that optimizing 3D particles and variational scores for one step alternatively performs the best.\n\n***Q8: Typos***\n\n**A**: Thanks for correcting our typos and helping to improve the writing quality. We will fix them in the final version correspondingly.\n\n***Q9: Could the authors expand on the importance of used camera pose priors, especially in the context of scene generation? Have the authors started investigating this dimension of the problem?***\n\n**A**: Enlarging the radius range of camera pose is beneficial for scene generation. It is critical for scene to not be with a degraded geometry (a textured sphere). The scene geometry will improve a lot if more advanced camera pose priors are proposed, and we leave it for future work.\n\n***Q10: Are different text prompts used for different camera poses?***\n\n**A**: Yes, we use view-dependent prompts for pretrained Stable Diffusion, following previous works such as Dreamfusion and Magic3D. However, we do not use view-dependent prompts for the variational model (i.e., the LoRA model) since the camera pose has been injected into the model explicitly.\n\nOnce again, thank you for your constructive feedback and for considering our paper for acceptance. We'll revise our paper according to your suggestions. We kindly request that you consider raising the score accordingly if we addressed your concerns."}, {"review_id": "j0s5dQ64xm", "reviewer": "Reviewer_Gck3", "summary": "This paper proposes an interesting and novel technique for the task of text-to-3D generation. It defines a distribution of the target 3D scene, which is implemented as particles. Given the textual description, the distribution is updated using the Wasserstein gradient flow. Moreover, the paper proposes to fine-tune a diffusion model based on LoRA to calculate the score function of noisy rendered images, which is used to compute the objective function. Experiments show that the approach significantly improve the generation quality.", "questions": "Please see the weakness section.", "limitations": "The authors have discussed the limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The proposed technique is novel and interesting. Modeling 3D scenes as a distribution is insightful, which could inspire the following research.\n2. It is a good idea to fine-tune LoRA-based diffusion models on noisy rendered images for calculating the objective function.\n3. The results of the proposed model are impressive and obviously outperform previous methods.", "weaknesses": "1. I am curious about how the number of particles affect the performance of the proposed model.\n2. Can this idea be used in amortized text-to-3D generative models? Does it consumes a lot of GPU memory?\n3. It would be better to perform ablation studies on the distillation time schedule and the density initialization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review and suggestions. Below we provide a point-to-point response to all comments. If our response has addressed the concerns and brings new insights to the reviewer, we will highly appreciate it if the reviewer considers raising the score.\n\n***Q1: I am curious about how the number of particles affect the performance of the proposed model.***\n\n**A**: Please refer to the **common response, Q3**.\n\n***Q2: Can this idea be used in amortized text-to-3D generative models? Does it consumes a lot of GPU memory?***\n\n**A**: Yes, it is quite interesting and promising to use our method for amortized text-to-3D generative model, and we leave it for future work. Currently, our method consumes 27GB GPU memory in NeRF training stage and 17GB in mesh finetuning stage, using stable-diffusion 2.1 and its corresponding LoRA model.\n \n***Q3: It would be better to perform ablation studies on the distillation time schedule and the density initialization.***\n\n**A**: We have provide ablation studies on the distillation time schedule in Figure 5 in our paper, which shows that the generated results with our proposed annealed-time-schedule method have more details. As for density initialization, we find that without scene initialization, scene-level generation did not work at all, so we did not post an image on this in our paper. We will add a corresponding discussion in the final version.\n\nOnce again, thank you for your constructive feedback and for considering our paper for acceptance. We'll revise our paper according to your suggestions. We kindly request that you consider raising the score accordingly if we addressed your concerns."}, {"review_id": "8Uj6MWfgJt", "reviewer": "Reviewer_eq7N", "summary": " The paper presents a nice distillation pipeline that can perform high-quality text-to-3D generation using 2D diffusion priors. By introducing the score on NeRF / mesh rendered images, the authors make score distillation work with a classifier-free guidance weight as low as 7.5. This approach largely improves the detail quality of the converged results and inspires the community with an exciting direction.", "questions": "Please refer to the weakness section.", "limitations": "Please refer to the weakness section.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors propose to estimate scores on rendered image with a LoRA framework fine-tuned for the target scene.\n- By introducing this additional score, the distillation framework is able to work with low classifier-free guidance weight at 7.5, yielding much better appearance than the original SDS loss in DreamFusion.\n- Both 2D and 3D experiments validate the expressiveness of the proposed framework.", "weaknesses": "- The paper starts with derivations to support the particle-based variational framework. However, using more particles doesn't seem to help with the generation results. Is using a single particle not enough? What's the benefit of multiple particles, and when do we need them?\n- In Eq.(8) for 3D experiments, the authors mention that there's a `c` for camera pose added for the conditioning of the diffusion model. How is this implemented? Is this camera pose necessary for the framework to work? It would be nice if there's an ablation study on removing the camera pose conditioning.\n- For 3D experiments, why do we need a LoRA model fine-tuned from a pre-trained Stable Diffusion? Does using a small U-Net work for 3D case?\n- For the mesh refinement stage, the authors follow Fantasia3D to refine the geometry and texture separately. SDS loss is used in the geometry stage that refines normal maps. I'm interested in whether the proposed vsd loss is helpful in this case. Adding a comparison experiment that uses VSD to refine the normal map will be great.\n- Existing methods mostly use view-dependent prompts, meaning that text embedding `y` and camera `c` are not independent. Do authors use view-dependent prompts in the experiments? \n- Why are the two rows in Eq. (5) equal? More derivations are appreciated.\n- Is v-prediction necessary for the LoRA model? Does the proposed framework work with eps prediction?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review and suggestions. Below we provide a point-to-point response to all comments. If our response has addressed the concerns and brings new insights to the reviewer, we will highly appreciate it if the reviewer considers raising the score.\n\n***Q1: Is using a single particle not enough? What's the benefit of multiple particles, and when do we need them?***\n\n**A**: We find that using a single particle is **enough** for 3D generation, since the LoRA model has a strong image prior and performs quite well in the few-shot learning scenario. Specifically, our demonstrated results in Fig.1(a-b) and Appendix A are all obtained by optmizing a single particle with our proposed pipeline. As for the benefits of multiple particles, please refer to the ***common response, Q3***. In general, increasing the number of particles brings more diversity and slightly increases the generated quality, and we need them when we want more diverse and higher-quality results. We will add a corresponding discussion in the final version to improve the writing.\n\n\n***Q2: How is the camera condition implemented in the variational model? Is this camera pose necessary for the framework to work? It would be nice if there's an ablation study on removing the camera pose conditioning.***\n\n**A**: As presented in our appendix G, the camera pose is fed into a 2-layer MLP and then added into the time step embeddings of the base model. This camera condition is beneficial for the algorithm to converge since it makes LoRA training easier. We provide an ablation study on this in the ***response pdf, Fig.4***, showing that the results without camera conditions are inferior to the results with camera conditions, and the results without camera condition tend to have floaters and degraded geometry.\n\n\n***Q3: For 3D experiments, why do we need a LoRA model fine-tuned from a pre-trained Stable Diffusion? Does using a small U-Net work for 3D case?***\n\n**A**: Due to the computation cost, the number of particles is limited to a small number (1~4) in 3D experiments. Thus, estimating the score functions of noisy rendered images is a **few-shot learning problem** of diffusion models. As LoRA has a strong image prior and is suitable for this few-shot learning setting, we find that using LoRA can greatly improve the sample quality by VSD. In addition, in our early experiments, we have tried using a small U-Net instead of LoRA, and we find that the samples have more artifacts than those of LoRA.\n\n\n***Q4: Is the proposed vsd loss is helpful in stage2 (mesh geometry finetuning)? Adding a comparison experiment that uses VSD to refine the normal map will be great.***\n\n**A**: Please refer to **common response, Q2**.\n\n\n***Q5: Existing methods mostly use view-dependent prompts, meaning that text embedding $y$ and camera $c$ are not independent. Do authors use view-dependent prompts in the experiments?***\n\n**A**: We indeed use view-dependent prompts for pretrained Stable Diffusion, following previous works such as Dreamfusion and Magic3D. However, we do not use view-dependent prompts for the variational model (i.e., the LoRA model) since the camera pose has been injected into the model explicitly. We will change the math notations correspondingly to align them with the settings of view-dependent prompts, as specified in the following response.\n\n\n***Q6: Why are the two rows in Eq. (5) equal? More derivations are appreciated.***\n\n**A**: Thank you for pointing it out. We appologize for abusing the notation $y$ in the submission. In fact, the $y$ appears in the pretrained distribusion $p_t(x_t|y)$ and model $\\epsilon_{\\text{pretrain}}(x_t,t,y)$ depends on the camera condition $c$. We will make the dependence explicit by using notation $y^c$. Then Eq.(5) naturally holds (actually the first line is redundant). We will fix it in the final version and note that the method and theorems are still sound and the implementation remains the same.\n\n***Q7: Is v-prediction necessary for the LoRA model? Does the proposed framework work with eps prediction?***\n\n**A**: In our early experiments, we find that eps-prediction for the LoRA model can also work, but the performance of eps-prediction is slightly inferior to v-prediction in terms of texture details. We also provide ablation study on this in ***response pdf, Fig.4*** to demonstrate this.\n\nOnce again, thank you for your constructive feedback and for considering our paper for acceptance. We'll revise our paper according to your suggestions. We kindly request that you consider raising the score accordingly if we addressed your concerns."}]

Author rebuttal:
We sincerely thank all reviewers's efforts and their appreciation of our novel contributions as well as very detailed and insightful suggestions to further improve our paper. We find there are common concerns to our paper, and we'd like to clarify here. We also add a pdf file to add more experiment results.

***Q1: Quantitative evaluation for VSD and SDS? (from W2L2,wz2D,krEH)***
- Table a: 3D Sample Quality by SDS or VSD, 100 prompts.

|Method(3D)|SDS|VSD(n=1)|
|-|-|-|
|3D-FID(↓)|118.92|107.02|

- Table b: 3D Sample Quality by SDS or VSD, 25 prompts.

|Method(3D)|SDS|VSD(n=1)|VSD(n=4)|
|-|-|-|-|
|3D-FID(↓)|191.82|186.87|185.88|

- Table c: 2D Sample Quality by Different Samplers, 1000 prompts.

|Method(2D)|SDS|VSD(n=4)|VSD(n=8)|DPM++|
|-|-|-|-|-|
|FID(↓)|90.09|68.02|66.68|47.91|



**A**: We provide quantitative results (FID score, following common 2D generative model evaluations) for comparing VSD and SDS in both 2D and 3D experiments, as shown in Table a, Table b and Table c. We find that **VSD outperforms SDS** in both 2D and 3D cases, which confirms our claim about the significance of VSD. We present the detailed settings and results as follows.

**Detailed settings**:
- For 3D experiments, we compute the FID score between rendered images by SDS/VSD and 2D sampled images by ancestral sampling, named as 3D-FID. Specifically, we select 100 prompts from previous works including DreamFusion, Magic3D and Fantasia3D. For each prompt, we use VSD (#particles n=1, CFG=7.5) or SDS (CFG=100) to optimize one 3D object and render 10 images uniformly from the circumference at an angle of 30° above the horizon, and collect 1k images in total. To isolatedly compare VSD with SDS, we run with the default setting of the stage-1 NeRF training of ProlificDreamer (i.e., both VSD and SDS are in 512 resolution and use annealed $t$). 
    - In Table a, We compute the FID score between the 1k samples from the 100 prompts and a 50k reference batch, which is sampled by 50-step DPM-Solver++ with 500 images per prompt.
    - In Table b, due to the time and computation resource limits, we compare the results for VSD with n=4 under only 25 randomly-selected prompts from the aforementioned 100 prompts, and compare SDS, VSD(n=1) and VSD(n=4) with the 50-step DPM-Solver++ under these 25 prompts. For VSD(n=4), as we can get 4 particles (3D objects) per prompt, we randomly select one particle per prompt and render the corresponding 10 images of the selected particle for fair comparison.

- For 2D experiments in Table c, we follow the common setting of evaluating text-to-image models by computing FID on MSCOCO2014 validation set. Specifically, we randomly select 1k prompts and sample one image per prompt by either 50-step DPM-Solver++, SDS(CFG=100), VSD(n=4,CFG=7.5) or VSD(n=8,CFG=7.5) to collect 1k samples for each method, and then compute the FID between the obtained samples and the entire COCO validation set. For VSD, as we can get n images per prompt, we randomly select one image per prompt for fair comparison.

**Results**:
- **VSD with n=1 still outperforms SDS in 3D (both with 512 resolution and annealed $t$)**, as shown in Table a and Table b. Such quantitative results agree with the quanlitative results shown in ***response pdf, Fig.1***, showing the effectiveness of VSD.
- **Using more particles is slightly better**. Due to the limitation of time and computation resoures, we only compare n=1 and n=4 in 3D experiments, and n=4 with n=8 in the 2D experiments. As shown in Table b, VSD with 4 particles slightly outperforms VSD with 1 particles in the 3D setting; and as shown in Table c, VSD with 8 particles slightly outperforms VSD with 4 particles in the 2D setting.
- **VSD outperforms SDS in 2D**. As shown in Table c, the FID by VSD is much better than SDS. As the 2D setting isolates the sampling algorithm from the 3D representations, we can directly compare different sampling algorithms, finding that VSD can get better sample quality than SDS (though still worse than SOTA diffusion samplers, but it can generalize to 3D cases).


***Q2: Why using SDS in stage-2 for the geometry optimization of mesh? (from W2L2,krEH,eq7N)***

**A**: VSD can also be used to generate geometry. To validate this, we provide an ablation example in the ***response pdf, Fig.(3a),(3b)***. As shown in the figure, VSD can obtain reasonable geometry. Although the some part of the geometry from VSD is with more details than SDS (including the tail of the horse), on the whole, the result from VSD is similar with SDS. We conjecture that this is because currently the triangle size of the mesh is relatively large and can't represent very fine details. Thus, for efficiency, we use SDS instead of VSD for mesh geometry optimization. We believe that VSD can be used to obtain high quality mesh if more advanced mesh represetation is available.

Moreover, despite that we use SDS to optmize the geometry in stage-2, VSD is still crucial in stage-1 and stage-3, in which VSD significantly improves the generated quality.

***Q3: How will the number of particles affect the results of VSD? (from krEH,Gck3,eq7N)***
**A**: In general, using more particles will make the variational distribution (LoRA model) easier to train, and thus improve the sample quality. We provide an empirical evidence in Table b and
c (see more details in the response of Q1). Moreover, as LoRA has a strong image prior and is suitable for few-shot learning, we find that using 1~4 particles is enough for high-quality 3D generation.


We think the the quality of our work has been improved a lot. We are welcome for further questions.

Meta-review: The paper proposed a novel theoretic perspective for text to 3D generation, and introduced variational score distillation which optimizes the KL divergence between image diffusion and the distribution of 3D shapes. The surrogate is implemented as LORA fine-tuning. Qualitative results demonstrated the effectiveness of VSD on the diversity of generated 3D assets, the over-saturation and the over-smoothing problem. The additional quantitative results from the rebuttal further strengthen the paper, and all the reviewers are positive about the paper. The proposed VSD poses a new avenue for a theoretical understanding of text-to-3D generation, can be applied to many other areas of 3D generative modelling, and can also inspire follow-ups on the related directions, and thus I recommend accepting the paper. 

As promised during the rebuttal and discussions, please revise the paper by adding the provided quantitative evaluations (already presented in the rebuttal), adding more clarifications on the experimental settings (whether or not using SDS on stage-2 training), more qualitative results of multi-particle experiments, and moving related work to the main paper, etc.
Decision: Accept (spotlight)