Title: ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation
Venue: NeurIPS 2023
Accepted: yes
Keywords: diffusion models

text to 3D
Abstract: Score distillation sampling (SDS) has shown great promise in text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-saturation, over-smoothing, and low-diversity problems. In this work, we propose to model the 3D parameter as a random variable instead of a constant as in SDS and present *variational score distillation* (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed *ProlificDreamer*, can generate high rendering resolution (i.e., 512$\times$512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic.

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation

Abstract

Score distillation sampling (SDS) has shown great promise in text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-saturation, over-smoothing, and low-diversity problems. In this work, we propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7 . 5 ). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer , can generate high rendering resolution (i.e., 512 × 512 ) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic. Project page: https://ml.cs.tsinghua.edu.cn/prolificdreamer/ .

1 Introduction

3D content and technologies enable us to visualize, comprehend, and interact with complex objects and environments that are reflective of our real-life experiences. Their pivotal role extends across a wide array of domains, encompassing architecture, animation, gaming, and the rapidly evolving fields of virtual and augmented reality. In spite of the extensive applications, the production of premium 3D content often remains a formidable task. It necessitates a significant investment of time and effort, even when undertaken by professional designers. This challenge has prompted the development of text-to-3D methods [19, 31, 34, 20, 4, 29, 55, 16]. By automating the generation of 3D content based on textual descriptions, these innovative methods present a promising way towards streamlining the 3D content creation process. Furthermore, they stand to make this process more accessible, potentially encouraging a significant paradigm shift in the aforementioned fields.

A chimpanzee dressed like

A pineapple.

A model of a house in

dog reading news on a cellphone.

An elephant skull.

Tudor style.

A snail on a leaf.

An astronaut is riding a horse.

A tarantula, highly detailed.

A DSLR photo of a

A Matte painting of a castle

made of cheesecake surrounded

by a moat made of ice cream.

A red fire hydrant

spraying water.

A DSLR photo of a

table with dim sum on it.

A DSLR photo of a hamburger inside a restaurant.

(b) ProlificDreamer can generate high rendering resolution (i.e., 512 × 512 ) and high-fidelity NeRF with rich structures and complex effects. Besides, the bottom results show that ProlificDreamer can generate complex scenes with 360 ◦ views because of our scene initialization (see Sec. 4.2).

A highly detailed sand castle.

A hotdog in a tutu skirt.

(c) ProlificDreamer can generate diverse and semantically correct 3D scenes given the same text.

Diffusion models [46, 14, 49] have significantly advanced text-to-image synthesis [36, 41, 38, 1], particularly when trained on large-scale datasets [43]. Inspired by these developments, DreamFusion [34] employs a pretrained, large-scale text-to-image diffusion model for the generation of 3D content from text in the wild, circumventing the need for any 3D data. DreamFusion introduces the Score Distillation Sampling (SDS) algorithm to optimize a single 3D representation such that the image rendered from any view maintains a high likelihood as evaluated by the diffusion model, given the text. Despite its wide application [20, 4, 29, 55], empirical observations [34] indicate that SDS often suffers from over-saturation, over-smoothing, and low-diversity problems, which have yet to be thoroughly explained or adequately addressed. Additionally, orthogonal elements in the design space for text-to-3D, such as rendering resolution and distillation time schedule, have not been fully explored, suggesting a significant potential for further improvement. In this paper, we present a systematic study of all these elements to obtain elaborate 3D representations.

We first present Variational Score Distillation (VSD), which treats the corresponding 3D scene given a textual prompt as a random variable instead of a single point as in SDS [34]. VSD optimizes a distribution of 3D scenes such that the distribution induced on images rendered from all views aligns as closely as possible, in terms of KL divergence, with the one defined by the pretrained 2D diffusion model (see Sec. 3.1). Under this variational formulation, VSD naturally characterizes the phenomenon that multiple 3D scenes can potentially align with one prompt. To solve it efficiently, VSD adopts particle-based variational inference [23, 3, 9], and maintains a set of 3D parameters as particles to represent the 3D distribution. We derive a novel gradient-based update rule for the particles via the Wasserstein gradient flow (see Sec. 3.2) and guarantee that the particles will be samples from the desired distribution when the optimization converges (see Theorem 2). Our update requires estimating the score function of the distribution on diffused rendered images, which can be efficiently and effectively implemented by a low-rank adaptation (LoRA) [18, 40] of the pretrained diffusion model. The final algorithm alternatively updates the particles and score function.

Weshow that SDS is a special case of VSD, by using a single-point Dirac distribution as the variational distribution (see Sec. 3.3). This insight explains the restricted diversity and fidelity of the generated 3D scenes by SDS. Moreover, even with a single particle, VSD can learn a parametric score model, potentially offering superior generalization over SDS. We also empirically compare SDS and VSD in 2D space by using an identity rendering function that isolates other 3D factors. Similar to ancestral sampling from diffusion models, VSD is able to produce realistic samples using a normal CFG weight (i.e., 7.5). In contrast, SDS exhibits inferior results, sharing the same issues previously observed in text-to-3D, such as over-saturation and over-smoothing [34].

We further systematically study other elements orthogonal to the algorithm for text-to-3D and present a clear design space in Sec. 4. Specifically, we propose a high rendering resolution of 512 × 512 during training and an annealed distilling time schedule to improve the visual quality. We also propose scene initialization , which is crucial for complex scene generation. Comprehensive ablations in Sec. 5 demonstrate the effectiveness of all the aforementioned elements particularly for VSD. Our overall approach can generate high-fidelity and diverse 3D results. We term it as ProlificDreamer 2 .

As shown in Fig. 1 and Sec. 5, ProlificDreamer can generate 512 × 512 rendering resolution and high-fidelity Neural Radiance Fields (NeRF) with rich structure and complex effects (e.g., smoke and drops). Besides, for the first time, ProlificDreamer can successfully construct complex scenes with multiple objects in 360 ◦ views given the textual prompt. Further, initialized from the generated NeRF, ProlificDreamer can generate meticulously detailed and photo-realistic 3D textured meshes.

2 Background

We present preliminaries on diffusion models, score distillation sampling, and 3D representations.

Diffusion models. A diffusion model [46, 14, 49] involves a forward process { q t } t ∈ [0 , 1] to gradually add noise to a data point x 0 ∼ q 0 ( x 0 ) and a reverse process { p t } t ∈ [0 , 1] to denoise/generate data. The forward process is defined by q t ( x t | x 0 ) := N ( α t x 0 , σ 2 t I ) and q t ( x t ) := ∫ q t ( x t | x 0 ) q 0 ( x 0 )d x 0 , where α t , σ t > 0 are hyperparameters satisfying α 0 ≈ 1 , σ 0 ≈ 0 , α 1 ≈ 0 , σ 1 ≈ 1 ; and the reverse process is defined by denoising from p 1 ( x 1 ) := N ( 0 , I ) with a parameterized noise prediction

network ϵ ϕ ( x t , t ) to predict the noise added to a clean data x 0 , which is trained by minimizing

where ω ( t ) is a time-dependent weighting function. After training, we have p t ≈ q t and thus we can draw samples from p 0 ≈ q 0 . Moreover, the noise prediction network can be used for approximating the score function of both q t and p t by ∇ x t log q t ( x t ) ≈ ∇ x t log p t ( x t ) ≈ -ϵ ϕ ( x t , t ) /σ t .

One of the most successful applications of diffusion models is text-to-image generation [41, 36, 38], where the noise prediction model ϵ ϕ ( x t , t, y ) is conditioned on a text prompt y . In practice, classifierfree guidance (CFG [15]) is a key technique for trading off the quality and diversity of the samples, which modifies the model by ˆ ϵ ϕ ( x t , t, y ) := (1 + s ) ϵ ϕ ( x t , t, y ) -s ϵ ϕ ( x t , t, ∅ ) , where ∅ is a special 'empty' text prompt representing for the unconditional case, and s > 0 is the guidance scale. A larger guidance scale usually improves the text-image alignment but reduces diversity.

Text-to-3D generation by score distillation sampling (SDS) [34]. SDS is an optimization method by distilling pretrained diffusion models, also known as Score Jacobian Chaining (SJC) [55]. It is widely used in text-to-3D generation [34, 55, 20, 29, 55, 4] with great promise. Given a pretrained text-to-image diffusion model p t ( x t | y ) with the noise prediction network ϵ pretrain ( x t , t, y ) , SDS optimizes a single 3D representation with parameter θ ∈ Θ , where Θ is the space of θ with the Euclidean metric. Given a camera parameter c with a distribution p ( c ) and a differentiable rendering mapping g ( · , c ) : Θ → R d , denote y c as the ' view-dependent prompt ' [34] (i.e., a text prompt with view information), q θ t ( x t | c ) as the distribution at time t of the forward diffusion process starting from the rendered image g ( θ, c ) with the camera c and 3D parameter θ . SDS optimizes the parameter θ by solving

where t ∼ U (0 . 02 , 0 . 98) , ϵ ∼ N ( 0 , I ) , and x t = α t g ( θ, c ) + σ t ϵ . Its gradient is approximated by

Notwithstanding this progress, empirical observations [34] show that SDS often suffers from oversaturation, over-smoothing, and low-diversity issues, which have yet to be thoroughly explained or adequately addressed.

3D representations. We employ NeRF [30, 32] (Neural Radiance Fields) and textured mesh [45] as two popular and important types of 3D representations. In particular, NeRF represents 3D objects using a multilayer perceptron (MLP) that takes coordinates in a 3D space as input and outputs the corresponding color and density. Here, θ corresponds to the parameters of the MLP. Given camera pose c , the rendering process g ( θ, c ) is defined as casting rays from pixels and computing the weighted sum of the color of the sampling points along each ray to composite the color of each pixel. NeRF is flexible for optimization and is capable of representing extremely complex scenes. Textured mesh [45] represents the geometry of a 3D object with triangle meshes and the texture with color on the mesh surface. Here the 3D parameter θ consists of the parameters to represent the coordinates of triangle meshes and parameters of the texture. The rendering process g ( θ, c ) given camera pose c is defined by casting rays from pixels and computing the intersections between rays and mesh surfaces to obtain the color of each pixel. The textured mesh allows high-resolution and fast rendering with differentiable rasterization.

3 Variational Score Distillation

We now present Variational Score Distillation (VSD) (see Sec. 3.1) that learns to sample from a distribution of the 3D scenes. By using 3D parameter particles to represent the target 3D distribution, we derive a principled gradient-based update rule for the particles via the Wasserstein gradient flow (see Sec. 3.2). We further show that SDS is a special case of VSD and constructs an experiment in 2D space to study the optimization algorithm isolated from the 3D representations, explaining the practical issues of SDS both theoretically and empirically (see Sec. 3.3).

3.1 Sampling from 3D Distribution as Variational Inference

In principle, given a valid text prompt y , there exists a probabilistic distribution of all possible 3D representations. Under a 3D representation (e.g., NeRF) parameterized by θ , such a distribution can

NeRF/Mesh

Sample

Particles of NeRF/Mesh

Render 2D

Image

Text to Image

Diffusion

LoRA

Back Propagation

Back Propagation

Add Noise

A highly detailed

sand castle.

+ Finetune

Add Noise

be modeled as a probabilistic density µ ( θ | y ) . Denote q µ 0 ( x 0 | c, y ) as the (implicit) distribution of the rendered image x 0 := g ( θ, c ) given the camera c with the rendering function g ( · , c ) , and denote p 0 ( x 0 | y c ) as the marginal distribution of t = 0 defined by the pretrained text-to-image diffusion model with the view-dependent prompt y c . To obtain 3D representations of high visual quality, we propose to optimize the distribution µ to align the rendered images of its samples with the pretrained diffusion model in all views by solving

This is a typical variational inference problem that uses the variational distribution q µ 0 ( x 0 | c, y ) to approximate (distill) the target distribution p 0 ( x 0 | y c ) .

Directly solving problem (4) is hard because p 0 is rather complex and the high-density regions of p 0 may be extremely sparse in high dimension [47]. Inspired by the success of diffusion models [14, 49], we construct a series of optimization problems with different diffused distributions indexed by t . As t increases to T , the optimization problem becomes easier because the diffused distributions get closer to the standard Gaussian. We simultaneously solve an ensemble of these problems (termed as variational score distillation or VSD) as follows:

where q µ t ( x t | c, y ) := ∫ q µ 0 ( x 0 | c, y ) p t 0 ( x t | x 0 )d x 0 and p t ( x t | y c ) := ∫ p 0 ( x 0 | y c ) p t 0 ( x t | x 0 )d x 0 are the corresponding noisy distributions at time t with the Gaussian transition p t 0 ( x t | x 0 ) = N ( x t | α t x 0 , σ 2 t I ) , and ω ( t ) is a time-dependent weighting function.

Compared with SDS that optimizes for the single point θ , VSD optimizes for the whole distribution µ , from which we sample θ . Notably, we prove that introducing the additional KL-divergence for t > 0 in VSD does not affect the global optimum of the original problem (4), as shown below.

Theorem 1 (Global optimum of VSD, proof in Appendix C.4.) . For each t > 0 , we have

3.2 Update Rule for Variational Score Distillation

To solve problem (5), a direct way can be to train another parameterized generative model for µ , but it may bring much computation cost and optimization complexity. Inspired by previous particle-based variational inference [23, 3, 9] methods, we maintain n 3D parameters 3 { θ } n i =1 as particles and derive a novel update rule for them. Intuitively, we use { θ } n i =1 to 'represent' the current distribution µ , and θ ( i ) will be samples from the optimal distribution µ ∗ if the optimization converges. Such optimization can be realized by simulating an ODE w.r.t. θ , as shown in the following theorem.

Theorem 2 (Wasserstein gradient flow of VSD, proof in Appendix C) . Starting from an initial distribution µ 0 , denote the Wasserstein gradient flow minimizing problem (5) in the distribution (function) space at each time τ ≥ 0 as { µ τ } τ ≥ 0 with µ ∞ = µ ∗ . Then we can sample θ τ from µ τ by

(c) Ancestral sampling [27] (CFG = 7.5)

(d) VSD (CFG = 7.5, ours )

firstly sampling θ 0 ∼ µ 0 ( θ 0 | y ) and then simulating the following ODE:

where q µ τ t is the corresponding noisy distribution at diffusion time t w.r.t. µ τ at ODE time τ .

According to Theorem 2, we can simulate the ODE in Eq. (7) for a large enough τ to approximately sample from the desired distribution µ ∗ . The ODE involves the score function of noisy real images and that of noisy rendered images at each time 4 τ . The score function of noisy real images -σ t ∇ x t log p t ( x t | y c ) can be approximated by the pretrained diffusion model ϵ pretrain ( x t , t, y c ) . The score function of noisy rendered images -σ t ∇ x t log q µ τ t ( x t | c, y ) is estimated by another noise prediction network ϵ ϕ ( x t , t, c, y ) , which is trained on the rendered images by { θ ( i ) } n i =1 with the standard diffusion objective (see Eq. (1)):

In practice, we parameterize ϵ ϕ by either a small U-Net [39] or a LoRA (Low-rank adaptation [18, 40]) of the pretrained model ϵ pretrain ( x t , t, y c ) , and add additional camera parameter c to the condition embeddings in the network. In most cases, we find that using LoRA can greatly improve the fidelity of the obtained samples (e.g., see results in Fig. 1). We believe that it is because LoRA is designed for efficient few-shot fine-tuning and can leverage the prior information in ϵ pretrain (the information of both images and text corresponding to y ).

Note that at each ODE time τ , we need to ensure ϵ ϕ matches the current distribution q µ τ t . Thus, we optimize ϵ ϕ and θ ( i ) alternately, and each particle θ ( i ) is updated by θ ( i ) ← θ ( i ) -η ∇ θ L VSD ( θ ( i ) ) , where η > 0 is the step size (learning rate). According to Theorem 2, the corresponding gradient is

where x t = α t g ( θ, c )+ σ t ϵ . Weshow the approach of VSD in Fig. 3 (see pseudo code in Appendix E).

3.3 Comparison with SDS

We now systematically compare VSD with SDS in both theory and practice.

SDS as a special case of VSD. Theoretically, comparing the update rules of SDS (Eq. (3)) and VSD (Eq. (9)), SDS is a special case of VSD by using a single-point Dirac distribution µ ( θ | y ) ≈ δ ( θ -θ (1) ) as the variational distribution (see Appendix C.3 for derivation). In particular, VSD not only employs potentially multiple particles but also learns a parametric score function ϵ ϕ even for a single particle (i.e., n = 1 ). Empirically, the learned neural network may potentially offer superior generalization ability over the Dirac distribution in SDS, thus it may provide more accurate updating directions in low-density regions. Moreover, by using LoRA, VSD can additionally exploit the text prompt y in the estimation ϵ ϕ ( x t , t, c, y ) , while the Gaussian noise ϵ used in SDS cannot leverage the information from y . Thus, VSD may provide samples which are more aligned with the prompt y .

VSD is friendly to CFG. As VSD aims to sample θ from the optimal µ ∗ defined by the pretrained model ϵ pretrain, the effects by tuning the CFG in ϵ pretrain for 3D sampling by VSD are quite similar to 2D sampling by the traditional ancestral sampling methods [14, 27]. Therefore, VSD can tune CFG as flexibly as the classic text-to-image methods, and we use the same setting of CFG (e.g. 7 . 5 ) as the common text-to-image generation task for the best performance. To the best of our knowledge, this for the first time addresses the problem in previous SDS [34, 20, 4, 29] that it usually requires a large CFG (i.e., 100 ).

VSD vs. SDS in 2D experiments that isolate 3D representations. To directly compare SDS and VSD, we consider a special case of the rendering function g ( θ ) to decouple the optimization algorithm from 3D representations. In particular, we set g ( θ, c ) ≡ θ for any c . Then the rendered image x = g ( θ, c ) = θ is the same 2D image as θ . In such a case, optimizing the parameter θ is equivalent to generating an image in 2D space, thereby independent of the 3D representation. We show the results of different sampling methods in Fig. 3. SDS exhibits failure under both small and large CFG weights. Particularly with the default CFG weight (i.e., 100) used in SDS, the 2D samples share the same issues previously observed in text-to-3D such as over-saturation and oversmoothing [34]. In contrast, VSD demonstrates flexibility in accommodating various CFG weights and produces realistic samples using a normal CFG weight (i.e., 7.5), behaving similarly to ancestral sampling from diffusion models. See more details and analysis in Appendix G.

As other 3D factors are isolated in this comparison, these theoretical and empirical results suggest that the aforementioned practical issues of SDS [34] stem from the oversimplified variational distribution and large CFG employed by SDS. Such results strongly motivate us to employ VSD for text-to-3D generation, where it still substantially and consistently outperforms SDS (see evidence in Sec 5).

4 ProlificDreamer

We further present a clear design space for text-to-3D in Sec. 4.1 and systematically study other elements orthogonal to the distillation algorithm in Sec. 4.2. Combining all improvements highlighted in Tab. 1, we arrive at ProlificDreamer , an advanced text-to-3D approach.

DreamFusion

Magic3D

Ours

Fantasia3D

A plate piled high with chocolate chip cookies.

4.1 Design Space of Text-to-3D Generation

We adopt the two-stage approach [20], with several improvements in the design space of text-to-3D generation as summarized in Table 1. Specifically, in the first stage, we optimize a high-resolution (e.g., 512 ) NeRF by VSD to utilize its high flexibility for generating scenes with complex geometry. In the second stage, we use DMTet [45] to extract textured mesh from the NeRF obtained in the first stage, and further fine-tune the textured mesh for high-resolution details. The second stage is optional because both NeRF and mesh have their own advantages in representing 3D content and are preferred in certain cases. Nevertheless, ProlificDreamer can generate both high-fidelity NeRFs and meshes.

4.2 3D Representation and Training

We systematically study other elements orthogonal to the algorithmic formulation. Specifically, we propose a high rendering resolution of 512 × 512 during training and an annealed distilling time schedule to improve the visual quality. We also carefully design a scene initialization, which is crucial for complex scene generation.

High-resolution rendering for NeRF training. We choose Instant NGP [32] for efficient highresolution rendering and optimize NeRF with up to 512 training resolution using VSD. By applying VSD, we obtain high-fidelity NeRFs with resolutions varying from 64 to 512 .

Scene initialization for NeRF training. We initialize the density for NeRF as σ init ( µ ) = λ σ (1 -|| µ || 2 r ) , where λ σ is the density strength, r is the density radius, and µ is the coordinate. For objectcentric scenes, we follow the object-centric initialization used in Magic3D [20] with λ σ = 10 and r = 0 . 5 ; For complex scenes, we propose scene initialization by setting λ σ = -10 to make the density 'hollow' and r = 2 . 5 that encloses the camera. We show in Appendix D that the scene initialization can help to generate high-fidelity complex scenes without other modifications to the existing algorithm. In addition, we can further add a centric object to the complex scene by using object-centric initialization for || µ || 2 < 5 / 6 and scene initialization for others, where the hyperparameter 5 / 6 ensures the initial density function is continuous.

Annealed time schedule for score distillation. We utilize a simple two-stage annealing of time step t in the score distillation objective, suitable for both SDS (Eq. (3)) and VSD (Eq. (9)). For the first several steps we sample time steps t ∼ U (0 . 02 , 0 . 98) and then anneal into t ∼ U (0 . 02 , 0 . 50) . The key insight is that, essentially, we aim to match the original q µ 0 ( x 0 | c, y ) with p 0 ( x 0 | y c ) . The KL-divergence for larger t can provide reasonable optimization direction during the early stage of training. During training, while x ∗ is approaching the support of p 0 ( x ∗ | y c ) , a smaller t can narrow the gap between p t ( x ∗ | y c ) and p 0 ( x ∗ | y c ) , and provide elaborate details aligning with p 0 ( x ∗ | y c ) .

Mesh representation and fine-tuning. We adopt a coordinate-based hash grid encoder inherited from NeRF stage to represent the mesh texture. We follow Fantasia3D [4] to disentangle the

(1) 64 rendering

(2) + 512 rendering

(3) + Annealed t

(4) + VSD

optimization of geometry and texture by first optimizing the geometry using the normal map and then optimizing the texture. In our initial experiments, we find that optimizing geometry with VSD provides no more details than using SDS. This may be because the mesh resolution is not large enough to represent high-frequency details. Thus, we optimize geometry with SDS for efficiency. But unlike Fantasia3D [4], our texture optimization is supervised by VSD with CFG = 7.5 with the annealed time schedule, which can provide more details than SDS.

5 Experiments

5.1 Results of ProlificDreamer

We show the generated results of ProlificDreamer in Fig. 1(a) and 1(b), including high-fidelity mesh and NeRF results. All the results are generated by VSD. For all experiments without mentioned, VSD uses n = 1 particle for a fair comparison with SDS. In Fig. 1(c), we also demonstrate VSD can generate diverse results, showing that different particles in a round are diverse (with n = 4 ).

Object-centric generation. We compare our method with three SOTA baselines, DreamFusion [34], Magic3D [20] and Fantasia3D [4]. All of the baselines are based on SDS. Since none of them is open-sourced, we use the figures from their papers. As shown in Fig. 4, ProlificDreamer generates 3D objects with higher fidelity and more details, which demonstrates the effectiveness of our method.

In appendix, we add user study (Section J) and quantitative results (Section K) of ProlificDreamer against the baselines to demonstrate the effectiveness of our method.

Large scene generation. As shown in Fig. 1(b), our method can generate 360 ◦ scenes with highfidelity and fine details. The depth map shows that the scenes have geometry instead of being a 360 ◦ textured sphere, verifying that with our scene initialization alone we can generate high-fidelity large scenes without much modification to existing components. See more results in Appendix B.

5.2 Ablation Study

Ablation on NeRF training. Fig. 5 provides the ablation on NeRF training. Starting from the common setting [34, 20] with 64 rendering resolution and SDS loss, we ablate our proposed improvements step by step, including increasing resolution, adding annealed time schedule, and adding VSD all improve the generated results. It demonstrates the effectiveness of our proposed components. We provide more ablation on large scene generation in Appendix D, with a similar conclusion.

Ablation on mesh fine-tuning. We ablate between SDS and VSD on mesh fine-tuning, as shown in Appendix D. Fine-tuning texture with VSD provides higher fidelity than SDS. As the fine-tuned results of textured mesh are highly dependent on the initial NeRF, getting a high-quality NeRF at the first stage is crucial. Note that the provided results of both VSD and SDS in mesh fine-tuning are based on and benefit from the high-fidelity NeRF results in the first stage by our VSD.

Ablation on CFG. We perform ablation to explore how CFG affects generation diversity. We find that smaller CFG encourages more diversity. Our VSD works well with small CFG and provides

considerable diversity, while SDS cannot generate plausible results with small CFG (e.g., 7 . 5 ), which limits its ability to generate diverse results. Results and more details are shown in Appendix H.

6 Related Works

Diffusion models. Score-based generative model [48, 49] and diffusion models [14] have shown great performance in image synthesis [14, 7]. Recently, large-scale diffusion models have shown great performance in text-to-image synthesis [36, 41, 38], which provides an opportunity to utilize it for zero-shot text-to-3D generation.

Text-to-3D generation. DreamField [19] proposes a text-to-3D method using CLIP [35] guidance. DreamFusion [34] proposes a text-to-3D method using 2D diffusion models. Score Jacobian Chaining (SJC) [55] derives the training objective of text-to-3D using a 2D diffusion model from another theoretical basis. Magic3D [20] extends text-to-3D to a higher resolution with mesh [45] representation. Latent-NeRF [29] optimizes NeRF in latent space. Fantasia3D [4] optimizes a mesh with DMTet [45] from scratch. Although Fantasia3D achieves remarkable zero-shot text-to-3D generation, it requires user-provided shape guidance for the generation of complex geometry. [17] propose score debiasing and prompt debiasing to mitigate multiface problem and is orthogonal to our work. TextMesh [52] is contemporary with us and proposes a different pipeline for high-fidelity text-to-3D generation. 3DFuse [44] proposes to incorporate 3D awareness into 2D diffusion for better 3D consistency in text-to-3D generation. In addition, adjusting time schedule has also been discussed in previous works [12, 55]. However, previous works either require carefully devising the schedule [12] or perform inferior to the simple random time schedule [55]. Instead, our 2-stage annealing schedule is easy to train and achieves better generation quality than the random time schedule.

Text-driven large scene generation. Text2Room [16] generates indoor rooms from a given prompt. However, it uses additional monocular depth estimation models as prior, and we do not use any additional models. Our method generates the wild text prompt and uses only a text-to-image diffusion model. Set-the-scene [5] is a contemporary work with us aimed at large scene generation with a different pipeline. Overall, ProlificDreamer uses an advanced optimization algorithm, i.e, VSD with our proposed two-stage annealed time schedule, which has a significant advantage over the previous SDS / SJC (see Appendix C.3 for details). As a result, ProlificDreamer achieves high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops) and photo-realistic mesh results.

7 Conclusion

In this work, we systematically study the problem of text-to-3D generation. In terms of the algorithmic formulation, we propose variational score distillation (VSD), a principled particle-based variational framework that treats the 3D parameter as a random variable and infers its distribution. VSD naturally generalizes SDS in the variational formulation and addresses the practical issues of SDS observed before. With other orthogonal improvements to 3D representations, our overall approach, ProlificDreamer, can generate high-fidelity NeRF and photo-realistic textured meshes.

Limitations and broader impact. Although ProlificDreamer achieves remarkable text-to-3D results, the generation takes hours of time, which is much slower than image generation by a diffusion model. Although large scene generation can be achieved with our scene initialization, the camera poses during training are regardless of the scene structure, which may be improved by devising an adaptive camera pose range according to the scene structure for better-generated details. Moreover, due to the limited expressiveness of the base 2D model, the generation for complex prompts may fail, and the generated samples may have multi-face Janus problem [57] in some cases because of the poor text-image alignment for the view-dependent prompts. In addition, content creation by generative models may cause harmful social impacts on the labor market. Also, like other generative models, our method may be utilized to generate fake and malicious contents, which needs more caution.

References

Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers, 2023.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mipnerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5470-5479, 2022.

Changyou Chen, Ruiyi Zhang, Wenlin Wang, Bai Li, and Liqun Chen. A unified particleoptimization framework for scalable bayesian sampling. arXiv preprint arXiv:1805.11659 , 2018.

Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.13873 , 2023.

Dana Cohen-Bar, Elad Richardson, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Setthe-scene: Global-local training for generating controllable nerf scenes. arXiv preprint arXiv:2303.13450 , 2023.

Gianluca Detommaso, Tiangang Cui, Youssef Marzouk, Alessio Spantini, and Robert Scheichl. A stein variational newton method. Advances in Neural Information Processing Systems , 31, 2018.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Lauro Langosco di Langosco, Vincent Fortuin, and Heiko Strathmann. Neural variational gradient descent. arXiv preprint arXiv:2107.10731 , 2021.

Hanze Dong, Xi Wang, Yong Lin, and Tong Zhang. Particle-based variational inference with preconditioned functional gradient flow. arXiv preprint arXiv:2211.13954 , 2022.

Yihao Feng, Dilin Wang, and Qiang Liu. Learning to draw samples with amortized stein variational gradient descent. arXiv preprint arXiv:1707.06626 , 2017.

Clement Fuji Tsang, Maria Shugrina, Jean Francois Lafleche, Towaki Takikawa, Jiehan Wang, Charles Loop, Wenzheng Chen, Krishna Murthy Jatavallabhula, Edward Smith, Artem Rozantsev, Or Perel, Tianchang Shen, Jun Gao, Sanja Fidler, Gavriel State, Jason Gorski, Tommy Xiang, Jianing Li, Michael Li, and Rev Lebaredian. Kaolin: A pytorch library for accelerating 3d deep learning research. https://github.com/NVIDIAGameWorks/kaolin , 2022.

Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-play priors. arXiv preprint arXiv:2206.09012 , 2022.

Yuan-Chen Guo, Ying-Tian Liu, Chen Wang, Zi-Xin Zou, Guan Luo, Chia-Hao Chen, YanPei Cao, and Song-Hai Zhang. threestudio: A unified framework for 3d content generation. https://github.com/threestudio-project/threestudio , 2023.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. arXiv preprint arXiv:2303.11989 , 2023.

Susung Hong, Donghoon Ahn, and Seungryong Kim. Debiasing scores and prompts of 2d diffusion for robust text-to-3d generation. arXiv preprint arXiv:2303.15413 , 2023.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields, 2022.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:2211.10440 , 2022.

Chang Liu, Jingwei Zhuo, Pengyu Cheng, Ruiyi Zhang, and Jun Zhu. Understanding and accelerating particle-based variational inference. In International Conference on Machine Learning , pages 4082-4092. PMLR, 2019.

Chang Liu, Jingwei Zhuo, Pengyu Cheng, Ruiyi Zhang, Jun Zhu, and Lawrence Carin. Accelerated first-order methods on the wasserstein space for bayesian inference. stat , 1050:4, 2018.

Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems , 29, 2016.

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object, 2023.

Chang Liua and Jun Zhub. Geometry in sampling methods: A review on manifold mcmc and particle-based variational inference methods. Advancements in Bayesian Methods and Implementations , 47:239, 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927 , 2022.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models, 2023.

Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint arXiv:2211.07600 , 2022.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 Conference Papers , pages 1-8, 2022.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACMTransactions on Graphics (ToG) , 41(4):115, 2022.

Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint arXiv:1810.09136 , 2018.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence , 44(3):1623-1637, 2020.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Simo Ryu. Low-rank adaptation for fast text-to-image diffusion fine-tuning. https://github. com/cloneofsimo/lora . 2023.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487 , 2022.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models, 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, JinHwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint arXiv:2303.07937 , 2023.

Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations, 2021.

Harold Stewart. The experiencing of the dream and the transference. The International Journal of Psycho-Analysis , 54:345, 1973.

Jiaxiang Tang. Stable-dreamfusion: Text-to-3d with stable-diffusion, 2022. https://github.com/ashawkey/stable-dreamfusion.

Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. arXiv preprint arXiv:2304.12439 , 2023.

Cédric Villani et al. Optimal transport: old and new , volume 338. Springer, 2009.

Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https: //github.com/huggingface/diffusers , 2022.

Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:2212.00774 , 2022.

Ziyu Wang, Tongzheng Ren, Jun Zhu, and Bo Zhang. Function space particle optimization for bayesian neural networks. arXiv preprint arXiv:1902.09754 , 2019.

Wikipedia contributors. Janus - Wikipedia, the free encyclopedia. https://en.wikipedia. org/w/index.php?title=Janus , 2023.

Jingbo Zhang, Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao. Text2nerf: Text-driven 3d scene generation with neural radiance fields, 2023.

A Additional Experiment Results of 3D Textured Meshes

We provide more results of ProlificDreamer of 3D textured meshes in Fig. 6 and Fig. 7.

ROTARY

A banana peeling itself.

A delicious croissant.

A blue jay standing on a large basket of rainbow macarons.

B Additional Experiment Results of 3D NeRF

Here, we provide more generated results in Fig. 8 of high quality NeRF. We compare with DreamFusion, as DreamFusion also uses NeRF representation. It can be seen from the figure that ProlificDreamer generates better NeRF results in terms of fidelity and details, which demonstrates the effectiveness of our VSD against SDS.

In Fig. 9, we provide more results of large scenes with 360 ◦ environment using our scene initialization.

In Fig. 10, we provide more results of multiple particle experiments.

C Theory of Variational Score Distillation

C.1 Particle-based Variational Inference

Particle-based variational inference (ParVI) [23, 6, 3, 56, 25, 9] aims to draw (particle) samples from a desired distribution by minimizing the KL-divergence between particle samples and the desired distribution, and is widely-used in Bayesian inference [23, 10, 21]. Specifically, denote P (Θ) as the set containing all the distributions on a support space Θ with Euclidean distance, and W 2 (Θ) := { µ ∈ P (Θ) : ∃ θ 0 ∈ Θ s.t. E µ ( θ ) [ ∥ θ -θ 0 ∥ 2 ] < + ∞} as the 2-Wasserstein space [53] equipped with the Wasserstein 2-distance [53]. Given a desired distribution µ ∗ ∈ P (Θ) , ParVI

A car made out of sushi.

A tarantula, highly detailed.

starts with some ( n ) particles { θ ( i ) 0 } n i =1 sampled from an initial distribution µ 0 ∈ P (Θ) and update these particles with a vector field v τ ( · ) on Θ by d θ ( i ) τ = v τ ( θ ( i ) τ )d τ at each time τ ≥ 0 , such that the distribution µ τ of { θ ( i ) τ } n i =0 converges to µ ∗ as τ → ∞ and n → ∞ . Typically, the measure µ τ follows the 'steepest descending curves' of a functional on W 2 (Θ) , which is known as the Wasserstein gradient flow . A common setting is to define the functional of µ τ as the KL-divergence D KL ( µ τ ∥ µ ∗ ) , and then µ τ follows ∂ τ µ τ = -∇ θ · ( µ τ ∇ θ log µ ∗ µ τ ) , and the corresponding vector field is v τ ( θ τ ) = ∇ θ log µ ∗ µ τ . However, directly computing the vector field is intractable because it is non-trivial to compute the time-dependent score function ∇ θ log µ τ ( θ ) . To tackle this issue, traditional ParVI methods either restrict the functional gradient within RKHS and leverage analytical kernels to approximate the vector field [23, 3, 22], or learn a neural network to estimate the vector field [8]; but all of these methods are hard to scale up to high-dimensional data such as images.

C.2 VSD as Particle-based Variational Inference

In this section, we formally propose the theoretical guarantee of VSD.

Denote W 2 (Θ) as the 2-Wasserstein space [53] of the probability distributions defined on the parameter space Θ . For any distribution µ ∈ W 2 (Θ) with a corresponding random variable θ ∼ µ ( θ | y ) , denote the implicit distribution for x 0 := g ( θ, c ) as q µ 0 ( x 0 | c, y ) , and let q µ t ( x t | c, y ) := ∫ q µ 0 ( x 0 | c, y ) q t 0 ( x t | x 0 )d x 0 be the marginal distribution of the random variable x t during the diffusion process. We want to optimize the distribution µ ( θ | y ) in the function space W 2 (Θ) by distilling a pretrained diffusion model p t ( x t | y c ) via minimizing the following functional of µ :

Compared with the vanilla SDS in Eq. (2) which optimizes the parameter θ in the parameter space Θ , here we aim to optimize the distribution (measure) µ ( θ | y ) in the function space W 2 (Θ) . Inspired by previous ParVI techniques [23, 3, 56], we firstly derive the gradient flow minimizing E [ µ ] in W 2 (Θ) and the update rule of each particle θ ∼ µ ( θ ) , as shown in the following theorem.

Theorem 3 (Wasserstein gradient flow of score distillation) . Starting from an initial distribution µ 0 ( θ | y ) , the gradient flow { µ τ } τ ≥ 0 minimizing E [ µ τ ] on W 2 (Θ) at each time τ ≥ 0 satisfies

and the corresponding update rule for each θ τ ∼ µ τ satisfies

Theorem 3 shows that by letting the random variable θ τ ∼ µ τ ( θ τ | y ) move across the ODE trajectory in Eq. (12), its underlying distribution µ τ will move by the direction of the steepest descent that minimizes E [ µ ] . Therefore, to obtain samples (in Θ ) from µ ∗ = arg min µ E [ µ ] , we can simulate the ODE in Eq. (12) by estimating two score functions ∇ x t log p t ( x t | y c ) and ∇ x t log q µ τ t ( x t | c, y ) at each ODE time τ , which corresponds to the VSD objective in Eq. (9).

C.3 Discussions of SDS / SJC and VSD

There are three main differences between our proposed VSD and previous SDS [34] / SJC [55]: (1) VSD can optimize multiple ( n ≥ 1 ) 3D objects, while SDS / SJC only optimizes a single ( n = 1 ) 3D object. (2) VSD can generate high-fidelity results with the common (e.g., 7 . 5 ) CFG (for both n = 1 and n > 1 ), while SDS / SJC needs extremely large (e.g. 100 ) CFG. (3) SDS / SJC is a special case of VSD by using a single-point Dirac distribution µ ( θ | y ) ≈ δ ( θ -θ (1) ) as the variational distribution.

̸

SDS / SJC as a special case of VSD. Here we explain SDS / SJC under our variational inference framework. If we directly approximate µ ( θ | y ) ≈ δ ( θ -θ (1) ) by its empirical distribution with the single sample θ (1) , then we have q µ t ( x t | c, y ) ≈ N ( x t | α t g ( θ (1) , c ) , σ 2 t I ) and thus -σ t ∇ x t log q µ t ( x t | c, y ) ≈ ( x t -α t g ( θ (1) , c )) /σ t = ϵ , which recovers the vanilla SDS / SJC. Therefore, SDS / SJC is a special case of VSD for the underlying distribution µ ( θ | y ) by the empirical distribution with a single point. Such an approximation has no generalization ability for θ = θ (1) , and thus the updating direction by the score function -σ t ∇ x t log q µ t ( x t | c, y ) (or equivalently, the Gaussian noise ϵ ) may be rather bad at low-density regions, resulting poor samples for the final θ (1) . Instead, VSD regards θ (1) as samples from the underlying distribution µ ( θ | y ) and trains a neural network ϵ ϕ to approximate the corresponding score functions, thus can leverage the generalization ability of neural networks for better approximating the underlying distribution µ ( θ | y ) . Moreover, by using LoRA, VSD can additionally exploit the text prompt y in the estimation ϵ ϕ ( x t , t, c, y ) , while the Gaussian noise ϵ used in SDS cannot leverage the information from y .

SDS / SJC as mode-seeking, VSD as sampling. VSD aims to sample θ from the optimal µ ∗ , while SDS / SJC aims to find the optimal θ ∗ minimizing the objective in Eq. (2). As the global optimum θ ∗ in SDS / SJC is the mode of Eq. (2), SDS / SJC is also known as performing mode-seeking [34]. To demonstrate the differences between sampling and mode-seeking, we consider a special case of the rendering function g ( θ ) to decouple the optimization algorithm and the 3D representation. In particular, we set g ( θ, c ) ≡ θ for any c and θ ∈ R d , then the rendered image x = g ( θ, c ) = θ is the same 2D image as θ . In such a case, we have q µ 0 = µ , and it is easy to prove that µ ∗ = p 0 (according to Theorem 1). For VSD, sampling θ from µ ∗ is corresponding to the traditional ancestral sampling [27] from p 0 ; while for SDS / SJC, we have ∇ θ L SDS ( θ ) = E t, ϵ [ ω ( t ) ϵ pretrain ( x t , t, y c )] ≈ E t, ϵ [ -σ t ω ( t ) ∇ x t log p t ( x t | y c )] , and thus the optimal θ ∗ is the mode of the 'averaged likelihood' of p t for all t . However, it is common that the mode of deep generative models may have poor sample quality [33]. We show in Fig. 3 that under the same CFG ( 7 . 5 ), both VSD and ancestral sampling can generate good samples but the sample quality of SDS is quite poor.

SDS / SJC requires large CFG, while VSD is friendly to CFG. As VSD aims to sample θ from the optimal µ ∗ defined by the pretrained model ϵ pretrain, the effects by tuning the CFG in ϵ pretrain for

3D samples θ by VSD are quite similar to the effects for the 2D samples by the traditional ancestral sampling [14, 27]. Therefore, VSD can tune CFG as flexibly as the classic text-to-image methods, and we use the same setting of CFG (e.g. 7 . 5 ) as the common text-to-image generation task for the best performance. To the best of our knowledge, this for the first time addresses the problem in previous SDS [34, 20, 4, 29] that it usually requires a large CFG (i.e., 100 ). Specifically, SDS (Eq. (3)) uses ( ϵ pretrain ( x t , t, y c ) -ϵ ) while VSD (Eq. (9))) uses ( ϵ pretrain ( x t , t, y c ) -ϵ ϕ ( x t , t, c, y )) . For example, for the 2D special case of g ( θ, c ) ≡ θ , we have ∇ θ L SDS ( θ ) = E t, ϵ [ ω ( t ) ϵ pretrain ( x t , t, y c )] and ∇ θ L VSD ( θ ) = E t, ϵ [ ω ( t )( ϵ pretrain ( x t , t, y c ) -ϵ ϕ ( x t , t, c, y ))] . Intuitively, to obtain highly detailed samples, the updating direction for θ needs to be 'fine' and 'sharp'. As SDS only depends on ϵ pretrain , it needs a large CFG ( = 100 ) to make sure ϵ pretrain to be 'sharp' enough; however, large CFG, in turn, reduces the diversity of the results and also hurts the quality. Instead, VSD leverages an additional score function ϵ ϕ ( x t , t, c, y ) to give a more elaborate direction than the zero-mean Gaussian noise, and the updating direction can be rather 'fine' and 'sharp' due to the difference between ϵ pretrain and ϵ ϕ . We empirically find that VSD can obtain much better quality than SDS in both 2D and 3D generation.

C.4 Proof of Main Theorem

C.4.1 Proof of Theorem 1

Proof of Theorem 1. Denote x q 0 as the random variable following q µ 0 ( x 0 | c, y ) and x q t as the random variable following q µ t ( x t | c, y ) . By the definition of q µ t , we have

where ϵ ∼ N ( 0 , I ) . Therefore, the characteristic functions of q µ t and q µ 0 satisfy

Similarly, the characteristic functions of p t and p 0 satisfy

Therefore, we have

C.4.2 Proof of Theorem 3

As the gradient flow minimizing E [ µ ] on W 2 (Θ) satisfies

thus we only need to compute the first variation δ E [ µ ] δµ . We propose the following lemmas for computing δ E [ µ ] δµ .

Lemma 1. For p, q ∈ W 2 ( R d ) , for any x ∈ R d ,

Proof. This is a classic conclusion in particle-based variational inference (e.g., see Sec.3.2. in [3]).

Lemma 2. For a fixed c and x t ,

where x 0 = g ( θ, c ) .

Proof of Lemma 2. By the definition of q µ t , we have

so

Lemma 3.

where x t = α t g ( θ, c ) + σ t ϵ , ϵ ∼ N ( ϵ | 0 , I ) .

Proof of Lemma 3. By the chain rule of functional derivative, according to Lemma 1 and Lemma 2, we have

where ϵ ∼ N ( ϵ | 0 , I ) , x 0 = g ( θ, c ) , x t = α t x 0 + σ t ϵ .

Below we provide the proof of Theorem 3.

Proof of Theorem 3. According to Lemma 3, we have

where x t = α t g ( θ, c ) + σ t ϵ . So

Thus, the measure µ τ at step τ during the Wasserstein gradient flow minimizing E [ µ ] on W 2 (Θ) satisfies

By the definition of Fokker-Planck equation, the corresponding process of each particle θ τ at time τ satisfies

D Additional Ablation Study

D.1 Ablation Study on Large Scene Generation

Here we perform an ablation study on large scene generation to validate the effectiveness of our proposed improvements. We start from 64 rendering resolution, with SDS loss and our scene initialization. The results are shown in Figure 11. It can be seen from the figure that, with our scene initialization, the results are with 360 ◦ surroundings instead of being object-centric. Increasing rendering resolution is slightly beneficial. Adding annealed time schedule improves the visual quality of the results. Replacing SDS with VSD makes the results more realistic with more details.

SDS

64 Rendering

Scene Initialization

+ 512

Rendering

+ Annealed

Time Schedule

+ VSD

Depth

D.2 Ablation Study on Mesh Fine-tuning

Here we provide an ablation study on mesh fine-tuning. Fine-tuning with textured mesh further improves the quality compared to the NeRF result. Fine-tuning texture with VSD provides higher fidelity than SDS. Note that both VSD and SDS in mesh fine-tuning are based on and benefit from the high-fidelity NeRF results by our VSD. And it's crucial to get a high-quality NeRF with VSD at the first stage.

D.3 Ablation on Number of Particles

Here we provide ablation study on number of particles. We vary the number of particles in 1 , 2 , 4 , 8 and examine how the number of particles affects the generated results. The CFG of VSD is set as 7 . 5 . The results are shown in Fig. 13. As is shown in the figure, the diversity of the generated results is slightly larger as the number of particles increases. Meanwhile, the quality of generated results is not affected much by the number of particles. Owing to the high computation overhead to optimize 3D representations and limitations on computation resources, we now only test at most 8 particles. We provide a 2D experiment with 2048 particles in Appendix G to demonstrate the scalability of VSD. We leave the experiments of more particles in 3D as future work.

Figure 12: Pipeline of ProlificDreamer along with ablation study of VSD. After generating a highquality NeRF, we extract and finetune a textured mesh. VSD provides high-fidelity texture, while SDS tends to generate smoother results. Note that both VSD and SDS in mesh fine-tuning are based on and benefit from the high-fidelity NeRF results by our VSD. The prompt here is an elephant skull .

Algorithm 1 Variational Score Distillation

Input: Number of particles n ( ≥ 1) . Large text-to-image diffusion model ϵ pretrain. Learning rate η 1 and η 2 for 3D structures and diffusion model parameters, respectively. A prompt y .

1: initialize n 3D structures { θ ( i ) } n i =1 , a noise prediction model ϵ ϕ parameterized by ϕ .

2: while not converged do

3: Randomly sample θ ∼ { θ ( i ) } n i =1 and a camera pose c .

4: Render the 3D structure θ at pose c to get a 2D image x 0 = g ( θ, c ) .

5: θ ← θ -η 1 E t, ϵ ,c [ ω ( t ) ( ϵ pretrain ( x t , t, y c ) -ϵ ϕ ( x t , t, c, y )) ∂ g ( θ,c ) ∂θ ]

6: ϕ ← ϕ -η 2 ∇ ϕ E t,ϵ || ϵ ϕ ( x t , t, c, y ) -ϵ || 2 2 .

7: end while

8: return

D.4 Ablation on Rendering Resolution

Here we provide an ablation study on the rendering resolution during NeRF training with VSD. As shown in Fig. 14, training with a higher resolution produces better results with finer details. In addition, our VSD still provides competitive results under a lower training resolution ( 128 or 256 ), which is more computationally efficient than the 512 resolution.

E Algorithm for Variational Score Distillation

We provide a summarized algorithm of variational score distillation in Algorithm 1.

We initialize one or several 3D structures { θ ( i ) } n i =1 , a noise prediction model ϵ ϕ parameterized by ϕ . At each iteration, we sample a camera pose c from a pre-defined distribution as previous works [34, 20]. Then we render 2D image from 3D structures at pose c with differentiable rendering x 0 = g ( θ, c ) . To optimize 3D parameters θ , we compute the gradient direction of 2D image and then back propagate to the parameter of NeRF, using VSD as Eq. (9). To model the score of the variational distribution, we train an additional diffusion model ϵ ϕ parameterized by LoRA. We optimize Eq. (8) to train LoRA after optimization of 3D parameters, using the rendered image x 0 = g ( θ, c ) and pose c . Note that, at each iteration, we perform differentiable rendering only once but use the rendered image twice for both computing gradient direction with VSD and training LoRA. Thus the computation cost will not increase much compared to SDS.

F More Details on Implementation and Hyper-Parameters

Training details. We use v-prediction [42] to train our additional diffusion model ϵ ϕ . The camera pose c is fed into a 2-layer MLP and then added to timestep embedding at each U-Net block. NeRF/mesh and LoRA batch sizes are set to 1 owing to the computation limit. A larger batch size of

1 particle

Seed 1

Seed 2

Seed 3

Seed 4

Seed 5

Seed 6

Seed 7

Seed 8

Seed 1

Seed 3

Seed 2

Seed 4

2 particles

Seed 1

Seed 2

4 particles

8 particles

Seed 1

NeRF/mesh may improve the generated quality and we leave it in future work. The learning rate of LoRA is 0 . 0001 and the learning rate of hash grid encoder is 0 . 01 . We render in RGB color space for high-resolution synthesis, unlike [29, 55] that render in latent space. In addition, we set ω ( t ) = σ 2 t . For most experiments, we only use n = 1 particle for VSD to reduce the computation time (and we only use a batch size of 1, due to the computation resource limits). For NeRF rendering, we sample 96 points along each ray, with 64 samples in coarse stage and 32 samples in fine stage. We choose a single-layer MLP to decode the color and volumetric density from the hash grid encoder as previous work [20].

For object-centric scenes, we set the camera radius as U (1 . 0 , 1 . 5) . The bounding box size is set as 1 . 0 . For large scene generation, we enlarge the range of camera radius to U (0 . 1 , 2 . 3) for better details

(1) 64 rendering.

(2) 128 rendering.

(3) 256 rendering.

(4) 512 rendering.

and geometry consistency. We enlarge the bounding box size to 5 . 0 . For large scenes with a centric object, we set the range of camera radius as U (1 . 0 , 2 . 0) .

DreamFusion [34] introduces an explicit shading model based on normal vectors during the training process to enhance the geometry. We currently disable the shadings and it reduces the computational cost and memory consumption. We leave incorporating the shading model into our method as future work.

Optimization. We optimize 25 k steps for each particle with AdamW [26] optimizer in NeRF stage. We optimize 15 k steps for geometry fine-tuning and 30 k steps for texture fine-tuning. At each stage, for the first 5 k steps we sample time steps t ∼ U (0 . 02 , 0 . 98) and then directly change into t ∼ U (0 . 02 , 0 . 50) . For large scene generation, we delay the annealing time to 10 k steps since large scene generation requires more iterations to converge. The NeRF training stage consumes 17 / 17 / 18 / 27 GB GPU memory with 64 / 128 / 256 / 512 rendering resolution and batch size of 1 . The Mesh fine-tuning stage consumes around 17 GB GPU memory with 512 rendering resolution and batch size of 1 . The whole optimization process takes around several hours per particle on a single NVIDIA A100 GPU. We believe adding more GPUs in parallel will accelerate the generation process, and we leave it for future work.

Licenses Here we provide the URL, citations and licenses of the open-sourced assets we use in this work.

G 2D Experiments of Variational Score Distillation

Here we describe the details of the experiments on 2D images with Variation Score Distillation in the main text of Fig. 3. Here we set the number of particles as 8 . We train a smaller U-Net from scratch to estimate the variational score since the distribution of several 2D images is much simpler than the distribution of images rendered from different poses of 3D structures. The optimization takes around 6000 steps. Additional U-Net is optimized 1 step on every optimization step of the particle images. The learning rate of particle images is 0 . 03 and the learning rate of U-Net is 0.0001. The batch size of particles and U-Net are both set as 8 . Here we do not use annealed time schedule for both VSD and SDS for a fair comparison.

To further demonstrate the effectiveness of our VSD, we increase the number of particles to a large number of 2048 . The optimization takes around 20 k steps. The batch size of U-Net is 192 and the batch size of particles is set as 16 . The results are shown in Figure 15. It can be seen from the figure that our VSD generates plausible results even under a large number of particles, which demonstrates the scalability of VSD. Although, due to the high optimization cost of 3D representation, the number of particles in 3D experiments is relatively small, we demonstrate that VSD has the potential to scale up to more particles.

To match the 3D experiment setting, we also provide 2D experiments with LoRA and add annealed time schedule. The results are shown in Figure 16. The number of particles is set as 6 and CFG = 7 . 5 . Our VSD provides high-fidelity and diverse results in this setting. We also set the number of particles as 2048 and the results are shown in Figure 17.

We also provide the results of SDS for comparison in Fig. 18. Compared to our VSD, the results of SDS are smoother and lack details.

Moreover, we visualize VSD/SDS training phase of 2D in Fig. 19. Since the gradient is not directly readable, we visualize x +∆ x , which is the updated results if current sample optimizes via this gradient direction. As shown in Fig. 19, SDS tends to provide over-saturated and over-smooth gradient while VSD provides more natural-looking gradients with more details. As a consequence, VSD provides better final results.

H How will CFG weights affect diversity?

In this section, we explore how CFG affects the diversity of generated results. For VSD, we set the number of particles as 4 and run experiments with different CFG. For SDS, we run 4 times

(t = 400)

0 iters

300 iters

600 iters

1000 iters

𝑥0 +Δ𝑥

𝑥0

(t = 850)

𝑥0 +Δ𝑥

(1) Gradient visualization of SDS.

0 iters

300 iters

600 iters

1000 iters

(t = 400)

𝑥0 +Δ𝑥

𝑥0

(t = 850)

𝑥0 +Δ𝑥

(2) Gradient visualization of VSD (

ours

).

of generation with different random seeds. The results are shown in Figure 20. As shown in the figure, smaller CFG provides more diversity. We conjecture that this is because the distribution of smaller guidance weights has more diverse modes. However, when the CFG becomes too small (e.g., CFG = 1 ), it cannot provide enough guidance strength to generate plausible results. Therefore, for all our 3D experiments shown in the results, we set CFG = 7 . 5 as a trade-off between diversity and optimization stability. Note that SDS could not work well in such small CFG weights. Instead, our VSD provides a trade-off option between CFG weight and diversity, and it can generate more diverse results by simply setting a smaller CFG.

I Limitations and Discussions

Although ProlificDreamer achieves remarkable text-to-3D results, the generation takes hours (especially for high-resolution NeRF training), which is much slower than the vanilla image generation by a diffusion model. Speeding up the text-to-3D generation is another critical problem, and we leave it in future work.

Besides, our proposed scene initialization demonstrates its effectiveness in generating expansive scenes, yet there are still limitations, particularly with respect to camera positioning. Our current model sets the camera with a fixed view toward the scene's center, which serves object-centric scenes effectively but may be suboptimal for scenes with intricate geometry and detailed textures. Furthermore, despite our efforts to produce outcomes with a rich structure, occasional failures occur and the geometry reverts to a simplistic textured sphere. In order to address these limitations, future research could focus on developing improved camera poses capable of capturing and rendering scenes in finer detail. Furthermore, our present model relies solely on the text-to-image diffusion model without the assistance of other models. The integration of off-the-shelf depth estimation models [37], as utilized in other studies [16, 58], could potentially enhance the accuracy and detail of the scene generation process.

In addition, the correspondence between text prompts and generated results is sometimes insufficient, especially for complex prompts. We conjecture that it is because the ability to generate from complex prompts is limited by the text encoder of Stable Diffusion. In addition, the multi-face Janus problem [57] also exists in our case. Nevertheless, we believe our proposed VSD and other contributions are orthogonal to these problems, and the generation quality can be further improved by introducing more techniques, such as using a more powerful text-to-image diffusion model that understands view-dependent prompts better [34, 41], or a diffusion model with more 3D priors [24].

J User Study

For completeness, we follow previous works [20, 4] and conduct a user study by comparing ProlificDreamer with DreamFusion [34], Magic3D [20] and Fantasia3D [4] under 15 prompts, 5 prompts for each baseline. Since none of the baselines have released their codes, we can only use

VSD

CFG = 7.5

VSD

CFG = 1

VSD

CFG = 2

VSD CFG = 4

VSD

CFG = 25

Particle 1

VSD

CFG = 100

Particle 2

Particle 3

Particle 4

(a) VSD provides more diversity with lower CFG between particles.

Seed 1

SDS

CFG = 100

Seed 2

Seed 3

Seed 4

SDS CFG = 7.5

(b) SDS works with only large CFG weights.

Seed 1

VSD

CFG = 7.5

Seed 2

Seed 3

Seed 4

VSD

CFG = 2

VSD

CFG = 4

(c) VSD with single particle and different seeds provide slightly less diversity.

the figures from their papers, which limits the number of results from baselines for us to compare. So we currently only compare under 15 prompts. The volunteers are shown the generated results of our ProlificDreamer and baselines and asked to choose the better one in terms of fidelity, details and vividness. We collect results from 109 volunteers, yielding 1635 pairwise comparisons. The results are shown in Table 3. Our method outperforms all of the baselines.

K Quantitative Results

In this section, we add some quantitative results to demonstrate the effectiveness of VSD.

Experiment Setting For 3D experiments, we compute the FID score between rendered images by SDS/VSD and 2D sampled images by ancestral sampling, named as 3D-FID. Specifically, we select 100 prompts from previous works including DreamFusion, Magic3D and Fantasia3D. For each prompt, we use VSD (number of particles n =1, CFG=7.5) or SDS (CFG=100) to optimize one 3D object and render 10 images uniformly from the circumference at an angle of 30° above the horizon, and collect 1k images in total. To isolatedly compare VSD with SDS, we run with the default setting of the stage-1 NeRF training of ProlificDreamer (i.e., both VSD and SDS are in 512 resolution and use annealed t ).

In Table 4, We compute the FID score between the 1k samples from the 100 prompts and a 50k reference batch, which is sampled by 50-step DPM-Solver++ [28] with 500 images per prompt. In Table 5, due to the time and computation resource limits, we compare the results for VSD with n =4 under only 25 randomly-selected prompts from the aforementioned 100 prompts, and compare SDS, VSD ( n =1) and VSD ( n =4) with the 50-step DPM-Solver++ under these 25 prompts. For VSD ( n =4), as we can get 4 particles (3D objects) per prompt, we randomly select one particle per prompt and render the corresponding 10 images of the selected particle for fair comparison. For 2D experiments in Table 6, we follow the common setting of evaluating text-to-image models by computing FID on MSCOCO2014 validation set. Specifically, we randomly select 1k prompts and sample one image per prompt by either 50-step DPM-Solver++, SDS (CFG=100), VSD ( n =4, CFG=7.5) or VSD ( n =8, CFG=7.5) to collect 1k samples for each method, and then compute the FID between the obtained samples and the entire COCO validation set. For VSD, as we can get n images per prompt, we randomly select one image per prompt for fair comparison.

Results VSD with n =1 still outperforms SDS in 3D (both with 512 resolution and annealed t ), as shown in Table 4 and Table 5, which demonstrates the effectiveness of VSD.

Using more particles is slightly better. Due to the limitation of time and computation resources, we only compare n =1 and n =4 in 3D experiments, and n =4 with n =8 in the 2D experiments. As shown in Table 5, VSD with 4 particles slightly outperforms VSD with 1 particles in the 3D setting; and as shown in Table 6, VSD with 8 particles slightly outperforms VSD with 4 particles in the 2D setting.

VSD outperforms SDS in 2D. As shown in Table 6, the FID by VSD is much better than SDS. As the 2D setting isolates the sampling algorithm from the 3D representations, we can directly compare different sampling algorithms, finding that VSD can get better sample quality than SDS (though still worse than SOTA diffusion samplers, it can generalize to 3D cases).

L Why using SDS in stage-2 for the geometry optimization of mesh?

VSD can also be used to generate geometry. To validate this, we provide an ablation example in Fig. 21 (3a),(3b). As shown in the figure, VSD can obtain reasonable geometry. Although the some part of the geometry from VSD is with more details than SDS (including the tail of the horse), on the whole, the result from VSD is similar with SDS. We conjecture that this is because currently the triangle size of the mesh is relatively large and can't represent very fine details. Thus, for efficiency, we use SDS instead of VSD for mesh geometry optimization. We believe that VSD can be used to obtain high quality mesh if more advanced mesh representation is available.

Moreover, despite that we use SDS to optimize the geometry in stage-2, VSD is still crucial in stage-1 and stage-3, in which VSD significantly improves the generated quality.

(1) Generate

NeRF with

VSD.

(2) Get coarse

mesh with

(3a) Mesh optimization with SDS.

(3b) Mesh optimization with VSD.

(4) Inherit

texture from

NeRF.

(5) Finetune

texture with

VSD.

LoRA sample

during stage

(5).

M More Comparisons with Baselines

Here, we provide more comparisons with baselines in Fig. 22, Fig. 23, Fig. 24, Fig. 25, Fig. 26 and Fig. 27. Since none of the baselines have released their codes, we can only directly copy the figures from the corresponding papers. Some baselines are missing given a specific prompt because the prompt is not included in the corresponding papers. To demonstrate geometry, some baselines choose textureless shading [34, 20], while the other [4] prefers the normal map. For ProlificDreamer, we uniformly show the normal map for consistency. As shown in the figures, our method achieves better results in terms of fidelity and details.

A model of a house in Tudor style.

Ours

DreamFusion

A blue tulip.

A lionfish.

A cauldron full of gold coins.

A sliced loaf of fresh bread.

DreamFusion

Ours

Ours

Ours

Ours

DreamFusion

DreamFusion

DreamFusion

A rotary telephone carved out of wood.

A typewriter.

A praying mantis wearing roller.

DreamFusion

Ours

Ours

Ours

DreamFusion

DreamFusion

DreamFusion

A plush dragon toy.

Ours

ROTARY

A Matte painting of a castle made of cheesecake surrounded by a moat made of ice cream.

Ours

Ours

A small saguaro cactus planted in a clay pot.

DreamFusion

A pineapple.

Ours

A delicious croissant.

A rabbit, animated movie character, high detail 3d model.

Ours

DreamFusion

Ours

DreamFusion

Magic 3D

OP

A car made out of sushi.

A marble bust of a mouse.

Ours

Magic 3D

Ours

DreamFusion

Magic 3D

DreamFusion

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a nice distillation pipeline that can perform high-quality text-to-3D generation using 2D diffusion priors. By introducing the score on NeRF / mesh rendered images, the authors make score distillation work with a classifier-free guidance weight as low as 7.5. This approach largely improves the detail quality of the converged results and inspires the community with an exciting direction.\n\nSTRENGTHS:\n- The authors propose to estimate scores on rendered image with a LoRA framework fine-tuned for the target scene.\n- By introducing this additional score, the distillation framework is able to work with low classifier-free guidance weight at 7.5, yielding much better appearance than the original SDS loss in DreamFusion.\n- Both 2D and 3D experiments validate the expressiveness of the proposed framework.\n\nWEAKNESSES:\n- The paper starts with derivations to support the particle-based variational framework. However, using more particles doesn't seem to help with the generation results. Is using a single particle not enough? What's the benefit of multiple particles, and when do we need them?\n- In Eq.(8) for 3D experiments, the authors mention that there's a `c` for camera pose added for the conditioning of the diffusion model. How is this implemented? Is this camera pose necessary for the framework to work? It would be nice if there's an ablation study on removing the camera pose conditioning.\n- For 3D experiments, why do we need a LoRA model fine-tuned from a pre-trained Stable Diffusion? Does using a small U-Net work for 3D case?\n- For the mesh refinement stage, the authors follow Fantasia3D to refine the geometry and texture separately. SDS loss is used in the geometry stage that refines normal maps. I'm interested in whether the proposed vsd loss is helpful in this case. Adding a comparison experiment that uses VSD to refine the normal map will be great.\n- Existing methods mostly use view-dependent prompts, meaning that text embedding `y` and camera `c` are not independent. Do authors use view-dependent prompts in the experiments? \n- Why are the two rows in Eq. (5) equal? More derivations are appreciated.\n- Is v-prediction necessary for the LoRA model? Does the proposed framework work with eps prediction?\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nPlease refer to the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

comments: SUMMARY:
This paper proposes an interesting and novel technique for the task of text-to-3D generation. It defines a distribution of the target 3D scene, which is implemented as particles. Given the textual description, the distribution is updated using the Wasserstein gradient flow. Moreover, the paper proposes to fine-tune a diffusion model based on LoRA to calculate the score function of noisy rendered images, which is used to compute the objective function. Experiments show that the approach significantly improve the generation quality.

STRENGTHS:
1. The proposed technique is novel and interesting. Modeling 3D scenes as a distribution is insightful, which could inspire the following research.
2. It is a good idea to fine-tune LoRA-based diffusion models on noisy rendered images for calculating the objective function.
3. The results of the proposed model are impressive and obviously outperform previous methods.

WEAKNESSES:
1. I am curious about how the number of particles affect the performance of the proposed model.
2. Can this idea be used in amortized text-to-3D generative models? Does it consumes a lot of GPU memory?
3. It would be better to perform ablation studies on the distillation time schedule and the density initialization.

QUESTIONS:
Please see the weakness section.

LIMITATIONS:
The authors have discussed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, a novel method for generating 3D representations from text prompts is proposed. More specifically, a novel variational score distillation (VSD) based optimisation strategy is proposed that allows for optimising NeRF- and mesh-based 3D representations by utilising pre-trained 2D diffusion models. It improves over the previously in Dreamfusion proposed score distillation sampling (SDS) by optimising a distribution over the 3D parameters instead of a single point Dirac distribution. Further, the authors made additional engineering improvements to the text-to-3d pipeline. The proposed system does not rely on high CFG weights as required for SDS and leads to impressive results that compare favourably against the state-of-the-art.\n\nSTRENGTHS:\n- The authors clearly identify an important shortcoming of current text-to-3D approaches: suffering from over-saturation, over-smoothing, and low-diversity problems. They identify the SDS-based optimisation as the core problem, and propose a theoretically-sound alternative that can be trained with only a small overhead compared to SDS.\n- The shown results clearly look impressive and the obtained scene representations clearly improve over the state-of-the-art results.\n- The manuscript is (mostly) very well written, has a great structure, and a great \"reading flow\".\n- The performed experiments, including the 2D experiments, clearly highlight the authors' contributions and show the effectiveness of the individual components of the proposed methods.\n- The authors provide a clear and correct overview over their core contributions in Table 1 and demonstrate their familiarity with the fast-moving field of text-to-3D generation.\n\nWEAKNESSES:\n- Number of particles: I am sceptical if n=1 or n=4 (L. 150) is really enough to capture the distribution. It would be interesting to investigate the relationship of number of particles and the quality of the results; as it seems that the prediction network for the noisy rendered images is trained with all particles, it could be that the quality overall improves with more samples. \n\n- Geometry optimisation and VSD/SDS: Accordingly to L. 247 ff., I believe that the authors explain that the mesh geometry is fine-tuned with SDS, not VSD (see also question below). If understood correctly, this part should express more clearly that it is only referring to the mesh geometry, not the overall geometry optimisation. Further, this seems to be an interesting limitation of the current method. It would be interesting to see a respective ablation figure comparing SDS and VSD for this mesh-geometry prediction stage.\n\n- Geometry Diversity: Figure 1 c.) shows the diversity of samples, and it appears that the geometry diversity is limited compared to the texture diversity. It would be interesting to discuss potential reasons for this.\n\n- Experimental Results: No quantitive metrics are reported and the only user study is \"hidden\" in Table 3 of the supplementary material. While it is true that the task of text-to-3D is very challenging to be measured quantitatively, metrics reported by previous methods such as Dreamfusion could also be reported, and the user study can be shown in the main paper or at least be referenced. \n\n- Camera Pose Priors: It would be very interesting to see how the camera sampling strategies affect the results; in particular for the \"scene generation\" experiments. Have the authors started investigating this dimension of the problem?\n\n- Related work: A good and thorough related work discussion is contained only in the supplementary. I would encourage the authors to add a discussion, if accepted, to the main manuscript. \n\n- Noisy prediction network: The scores for the noisy real and the noisy rendered images are predicted by two separate models (L. 166), and the one for the rendered images is smaller / a low-rank approximation. Is this introducing some form of imbalance or is this not a problem for the optimisation?\n\n- Training of noisy renderings score prediction network: If understood correctly from the supplementary, the network for predicting the scores of the noisy rendered images is trained simultaneously with the NeRF model. It could be interesting to study how different optimisation schemes affect the results.\n\n- Input text prompt: I believe it is not stated whether the same text prompt or different text prompts are used for different camera poses.\n\nTypos:\n- L 178: speical case -> special case\n- L 283: variation score distillation -> variational score distillation\n- L. 284: 3D parameter -> 3D parameters\n- L 637 of supp mat: an addition diffusion model -> an additional diffusion model\n\nQUESTIONS:\n- Have the authors investigated whether there is a relationship between number of particles and the quality of the results?\n- Could the authors expand on the limited geometry variance in contrast to the texture variance, and why the use of the VSD is not improving over SDS for the mesh geometry prediction?\n- Could the authors expand on the importance of used camera pose priors, especially in the context of scene generation?\n- Are different text prompts used for different camera poses?\n\nLIMITATIONS:\nThe authors discuss limitations both in the main paper as well as more extensively in the supplementary. I believe the discussion is thorough and good as is.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Variational Score Distillation (VSD), which is a new generalization of Score Distillation Sampling (SDS) that is based on\n\nSTRENGTHS:\nThe biggest strength of this paper is likely in its practical applicability; notably, I find it impressive that the paper is able to produce good results with a batch size = 1. Many prior works need larger batch sizes, and especially mesh generation (as in Magic3D) suffer if they do not have large batch sizes which significantly hurts performance. This is significant, as it makes text-to-3D generation much more feasible without access to a large GPU cluster / DGX sort of machine. However, it's not unclear if this is really due to VSD (more on this in the weakness section).\n\nThere are also other practical benefits to this algorithm, like the additional diversity in the output that they can offer. This is also significant, as one of the practical needs in deploying a text-to-X system is being able to 'reroll' for better outputs that better align with the user's intent through the text prompt. This is something that VSD uniquely offers.\n\nThe paper is also very well written, and the writing is very clear despite the complexity of what is being described. I do have a couple of more detail-oriented questions that I will ask in the questions section.\n\nWEAKNESSES:\nI am slightly unconvinced about the effectiveness of VSD when it comes to increased quality. \n\nFirst, as a baseline, the generated results qualitatively do look much better than DreamFusion or Magic3D. The user studies agree, with an overwhelming margin (90+%). \n\nHowever, it is also the case that the generated results in this paper _without_ VSD and with just the higher resolution prior (and annealed t) already looks arguably significantly better than DreamFusion or Magic3D. Without a user study to then compare against this result with and without VSD, it's hard to make the claim that VSD itself is the factor that contributes most significantly to the improved quality. It's not entirely made clear in the paper what exactly is the factor that leads to this significantly quality increase. Empirically, high resolution NeRF training for SDS with Stable Diffusion has been difficult to make work (just even based on open source re-implementations of various papers), _especially_ with a small batch size of N=1 that they claim in the paper... so I am very curious what leads to the quality increase. But also it makes me question whether VSD really adds much in terms of quality without more examples or a user study. (Figure 5 does show an example where there seems to be perceived difference, but in my own experience, this is a minor difference that could just be caused by a slightly different hyper parameter)\n\nFigure 17, although it's only on a single example, somewhat confirms this since the high CFG weight seems to result in very similar results with the VSD results with low CFG. Although high CFG weight is a factor for over saturation, some of the results with VSD also seems to suffer from over saturation anyways. \n\nRegardless, VSD does offer other benefits like diversity. But it would be amazing to hear from the authors on their thoughts on this quality difference in the SDS baseline. Ideally a user study can compare between their baseline NeRF generation results with SDS against VSD, but I understand if that's difficult. \n\nAt the very least, there should be more examples (beyond the single example they show in Figure 5) that compare between their baseline NeRF results with SDS vs VSD. This would make this paper an extremely solid contribution. (it's also possible I just missed something).\n\nQUESTIONS:\n1. In Figure 3, the VSD results generate slightly more 'oversaturated' results in comparison to ancestral sampling. Similarly, the mesh results in the paper generally seem to still suffer from over saturation issues (although some of this is likely just due to not modeling accurate physically based rendering). Is there any explanation for the perceived differences between VSD and ancestral sampling and why they lead to these qualitative differences?\n\n2. For the SDS experiments (in Figure 17 in the appendix, for example), what are the experimental settings? Do they use the same batch size = 1 with all of the same hyper parameters as the VSD? What hyper parameter differences exist between this and DreamFusion, Magic3D?\n\n3. How does the SDS NeRF results from this paper (without VSD) compare to DreamFusion and Magic3D in terms of perceived quality? What are the factors that cause this quality difference?\n\nLIMITATIONS:\nThe limitations are addressed, although I have a lot of questions about the actual efficacy of VSD with respect to quality improvements. It would also be nice to have a broader statement on the societal impacts (even if it's somewhere in the appendix, unless I just didn't see it) since text-to-3D models are something that can bring potentially harmful impacts to the labor market around content creation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce the VSD algorithm that generalizes SDS to optimizing a distribution of shapes in terms of KL divergence to the image diffusion model.\nLacking analytic score of the implicit rendered image distribution, they train a surrogate by using LORA finetuning to the base image diffusion model.\nThe resulting algorithm is demonstrated for optimizing an empirical distribution of particles and interpreted in framework of particle VI.\nThey empirically find that a lower CFG is possible when using VSD as opposed to SDS which alleviates artefacts (even in the n=1 case).\n\nAdditionally, a simple NERF initialization for scenes, annealed time schedule and higher resolution rendering are proposed as orthogonal improvements.\n\nSTRENGTHS:\nNovel method: Using an auxiliary fine-tuned image diffusion model to approximate the score of an implicit distribution is a very neat idea and could have potential wider applicability in other areas of generative modelling.\n\nPresentation: The motivation behind the method, technical background and mathematical framework are presented well and the writing is clear in these aspects. Details regarding the method are also fairly clear.\n\nStrong qualitative result: The ability to use lower CFG is an interesting finding and the authors give evidence for its effectiveness in producing high quality results. Combined with the other practical additions to the training pipeline, this results in very impressive SOTA visual quality.\n\nWEAKNESSES:\nClearer experiments: There are a lot of different experimental settings which dilutes the evaluation of the core VSD contribution imo.\nThe paper seems to have 4 main experimental settings:\na. Single particle textured mesh objects\nb. Single particle NERFs (object and scene)\nc. Multi-particle NERF\nd. 2D VSD results  \n\nIn terms of multiple particles, these are only demonstrated for 1 2D prompt and 2 NERF prompts (and maybe 1 mesh in the appendix). Can multiple particle results be demonstrated for the prompts used in a/b.\n\nGiven the lack of quantitative metrics, the contributions of VSD, annealing t and higher resolution are only evaluated qualitatively on 1 or 2 prompts. Ablating VSD for more prompts would help strengthen the case for it. Currently it is only done for one of the prompts in setting a.\n\nLack of quantitative metrics: I understand there aren’t great established metrics in this area but currently the comparison is only done by showing a few qualitative examples and a user study with 5 prompts picked per baseline.\nEspecially since one of the claims in the paper is that VSD allows diverse generation, quantifying this over a representative set of prompts would greatly strengthen the evaluation section.\nR-precision has been reported in previous papers and given the motivation of matching in KL, a conditional FID to the image diffusion model could also be considered. \nIf computation is a bottleneck, then these could have also been obtained for a coarser version/2D case.\n\nClarity of explaining VSD results: Although VSD was motivated from the stand point of enabling diverse generations, the empirical benefits it brings in the single particle case is surprising and remains not well explained.\nHence, some of the discussions in section 3.3 (about “superior generalization” and regarding CFG on lines 185,186) seem too vague.\nI would recommend the authors could change the wording to make it more clear which results are empirical findings that may not have theoretical justification yet and further to write any intuition/speculation about the reason more clearly.\n\nQUESTIONS:\n- On line 247 it is mentioned that for mesh finetuning, VSD was not helpful for geometry. To clarify, the pipeline for the textured mesh then is: 1. VSD for Nerf init, 2. SDS on DMTet geom finetuning, 3. VSD on Texture finetuning? How would this compare to following Fantasia3D and skipping the 1st NERF stage? Can you show the intermediate results during each stage? Does the SDS stage affect the geometry much in stage 2?\n- Is the ice cream ablation in Figure 12 using the textured mesh pipeline?\n- In the appendix, there was some intuition about VSD proving more fine/sharp update direction. What do samples from the LoRA tuned model and the corresponding update direction look like?\n- The paper claims that SDS is one of the main bottlenecks to scaling resolutions to 512 (line 226). Could you elaborate on this, since it seems they are orthogonal?\n- For the 2D experiment, you tried the small U-Net with 2048 particles. Did you also try n=2048 for the LoRA? Are there issues with scaling LoRA?\n\nLIMITATIONS:\nThe limitations as expanded upon in the appendix is sufficient. Maybe the authors can expand more about the robustness of the method and sensitivity to hyperparameters which is a common issue in related works.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}