DreamFusion: Text-to-3D using 2D Diffusion
Authors: Ben Poole
Ajay Jain
Jonathan T. Barron
Ben Mildenhall
Venue: ICLR 2023 notable top 5% (2023)
Keywords: diffusion models
score-based generative models
NeRF
neural rendering
3d synthesis

Abstract: Recent breakthroughs in text-to-image synthesis have been driven by diffusion models trained on billions of image-text pairs. Adapting this approach to 3D synthesis would require large-scale datasets of labeled 3D or multiview data and efficient architectures for denoising 3D data, neither of which currently exist. In this work, we circumvent these limitations by using a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. We introduce a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image generator. Using this loss in a DeepDream-like procedure, we optimize a randomly-initialized 3D model (a Neural Radiance Field, or NeRF) via gradient descent such that its 2D renderings from random angles achieve a low loss. The resulting 3D model of the given text can be viewed from any angle, relit by arbitrary illumination, or composited into any 3D environment. Our approach requires no 3D training data and no modifications to the image diffusion model, demonstrating the effectiveness of pretrained image diffusion models as priors.

Published as a conference paper at ICLR 2023
DREAMFUSION: TEXT-TO-3D USING 2D DIFFUSION
Ben Poole', Ajay Jain', Jonathan T. Barron', Ben Mildenhall 'Google Research, PUC Berkeley
{pooleb, barron, umild]@googlle com, ajayj@berkeleyy.dd
ABSTRACT
Recent breakthroughs in text-to-image image synthesis have been driven by diffusion models trained on billions of image-tex pairs. Adapting this approach to 3D synthe- sis would require large-scale datasets of labeled 3D data and efficient architectures for denoising 3D data, neither of which currently exist. In this work, circum- vent these limitations by using a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. We introduce loss based on probability density distillation that enables the use of a 2D diffusion model as prior for optimization of parametric image generator. Using this loss DeepDream -iieeeaakkkeeeeeee procedure, we optimize a randomly- initialized 3D model (a Neural Radiance Field, or NeRF) -nnitaly-initaaiizeedd
via gradient descent such that its 2D renderings from random angles achieve a low loss. The resulting 3D model of the given text can be viewed from any angle, relit by arbitrary illumination, or composited into any 3D environment. Our approach requires no 3D training data and no modifications to the image diffusion model, demonstrating the effectiveness of pretrained image diffusion models as priors. See dreamfusi onpaeerrrrrr...r.. gi github. io for more immersive view into our 3D results.
INTRODUCTION
Generative image models conditioned on text now support high-fidelity diverse and controllable image synthesis (Nichol et al., 2022; Ramesh et al., 2021; 2022; Saharia et al., 2022; 2021a: Yu et al., 2022; Saharia al., 2021b). These quality improvements have come from large aligned image-text datasets (Schuhmann et al., 2022) and scalable generative model architectures. Diffusion models are particularly effective at learning high-quality image generators with stable and scalable denoising objective (Ho et al., 2020; Sohl-Dickstein et al., 2015; Song et al., 2021). Applying diffusion models other modalities has been successful, but requires large amounts of modality-speciff training data (Chen et al.. 2020; Ho et al.. 2022: Kong et al., 2021). In this work, we develop techniques to transfer pretrained 2D image-text diffusion models to 3D object synthesis, without any 3D data (see Figure 1). Though 2D image generation is widely applicable, simulators and digital media like video games and movies demand thousands of detailed 3D assets to populate rich interactive environments. 3D assets currently designed by hand in modeling software like Blender and Maya3D, a process requiring great deal time and expertise. Text-to-3D generative models could lower the barrier to entry for novices and improve the workflow of experienced artists.
3D generative models can be trained on explicit representations of structure like voxels (Wu et al., 2016; Chen et al., 2018) and point clouds (Yang et al.. 2019; Cai et al., 2020; Zhou et al., 2021), but the 3D data needed is relatively scarce compared to plentiful 2D images. Our approach learns 3D structure using only 2D diffusion model trained images, and sidesteps this issue. GANs can learn controllable 3D generators from photographs of a single object category, by placing an adversarial loss on 2D image renderings of the output 3D object scene (Henzler et al., 2019; Nguyen-Phuoc et al., 2019; Or-El et al., 2022). Though these approaches have yielded promising results on specific object categories such as faces, they have not yet been demonstrated support arbitrary text.
Neural Radiance Fields, or NeRF (Mildenhall et al., 2020) are an approach towards inverse rendering which a volumetric raytracer is combined with a neural mapping from spatial coordinates to color and volumetric density. NeRF has become a critical tool for neural inverse rendering (Tewari et al., 2022). Originally, NeRF was found work well for "classic" 3D reconstruction tasks: many images of scene are provided as input to model, and a NeRF is optimized to recover the geometry of that specific scene, which allows for novel views of that scene from unobserved angles to be synthesized.

---

Published as a conference paper at ICLR 2023
an orangutan making a clay throwing
racoonn astronaut holding
jay standing large rainbow macarons*
vorgi laking
table on
reading newspaper"
306
Michelangelo style dog reading news cellphone
liger doctor*
engine train, high
frog wearing
humanoid robot playing cello*
Sydney opera actial view
ltility vehicle driving stream!
chimpanzee dressed like VIII king of England
bahy hunny sitting top stack of pancakes
sliced loaf fresh bread
bulldozer clearing away pile of snow*
classic Packard
zoomed view of Tower Bridge made vut of gingerbread and candy* robot und dinosaur playing chess, high resolution
squirrel gesturing fromt of eusel showing colorful pie charts
pretrained text-to-image diffusion model generate realistic 3D models from text prompts. Rendered 3D models are presented from two views, with textureless renders and normals the right. See dreamfusil comffusionnaann onpaper .gi thub. io for videos of these results. Symbols indicate the following prompt prefixes which we found helped to improve the quality and realism:
Figure DreamFusion
DSSRR photo of...
zoomed out DSLR photo of...
wide angle zoumed DSLR photo of...
2

---

Published as a conference paper at ICLR 2023
Many 3D generative approaches have found success in incorporating NeRF-like models in the generative process (Schwarz et al., 2020; Chan et al., 2021b;a; Gu et al., 2021; Liu et al., 2022). One such approach is Dream Fields (Jain et al., 2022), which uses the frozen image-text joint embedding models from CLIP (Radford et al., 2021) and an optimization-base work showed that pretrained 2D image-text models may be used for 3D synthesis, though 3D objects produced by this approach tend to lack realism and accuracy. CLIP has been used to guide other approaches based on voxel grids and meshes (Sanghi al., 2022; Jetchev, 2021; Wang et al., 2022).
approach to train NeRFs. This
We adopt a similar approach to Dream Fields, but replace CLIP with a loss derived from distillation of a 2D diffusion model. Our loss is based on probabilty density distillation, minimizing the KL divergence between family of Gaussian distribution with shared means based on the forward process of diffusion and the score functions learned by the pretrained diffusion model. The resulting Score Distillation Sampling (SDS) method enables sampling via optimization in differentiable image parameterizations. By combining SDS with a NeRF variant tailored to this 3D generation task, DreamFusion generates high-fidelity coherent 3D objects and scenes for diverse set of user-provided text prompts.
DIFFUSION MODELS AND SCORE DISTILLATION SAMPLING
Diffusion models are latent-variable generative models that learn to gradually transform a sample from a tractable noise distribution towards data distribution (Sohl-Dickstein et al., 2015; Ho et al., 2020). Diffusion models consist of a forward process q that slowly removes structure from data by adding noise, and a reverse process or generative model that slowly adds structure starting from noise 2, The forward process is typically a Gaussian distribution that transitions from the previous less noisy latent at timestep & to noisier latent at timestep + 1. We can compute the marginal distribution of the latent variables at timestep t given an initial datapoint x by integrating out intermediate timesteps: (Zz x) (a;x,, I) The marginals integrating out the data density q(x) are ((z)) f (Zz,) x))(x) dx, and correspond to smoothed versions of the data distribution. The coefficients ay and og are chosen such that ((z,) is close to the data density the start of the process (oo = 0) and close to Gaussian at the end of the forward process (oT = 1), with at = 1 1 of
chosen to preserve variance (Kingma et al., 2021; Song et al., 2021).
The generative model p is trained to slowly add structure starting from random noise p(Zr) N
with transitions P& (z.-1/ |2t). Theoretically, with enough timesteps, the optimal reverse process step is also Gaussian and related to optimal MSE denoiser (Sohl-Dickstein et al., 2015). Transitions are typically parameterized as Pa (z/ = g(zc- 1 IZe, X = X (Zr; 8=(2;; f)) where g L2. x) is a posterior distribution derived from the forward process and 8 (2;; t) is a learned approximation of the optimal denoiser. Instead of directly predicting X Ho et al. (2020) trains an image-to-imagee U-Net <z(z;; t) that predicts the noise content of the latent zP: E[x| = X (z;it) (z, otEg(Zz; t)) /an- The predicted noise can be related to a predicted score function for the smoothed density Vz, log)(z)) through Tweedie's 's formula (Robbins, 1992): <0(2;; f) -2)5((1;))
Training the generative model with a (weighted) evidence lower bound (ELBO) simplifies to a weighted denoising score matching objective for parameters (Ho et al., 2020; Kingma et al., 2021):
Cpar( x) = E.-2:00.).c [u(t)]le(axxx + tte;t) =||2
(1)
where w(t) is a weighting function that depends on the timestep t. Diffusion model training can thereby be viewed either learning a latent-variable model (Sohl-Dickstein et al., 2015; Ho et al., 2020), or learning sequence of score functions corresponding to noisier versions of the data (Vincent,
2011; Song & Ermon, 2019; Song et al., 2021). We will use po(z;; t) to denote the approximate marginal distribution whose score function is given by s (Z:; () (ta L)/o1-
Our work builds on text-to-image diffusion models that learn c (2;; 1, y) conditioned on text embed- dings y (Saharia et al., 2022; Ramesh et al., 2022; Nichol et al., 2022). These models use classifier-frs free guidance (CFG, Ho & Salimans, 2022), which jointly learns an unconditional model to enable higher quality generation via a guidance scale parameter w: éØ y. f) = (1 + w) (Z:; y, t) we (zz; t). CFG alters the score function to prefer regions where the ratio of the conditional density to the unconditional density is large. In practice, setting > 0 improves sample fidelity at the cost of diversity. We use and p throughout to denote the guided version of the noise prediction and marginal distribution.
3

---

Published as a conference paper at ICLR 2023
Ancestral Sampling
Score Distillation Sampling
g(f,)
[(xi)
8
Updates sample pixel space: F,-1 ddpmn updale(pd
Updates parameters SGD: 0.:1 opt.step(0,, VaL(z.))
Figure 2: Comparison of 2D sampling methods from mext-to-image diffusion model with text "a
photo of tree frog wearing sweater." For score distillation sampling, an example we use an image generator that restricts images to be symmetric by having x (f)ip(0):
2.1 How CAN WE SAMPLE IN PARAMETER SPACE, NOT PIXEL SPACE?
Existing approaches for sampling from diffusion models generate sample that is the same type and dimensionality as the observed data the model was trained on (Song et al., 2021; 2020). Though conditional diffusion sampling enables quite bit of flexibility (e.g. inpainting), diffusion models trained on pixels have traditionally been used to sample only pixels. We are not interested in sampling pixels; we instead want to create 3D models that look like good images when rendered from random angles. Such models can be specified as a differentiable image parameterization (DIP, Mordvintsev al., 2018), where differentiable generator g transforms parameters 0 to create an image g()). DIPs allow us to express constraints, optimize in more compact spaces (e.g. arbitrary resolution coordinate- based MLPs), or leverage more powerful optimization algorithms for traversing pixel space. For 3D, we let 0 be parameters of 3D volume and g a volumetric renderer. To learn these parameters, we require a loss function that can be applied to diffusion models.
Our approach leverages the structure of diffusion models to enable tractable sampling via optimization loss function that, when minimized, yields a sample. We optimize over parameters 0 such that g(0) looks like sample from the frozen diffusion model. To perform this optimization, need a differentiable loss function where plausible images have low loss, and implausible images have high loss, in similar style DeepDream (Mordvintsev al., 2015). We first investigated reusing the diffusion training loss (Eqn. 1) to find modes of the learned conditional density p(x yy. While modes of generative models in high dimensions are often far from typical samples (Nalisnick et al., 2018), the multiscale nature of diffusion model training may help to avoid these pathologies. Minimizing the diffusion training loss with respect to generated datapoint x g()) gives 0r = arg ming Cpitré, x = g()). In practice. we found that this loss function did not produce realistic samples even when using an identity DIP where x Concurrent work from Graikos et al. (2022) shows that this method can be made to work with carefully chosen timestep schedules, but we found this objective brittle and its timestep schedules challenging tune.
To understand the difficulties of this approach, consider the gradient of Lpi
VaLDin(o. =g(0)) Er,, w(t) (Eg(zis v. E) des(Zi:
Ōx D0
(2)
Ze
Residual
Jacobian
Jacobian
where we absorb the constant a; = /zz/Ox into w(t), and use the classifier weessffier-fre-guiiie free-guided é.. In practice, the U-Net Jacobian term is expensive compute (requires backpropagating through the diffusion model U-Net), and poorly conditioned for small noise levels as it is trained to approximate the scaled Hessian of the marginal density. We found that omitting the U-Net Jacobian term leads to an effective gradient for optimizing DIPs with diffusion models:
Ox
VoLsDs x =()) Ey, wo()) (Ed(2;;)))) -)
00
Intuitively, this loss perturbs x with a random amount of noise corresponding to the timestep I, and estimates an update direction that follows the score function of the diffusion model to move to a higher density region. While this gradient for learning DIPs with diffusion models may appear ad hoc, in Appendix A.4 we show that it the gradient of a weighted probability density distillation loss (van den Oord et al., 2018) using the learned score functions from the diffusion model:
V VoLsns(áh (g. (()) VoR,
1)ll y, v.)))
4

---

Published as a conference paper at ICLR 2023
RIA grote praceck on curfbeard"
Pilght
Sol
Bormals 7
sbudingg
U-Nes
Albsllop
P(camera)
NolFF
MLP((: 0)
Eolit on
( (gr:4)
Figure 3: DreamFusion generates 3D objects from a natural language caption such as "a DSLR photo of a peacock on surfboard."" " The scene is represented by a Neural Radiance Field that is randomly initialized and trained from scratch for each caption. Our NeRF parameterizes volumetric density and albedo (color) with an MLP. We render the NeRF from random camera, using normals computed from gradients of the density to shade the scene with a random lighting direction. Shading reveals geometric details that are ambiguous from single viewpoint. To compute parameter updates, DreamFusion diffuses the rendering and reconstructs it with (frozen) conditional Imagen model to predict the injected noise e (.,.///: y;: t). This contains structure that should improve fidelity, but is high variance. Subtracting the injected noise produces low variance update direction stopgrad [E -e] that is backpropagated through the rendering process to update the NeRF MLP parameters.
We name our sampling approach Score Distillation Sampling (SDS) as it is related to distillation, but uses score functions instead of densities. We refer as a sampler because noise the variational family g(z,)... ...) disappears as / + 0 and the mean parameter of the variational distribution (() becomes the sample of interest. Our loss is easy to implement (see Fig. 8), and relatively robust to the choice of weighting w()). Since the diffusion model directly predicts the update direction, do not need to backpropagate through the diffusion model; the model simply acts like an efficient, frozen critic that predicts image-space edits.
Given the mode seeking nature Lsus. may be unclear minimizing this loss will produce good samples. In Fig. 2, demonstrate that SDS generate constrained images with reasonable quality. Empirically, we found that setting the guidance weight w to a large value for classifier-fre guidance improves quality (Appendix Table 10). SDS produces detail comparable ancestral sampling, but enables new transfer learning applications because it operates in parameter space.
3 THE DREAMFUSION ALGORITHM
Now that have demonstrated how diffusion model can be used loss within generic continuous optimization problem to generate samples, we will construct our specific algorithm that allows us generate 3D assets from text. For the diffusion model, we the Imagen model from Saharia et al. (2022), which has been trained to synthesize images from text. We only use the 64 64 base model (not the super-resolution cascade for generating riggherrreooution images), and use this pretrained model as-is with no modifications. To synthesize scene from text, we initialize NeRF-like model with random weights, then repeatedly render views of that NeRF from random camera positions and angles, using these renderings as the input to our score distillation loss function that wraps around Imagen. As we will demonstrate, simple gradient descent with this approach eventually results in 3D model (parameterized NeRF) that resembles the text. See Fig. 3 for an overview our approach.
3.1 NEURAL RENDERING OF A 3D MODEL
NeRF a technique for neural inverse rendering that consists a volumetric raytracer and a multilayer perceptron (MLP). Rendering image from NeRF is done by casting ray for each pixel from a camera's 's center projection through the pixel's location the image plane and out into the world. Sampled 3D points z along each ray then passed through MI.P, which produces
5

---

Published as a conference paper at ICLR 2023
scalar values output: volumetric density T (how opaque the scene geometry at that 3D coordinate is) and an RGB color c. These densities and colors are then alpha-compositee from the back of the ray towards the camera, producing the final rendered RGB value for the pixel:
C -= w.
wi = aasIIj= - aa)),
a; =
exp (-T; 1 -4:-11l)))
(5)
In the traditional NeRF use-case we are given a dataset of input images and associated camera positions and the NeRF MLP is trained from random initialization using a mean squared error loss function between each pixel's rendered color and the corresponding ground-truth color from the input image. This yields 3D model (parameterized by the weights of the MLP) that can produce realistic renderings from previously-unse
views. Our model built upon mip-NeRF 360 (Barron et al., 2022), which is an improved version of NeRF that reduces aliasing. Though mip-NeRF 360 was originally designed for 3D reconstruction from images, its improvements are also helpful for our generative text-to-3D task (see Appendix for details).
Shading, Traditional NeRF models emit radiance, which is RGB color conditioned on the ray direction from which the 3D point is being observed. In contrast, our MLP parameterizes the color of the surface itself, which is then lit by an illumination that we control (a process commonly referred to as "shading"). ) Previous work generative or multiview 3D reconstruction using NeRF-like models have proposed a variety of reflectance models (Bi et al., 2020; Boss et al., 2021; Srinivasan et al., 2021; Pan et al., 2022). We use an RGB albedo p (the color of the material) for each point:
(T, p) = MLP (p:: 8)
(6)
where T is volumetric density. Calculating the final shaded output color for the 3D point requires normal vector indicating the local orientation of the object's geometry. This surface normal vector can be computed by normalizing the negative gradient of density with respect to the 3D coordinate μi; n = P / اا (Yariv et al., 2020; Srinivasan et al., 2021). With each normal n and material albedo p. assuming some point light source with 3D coordinate l and color Lp, and an ambient light color l,, we render each point along the ray using diffuse reflectance (Lambert, 1760; Ramamoorthi & Hanrahan, 2001) produce color e for each point:
c p o (ep o max (0, n (e (e-w)) /1// l1e - 11l) + l.)
With these colors and our integral with the same rendering weights w; used in standard NeRF (Equation 5). As previous work on text-to-3D generation (Hong et al., 2022; Michel et al., 2021), we find it beneficial to randomly replace the albedo color P with white (1, 1) to produce a "textureless" shaded output. This prevents the model from producing degenerate solution in which scene content is drawn onto flat geometry to satisfy the text conditioning. For example, this encourages optimization to yield a 3D squirrel instead of flat surface containing an image of squirrel, both of which may appear identical from certain viewing angles and illumination conditions.
densities, we approximate the volume rendering
Scene Structure. While our method can generate some complex scenes, we find that it is helpful to only query the NeRF scene representation within a fixed bounding sphere, and use an environment map generated from second MLP that takes custtinnally-encdde ray direction as input compute a background color (architecture details in Appendix A.2). We composite the rendered ray color on top of this background color using the accumulated a value. This prevents the NeRF model from filling up space with density very close to the camera while still allowing it to paint an appropriate color or backdrop behind the generated scene. For generating single objects instead of scenes, a reduced bounding sphere can be useful.
Geometry regularizers. The mip-NeRF 360 model we build upon contains many other details that we omit for brevity. We include regularization penalty on the opacity along each ray similar to Jain et al. (2022) to prevent unneccesarily filling in of empty space. To prevent pathologies in the density field where normal vectors face backwards away from the camera we use a modified version of the orientation loss proposed in Ref-NeRF (Verbin et al., 2022). This penalty is important when including textureless shading as the density field will otherwise attempt to orient normals away from the camera so that the shading becomes darker. Full details on these regularizers and additional hyperparameters of NeRF are presented in the Appendix A.2.
3.2 TEXT-TO-3D SYNTHESTS
Given a pretrained toxt-to-ing -mage diffusion model, a differentiable image parameterization in the form of NeRF, and loss function whose minima good samples, we have all the components needed

---

Published as a conference paper at ICLR 2023
DSLR placto
wearing losther judket
rding molorcycle
on rood
a squircll
nancclu uuz
severiag
playuss the
thrugh foll
briedd
modievel wat artierr
eatla
laca
Figure 4: DreamFusion can used to create and refine 3D scenes. Here we iteratively refine an example text prompt, while rendering each generated scene from four different viewpoints.
for text-to-3D synthesis using no 3D data. For each text prompt, we train randomly initialized NeRF from scratch. Each iteration of DreamFusion optimization performs the following: (1) randomly sample a camera and light, (2) render an image of the NeRF from that camera and shade with the light, (3) compute gradients of the SDS loss with respect to the NeRF parameters, (4) update the NeRF parameters using an optimizer. We detail each of these steps below, and present pseudocode in Appendix 8.
1. Random camera and light sampling. At each iteration, a camera position is randomly sampled in spherical coordinates, with elevation angle 6eam [-100, 90%) azimuth angle Ocam E [0 360%), and distance from the origin in [1, 1.5). For reference, the scene bounding sphere described previously has radius 1.4. We also sample "look-at" point around the origin and an "up" vector, and combine these with the camera position to create a camera pose matrix. We additionally sample a focal length multiplier Afocal E 20077, 1.35) such that the focal length Afocal W. where w 64 the image width in pixels. The point light position e is sampled from distribution centered around the camera position. We found using a wide range of camera locations be critical for synthesizing coherent 3D scenes, and wide range of camera distances helps to improve the resolution of the learned scene.
Rendering. Given the camera pose and light position, we render the shaded NeRF model at 64 x 64 resolution as described in Section 3.1. We randomly choose between the illuminated color render, a textureless render, and a rendering of the albedo without any shading.
3. Diffusion loss with view-dependers conditioning. Text prompts often describe canonical views of an object that are not good descriptions when sampling different views. We therefore found beneficial append view-dependent text to the provided input text based on the location of the randomly sampled camera. For high elevation angles Peam > 60", we append "overhead view." For deam 60%,
weighted combination of the text embeddings for appending "front view," "side view." or "back view" depending on the value of the azimuth angle Oeam (see App. A.2 for details). We use the pretrained 64 64 base text-to-image model from Saharia et al. (2022). This model was trained on large-scale web-image-text data, and is conditioned on T5-XXL text embeddings (Raffel al., 2020). We a weighting function of wt()) of. but found that a uniform weighting performed similarly. We sample t (0.022, 0.98), avoiding very high and low noise levels due to numerical instabilities. For classifier-fres guidance, we set w 100, finding that higher guidance weights give improved sample quality. This is much larger than image sampling methods, and is likely required due to the mode-seeking nature of our objective which results oversmoothing at small guidance weights (sce Appendix Table. 10). Given the rendered image and sampled timestep t, we sample noise e and compute the gradient of the NeRF parameters according to Eqn. 3.
4. Optimization. Our 3D scenes are optimized
TPUv4 machine with 4 chips. Each chip renders a separate view and evaluates the diffusion U-Net with per-device batch size of 1. We optimize for 15,000 iterations which takes around 1.5 hours. Compute time is split evenly between rendering the NeRF and evaluating the diffusion model. Parameters are optimized using the Distributed Shampoo optimizer (Anil et al., 2020). See Appendix A.2 for optimization settings.
7

---

Published as a conference paper at ICLR 2023
Table 1: Evaluating the coherence of DreamFusion generations with their caption using different CLIP retrieval models. We compare the ground-truth MS-COCO images the object-centric subset of Jain et al. (2022) well as Khalid al. (2022). Evaluated with only I seed per prompt. Metries shown in parentheses may be overfit, as the same CLIP model is used during training and eval.
Dream Fields
Dream
Fields
(riimpl.
CLIP-
R- R-PPreisson f CLIP-B/32 CLIP-B/I6 CLIP-L/I4 Geo Color Geo
Mesh
Method
Color Geo Color
Dream-
GT Images
77.1
79.1
Fusion
Dream Fields
68.3
74.2
(Ouzs)
(reimpl.))
78.6
1.3
(99.9)
(0.8)
82.9
CLIP-Mesh
67.8
75.8
74.51
cArtO pabetiig of moelle saad somonakii
with pink Bowers
hamburger
DreamFusion (base)
81.5
0.02
84.7
0.03
88.4
0.05
DreamFusion
75.1
42.5
77.5
46.6
79.7
58.5
Figure 5: Qualitative comparison with baselines.
EXPERIMENTS
We evaluate the ability of DreamFusion to generate coherent 3D scenes from variety of text prompts. We compare to existing zero-shot text-to-3D generative models, identify the key components of our model that enable accurate 3D geometry, and explore the qualitative capabilities of DreamFusion such as the compositional generation shown in Figure
3D reconstruction tasks typically evaluated using reference-based metrics which compares recovered geometry to some ground truth. The view-synthesis literature often uses PSNR to compare rendered views with held-out photograph. These reference- based metrics diffieult to apply to zero-shot text-to-3D generation, as there is no "true" 3D scene corresponding to our text prompts. Following Jain et al. (2022). evaluate the CLIP R-Precision (Park et al., 2021a), an automated metric for the consistency of rendered images with respect to the input caption. The R P-Precision is the accuracy with which CLIP (Radford al., 2021) retrieves the correct caption among set of distractors given rendering of the scene. We use the 153 prompts from the object-centric COCO validation subset of Dream Fields. We also measure CLIP R- R-Precision on textureless renders to evaluate geometry since we found existing metrics do not capture the quality of the geometry, often yielding high values when texture is painted on flat geometry.
Table 1 reports CLIP R-Precision for DreamFusion and several baselines. These include Dream Fields, CLIP-Mesh (which optimizes a mesh with CLIP), DreamFusion without any view augmentations, view-dependent prompts, or shading (base) and an oracle that evaluates the original captioned image pairs AS-COOO. We also compare against an enhanced reimplementation use own 3D representation (Sec. 3.1). Since this evaluation is based on CLIP, Dream Fields and CLIP-Mesh have an unfair advantage as they use CLIP during training. Despite this, DreamFusion outperforms both baselines on color images, and approaches the performance of ground truth images. While our implementation of Dream Fields performs nearly at chance when evaluating geometry (Geo) with textureless renders, DreamFusion is consistent with captions 58.5% of the time. See Appendix A.3 for more details of the experimental setup.
Dream Fields where
Ablations. Fig. shows CLIP R- R-Prrcissin for simplified DreamFusion ablation and progressively adds in optimization choices: large ranges of viewpoints (ViewAuug), -eeendepndd prompts (ViewDep), optimizing illuminated renders in addition to unlit albedo color renders (Lighting), and optimizing textureless shaded geometry images (Textureless). We measure R- R-Precision on the albedo render as in baselines (left), the full shaded render (middle) and the textureless render (right) to check geometric quality. Geometry significantly improves with each of these choices and full renderings improve by +12.5%. Fig. 6 shows qualitative results for the ablation. This ablation also highlights how the albedo renders can be deceiving: our base model achieves the highest score, but exhibits poor geometry (the dog has multiple heads). Recovering accurate geometry requires view-dependent prompts, illumination and textureless renders.
8

---

Published as a conference paper at ICLR 2023
Base
1.0
+ +ViewAug (i)
+ViewDep (ii)
0.8
Lighhting (iii
R-Precision
Textureless (iv)
0.6
0.4
0.2
0.0 Albedo
Shaded
Textureless
Render Type
(ii)
(iii)
(iv)
We evaluate components of our unlit renderings on albedo, full shaded and illuminated renderings and textureless illuminated geometry using CLIP L/14 on object-centric COCO. Right: visualizations of the impact of each ablation for "A bulldog is wearing black pirate hat." on albedo (top), shaded (middle), and textureless renderings (bottom). The base method (i) without view-dependent prompts results in multi-faced dog with flat geometry. Adding in view dee-deepndeett prompts (ii) improves geometry. but the surfaces are highly non-smooth and result in poor shaded renders. Introducing lighting (iii) improves geometry but darker areas (e.g. the hat) remain non-smooth. Rendering without color (iv) helps smooth the geometry, but also causes some color details like the skull and crossbones to be "carved" into the geometry.
Figure 6: An ablation study of DreamFusion. Left:
DISCUSSION
We have presented DrcamFusion. effective technique for text-to-3D synthesis for wide range of text prompts. DreamFusion works by transferring scalable, high-quality 2D) image diffusion models to the 3D domain through use of novel Score Distillation Sampling approach and novel NeRF-like rendering engine. DreamFusion does not require 3D or multi-view training data, and uses only pre-trained 2D diffusion model (trained on only 2D images) perform 3D synthesis.
Though DreamFusion produces compelling results and outperforms prior work on this task, it still has several limitations. SDS is not perfect loss function when applied to image sampling, and often produces oversaturated and oversmoothed results relative to ancestral sampling. While dynamic thresholding (Saharia al., 2022) partially ameliorates this issue when applying SDS images, did not resolve this issue in NeRF context. Additionally, 2D image samples produced using SDS tend to lack diversity compared to ancestral sampling, and our 3D results exhibit few differences across random seeds. This may be fundamental to our use of reverse KL divergence, which has been previously noted to have mode-seeking properties in the context of variational inference and probability density distillation.
DreamFusion uses the 64 x 64 Imagen model, and as such our 3D synthesized models tend to lack fine details. Using higher-resolution diffusion model and a bigger NeRF would presumably address this, but synthesis would become impractically slow. Hopefully improvements in the efficiency of diffusion and neural rendering will enable tractable 3D synthesis high resolution in the future.
The problem of 3D reconstruction from 2D observations is widely understood to be highly ill-posed, and this ambiguity has consequences in the context of 3D synthesis. Fundamentally, our task is hard for the same reason that inverse rendering is hard: there exist many possible 3D worlds that result in identical 2D images. The optimization landscape of our task is therefore highly non-convex, and many of the details of this work are designed specifically to sidestep these local minima. But despite our best efforts we still sometimes observe local minima, such as 3D reconstructions where all scene content is "painted" onto a single flat surface. Though the techniques presented in this work are effective, this task of "lifting" 2D observations into 3D world is inherently ambiguous, and may benefit from more robust 3D priors.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Generative models for synthesizing images carry with them several ethical concerns, and these concerns are shared by (or perhaps exacerbated in) 3D generative models such as ours.
Because DreamFusion uses the Imagen diffusion model as a prior, it inherits any problematic biases and limitations that Imagen may have. While Imagen's 's dataset was partially filtered, the LAION- 400M (Schuhmann et al., 2022) subset of its data was found contain undesirable images (Birhane et al., 2021). Imagen is also conditioned on features from a pretrained large language model, which itself may have unwanted biases. It is important to be careful about the contents of datasets that are used in text-to-image and image-to-3D models so to not propagate hateful media.
Generative models, in the hands of bad actors, could be used to generate disinformation. Disinforma- tion in the form of 3D objects may be more convincing than 2D images (though renderings of our synthesized 3D models are less realistic than the state of the art in 2D image synthesis).
Generative models such That said, these tools may also enable growth and improve accessibility for the creative industry.
may have the potential displace creative workers via automation.
REPRODUCIBILITY STATEMENT
The mip-NeRF 360 model that we build upon is publicly available through the "MultiNeRI" code repository (Mildenhall et al., 2022). While the Imagen diffusion model not publicly available, other conditional diffusion models may produce similar results with the DreamFusion algorithm. To aid reproducibility, we have included schematic overview of the algorithm Figure 3, pseudocode for Score Distillation Sampling in Figure 8, hyperparameters in Appendix A.2, and additional evaluation setup details Appendix A.3. Derivations for our loss are also included in Appendix A.4.
REFERENCES
Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning, 2020. URL https://arxiv.cr wttpvvv...r/bbb///2.00000 20tps//.............
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv: 2016.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Bruuala,, and Pratul P Srinivasan. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. ICCV, 2021.
Jonathan Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv:2008.03822 2020.
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pomography, and malignant stereotypes. arXiv:2110001633, 2021.
Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch. NeRD: Neural reflectance decomposition from image collections. ICCV, 2021.
Ruojin Cai, Guandao Yang. Hadar Averbuch-Elor. Zekun Hao, Serge Belongie. Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. ECCV, 2020.
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. arXiv, 2021a
Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-GAN: Periodic implicit generative adversarial networks for 3d-aware image synthesis. CVPR, 2021b.
10

---

Published as a conference paper at ICLR 2023
Kevin Chen, Christopher Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings.
arXiv:1803.08495, 2018.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXhv:2009.00713 2020.
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-play priors. arXiv:2206.09012 2022.
Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, and Richard Zemel. Learning the stein discrepancy for training and evaluating merggy-based models without sampling. ICML, 2020.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. StyleNeRF: A style-based 3d-aware generator for high-resolution image synthesis, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. CVPR, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 1rXiv:/60600841555 2016.
Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Escaping Plato's Cave: 3D shape from adversarial rendering. ICCV, 2019.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12599, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 2020.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. vrXiv:2204.0345. 2022.
Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. AvatarCLIP: Zero-shot text-driven generation and animation of 3D avatars. ACM Transactions on Graphics (TOG), 2022.
Tianyang Hu, Zixiang Chen, Hanxi Sun, Jincheng Bai, Mao Ye, and Guang Cheng. Stein neural sampler. arXiv:1810.0354 2018.
Chin-Wei Huang, Faruk Ahmed, Kundan Kumar, Alexandre Lacoste, and Aaron C. Courville. Probability distillation: A caveat and alternatives. UAI, 2019.
Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. CVPR, 2022.
Nikolay Jetchev. ClipMatrix: Text-controlled creation of 3d textured meshes, arX9v:2109.12922. 2021.
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Popa Tiberiu. CLIP-Mesh: Gen- crating textured meshes from text using pretrained image-text models. SIGGRAPH Asia 2022 Conference Papers, 2022.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. NeurlPS, 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. ICLR, 2021.
Johann Heinrich Lambert. Photometria sive de mensura el gradibus luminis, colorum el umbrue. sumptibus vidvae F. Klett, typis CP Detleffsen, 1760.
Jae Hyun Lim, Aaron C. Courville, Christopher J. Pal, and Chin- Wei Huang. AR-DAE: towards unbiased neural entropy gradient estimation. JCML, 2020.
11

---

Published as a conference paper at ICLR 2023
Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi-Wing Fu. Towards implicit text-guided 3d shape generation. CVPR, 2022.
Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Tex12Mesh: Text-driven neural stylization for meshes. arXiv:2112.03221, 2021.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020.
Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, Peter Hedman, Ricardo Martin-Brualle and Jonathan T. Barron. MultiNekF: A Code Release for Mip-NeRF 360, Ref-NeRF, and RawNeRF, 2022. URL https://gt www...coooo.a...oooococcoo htto://goggg-rreeeerch//mmttttm
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks, 2015. URL https://researchh httpss//reearrc...ggllbbooooooo wtt.om/////////1 sm- snceptionisnn-oonn--oo -oing-deepr-- inteper-into-euraaae ntml.
Alexander Mordvintsev, Nicola Pezzotti, Ludwig Schubert, and Chris Olah. Differentiable image parameterizzaions. Distill, 2018. doi: 10.23915/disiil.000012.
Bric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know?, 2018. URL httpp:://arixt http:///aaxvvooo//////// org/abs/ 1810. 09136..
Thu Nguyen-Phuoe Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. HoloGAN: Unsupervised learning of 3d representations from natural images. ICCV, 2019.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation. CVPR, 2022.
Xingang Pan, Ayush Tewari, Lingjie Liu, and Christian Theobalt. Gan2x: Non-lambertian inverse rendering of image gans. 3DV, 2022.
Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In NeurIPS Datasets and Benchmarks, 202
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. ICCV, 2021b.
Wei Ping, Kainan Peng, and Jitong Chen. ClariNet: Parallel wave generation in end-to-end text-to- speech. ICLR, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. ICML, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020.
Ravi Ramamoorthi and Pat Hanrahan. A sional-processing framework for inverse rendering. SIG- GRAPH, 2001.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ICMI., 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. URL http::///xxxxx.ooo/a//as/ 06125.
12

---

Published as a conference paper at ICLR 2023
Herbert E. Robbins. An Empirical Bayes Approach Statistics. Springer New York, 1992.
Geoffrey Roeder, Yuhuai Wu, and David Duvenaud. Sticking the landing: Simple, lower-variance http://xxvv.oogggbbb.aas 091944
gradient estimators for variational inference, 2017. URL
Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho. Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models, 2021a. URL https: //arxiv. org/ attaaa///111 .08226.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement, 2021b. URL httpp:://arxiii htttss///arriv.oo//ab////eee org/abs////... 07636.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. arXiv:2205. 1487, 2022.
Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin- Yi Cheng, and Marco Fumero. CLIP-Forge: Towards zero-shot text-to-shape generation. CVPR, 2022.
Christoph Schuhmann, Romain Beaumont, Cade W Gordon, Ross Wightman, mehdi cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Richard Vencu, Ludwig Schmidt, Robert Kaezmarezyk, and Jenia Jitsev. LAION-5b: An open scage-scale dataset for training next generation image-text models. NeurlPS Datasets and Benchmarks Track, 2022.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. GRAF: Generative radiance fields for 3d-aware image synthesis. NeurIPS, 2020.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. JCML, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. CoRR, abs/2010.02502, 2020. URL mttps://arxiit http:///axxii.oogaa//////// 02502.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurlPS, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021.
Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. NeRV: Neural reflectance and visibility fields for relighting and view synthesis. CVPR, 2021.
Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan. Edgar Tretschk, W Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, al. Advances in neural rendering. Computer Graphics Forum, 2022.
Aäron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. Parallel WaveNet: Fast high-fidelity speech synthesis. ICML, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. CVPR, 2022.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 2011.
13

---

Published as a conference paper at ICLR 2023
Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. CLIP-NeRF: Text-and- image driven manipulation of neural radiance fields. CVPR, 2022.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T Freeman, and Joshua Tenenbaum. Learning probabilistic latent space of object shapes via 3d generative-adversarial modeling. NeurfPS, 2016.
Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. ICCV, 2019.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. NeurIPS, 2020.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. arXiv:2206. 10789, 2022.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3D shape generation and completion through point-voxel diffusion. ICCV, 2021.
14

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 PSEUDOCODE FOR ANCESTRAL SAMPLING AND OUR SCORE DISTILLATION SAMPLING.
2_t = random. random.normal (img_saeep)))
for t in linspace (araoce((mman (tmax, tmin, nstep):
epshat_t di fifuusoon_modeleepptal if t tmin:
y, t) Score function evaluation.
eps = www.aoooooaammmm..m..ao.
ddpm_updatee2_tt, epshat_t, eps) 1 iteration, decreases noise level.
di diffusion_modell xhat(z_t, epshat_t, t_min) Tweedie's formula: denoise the last step.
return x
Figure 7: Pseudocode for ancestral sampling from DDPM where y the optional conditioning signal e.g. caption. Typically, tmax and tmin = 1/nstep. Timestep t monotonically decreases.
params = gemerator. inni(()
opt_state optimizer.. init (parama)
dif diffusion_model diffusion. load_ [aadd__doee(
for nstep iterations:
t = random. www.:unmfff........ 1.) alpha_t, sigma_t eps random.
generator (params, <other arguments?... Get an image observation.
z_t alpha_t t * x sigma_t eps # Diffuse observation.
epshat_t = @Fffussio__mddell_ epshhatt_tt,,, y, t) # Score function evaluation.
(t) * iensssooggrrdienttesttttt..... eps], x) params)
params, opt_state aptimizer.update(G. opt_state) Update params with optimizer. return params
Figure 8: Pseudocode for Score Distillation Sampling with an application-specific generator that defines a differentiable mapping from parameters to images. The gradient g is computed without backpropagating through the diffusion model's U-Net. We used the stopgradien operator to express the loss, but the parameter update can also be easily computed with an explicit VJP: g matmul (weight(t) (-epshat_t eps), grad(x, params)).
A.2 NERF DETAILS AND TRAINING HYPERPARAMETERS
Our model builds upon mip-NeRF 360 (Barron et al., 2022) (starting from the publicly available implementation 2022), which is an improved version of NeRF (Mildenhall al., 2020). The main modification this model makes to NeRF is in how 3D point information is passed to the NeRF MLP. In NeRF, each 3D input point is mapped a higher dimensional space using sinusoidal positional encoding function (Vaswani et al., 2017). In mip-NeRF, this is replaced by an integrated positional encoding that accounts for the "width" of the ray being rendered (based on its pixel footprint in the image plane) and the length of each interval [d;, de+1] sampled along the ray (Barron et al., 2021). This allows each interval along ray to be represented as a Gaussian distribution with mean μa and covariance matrix Σ that approximates the interval's 3D volume.
MiP-NeRR covariance annealing. As in mip-NeRF, each mean is the 3D coordinate of the center of the ray interval, but unlike mip-NeRF we do not covariance derived from camera geometry, but instead define each as:
Σ 1
Where Ay is scale parameter that is linearly annealed from large value small value during training. Representative settings are x 10 and 2 x 10-3 for the initial and final values of Ax, linearly annealed for the first 5k steps of optimization (out of 15k total). This "coarse fine"
15

---

Published as a conference paper at ICLR 2023
annealing of scale parameter has similar effect the annealing used by Park et al. (2021b) but uses integrated positional encoding instead of traditional positional encoding. The underlying sinusoidal positional encoding function uses frequencies 2 ..... ....,,2!1, where set L = 8.
MLP architecture changes, Our NeRF MLP consists of 5 ResNet blocks (He et al., 2016) with 128 hidden units, Swish/SiLU activation (Hendrycks & Gimpel, 2016), and layer normalization (Ba et al., 2016) between blocks. We use an exp activation to produce density T and sigmoid activation produce RGB albedo p.
Environment Map MLP. We use an additional MLP to create an environment map allowing the bounded NeRF MLP to be rendered on a conlextullly-relevat and optimized background. This MLP takes a positionally-encoded ray direction as input, and outputs an RGB color. We use positional encodings with frequencies 2 2"....,2" 2t, and an MLP with hidden layers of 64 units. The output of this MLP is passed through sigmoid activation to produce RGB values between 0 and 1.
Shading hyperparameters. For the first 1 steps of optimization we set the ambient light color la to 1 and the diffuse light color L, to O, which effectively disables diffuse shading. For the remaining steps we set La = [0.1, 0.1, 0.1] and l, [0.9, 0.9, 0.9] with probability 0.75, otherwise La 1, tp = O, i.e. we use diffuse shading 75% of the time. When shading is on (lp- 0), we choose textureless shading (p- = 1) with probability 0.5.
Spatial density bias. To aid in the early stages of optimization, we add small "blob" of density around the origin to the output of the MLP. This helps focus scene content at the center of the 3D coordinate space, rather than directly next to the sampled cameras. We use Gaussian PDF to parameterize the added density:
||ll|
Tinit (14) = Ar exp
exx
20
Representative settings are A = for the scale parameter and a, = 0.2 for the width parameter. This density is added to the output of the NeRF MLP.
Additional camera and light sampling details. Uniformly sampling camera elevation deam in angular space does not produce uniform samples over the surface of the sphere - the area around the pole is oversampled. We found this bias to be helpful in practice, so we sample Φeam from this biased distribution with probability 0.5, otherwise we sample from a true uniform-in -infaarm--n-acaa distribution on half-sphere. The sampled camera position is perturbed by a small uniform offset (((00.1, 0.1, 0.1) The "look-at" point is sampled from A A(0, O.2I) and the default "up" vector is perturbed by noise sampled from (0, 0.021). This noise acts an additional augmentation, and ensures wider diversity of viewpoints are seen during training.
We separately sample the direction and norm of the light position vector l. To sample the direction, we sample from N (Peam I) where Peam is the camera position (this ensures that the point light usually ends up on the same side of the object the camera). The norm ||l|| is sampled from ((.88, 1.5). while Peam| ~ M(1.0, 1.5).
Regularizer hyperparameters. We use the orientation loss proposed by Ref-NeRF (Verbin et al., 2022) to encourage normal vectors of the density field to face toward the camera when they are visible (so that the camera does not observe geometry that appears to face "backwards" when shaded). We place a stop-gradient on the rendering weights wi, which helps prevent unintended local minima where the generated object shrinks or disappears:
Lorient
Stopp-grrd
m0, n. t))
(10)
where is the direction of the ray (the viewing direction). We also apply small regularization to the accumulated alpha value (opacity) along each ray: Lopacity = V(E, w;)2 12 0.01. This discour- ages optimization from unnecessarily filling in empty space, and improves foreground/backgrruund separation.
For orientation loss Coriens we find reasonable weights to lie in [10-1, 10 3]. If orientation loss is too high, surfaces become oversmoothed. In most experiments, we set the weight 10-2, This weight
16

---

Published as a conference paper at ICLR 2023
is annealed in starting from 10-4 over the first 5k (out of 15k) steps. For accumulated alpha loss Lopacity, we find reasonable weights to lie in [10-8 x 10 10-3)
View-dependen prompting. We interpolate between front/side/back view prompt augmentations based on which quadrant contains the sampled azimuth 6eam- We experimented with different ways of interpolating the text embeddings, but found that simply taking the text embedding closest to the sampled azimuth worked well.
Optimizer. We use Distributed Shampoo (Anil al., 2020) with 81 0.9, B2 0.9, exponent override = 2, block size = 128, graft type = SQRT N, c = 10-6 and a linear warmup of learning rate over 3000 steps from 10-3 to 10-4 followed by cosine decay down 10-6. We found this long warmup period to be helpful for improving the coherence of generated geometry.
A.3 EXPERIMENTAL SETUP
Our computation of R- Preccissin differs slightly from baselines, As mentioned. CLIP- text-to-3D systems are prone to overfitting the evaluation CLIP R-Precision metric since the model used for training is similar to the evaluation model. To minimize this problem, Dream Fields (Jain et al., 2022) and ELih-Mesh (Khalid et al., 2022) evaluate renderings at single held out view at 45 elevation, higher than seen during training (maximum 30%). DreamFusion evaluates at 30 since is not prone to this issue, but averages the metric over multiple azimuths to reduce variance. In our main results in Table 1, we evaluate all captions with 2 generation seeds unless otherwise noted.
A.4 DERIVING THE SCORE DISTILLATION SAMPLING LOSS AND GRADIENTS
The score distillation sampling loss LSDS presented in Eqn. 4 was inspired by work on probability density distillation (Huang et al., 2019; Ping et al., 2019; van den Oord et al., 2018). We use this loss to find modes of the score functions that are present across all noise levels in the diffusion process. Here we show how the gradient of this loss leads to the same update as optimizing the training loss LDar, but without the term corresponding to the Jacobian of the diffusion U-Net. First, we consider gradients with respect to a single KI. term:
KL(g(Zx(x = ((0)) (2: = E, [loga(z)/= = g(0)) log P (ze (2e)))
(11)
VoKL(g(Zr|x = (00))llppzzllv) = E. V Volog q(z: x = s(0)) Vo log log
(12)
(A)
(B)
The second term (B) can be related to by the chain rule, relying on s (2rly))
RM
Vz,
log logpo( (o(()))):
Ve log (z.lv() (z(ll)y) Oz: 00 asss(Zely)
Ōx 0t 20 (z)
8x
(13)
00 or
00
The first term (A) is the gradient of the entropy of the forward process with respect to the mean parameter, holding the variance fixed. Because of the fixed variance, the entropy constant for given t and the total gradient of (A) is O. However, we can still write out its gradient in terms of the "score function"' (the gradient of the log probability with respect to parameters) and path derivative (the gradient of the log probability with respect to the sample):
Ve log ((z)(x)= Olog q(z (x)
i log qZz x) Oz:
dx
at
Ox at
Ox
Oz.
8x
Du 00
=
0z
- c
00
= 0. (14)
parameter score
path derivative
Sticking-ther Ltannnngg (Roeder et al., 2017) shows that keeping the path derivative gradient while discarding the score function gradient can lead to reduced variance as the path derivative term can be correlated with other terms in the loss. Here, the other term (B) corresponds to a prediction of which is certainly correlated with the RHS of equation 14. Putting these together, we can use seiicknng-te-landdeee -sstle gradient of our loss by thinking E as a control variate for é:
V oo Lsss = 12,,2
(1)
Tt (oKL
x g(0)))lpo(z=lu))
(15)
0x
= E,,
(o(t) (?(2.ly) c)
(16)
17

---

Published as a conference paper at ICLR 2023
In practice, we find that including in the gradient leads to variancriance gradients that speed up optimization and can produce better final results.
In related work, Graikos et al. (2022) also sample diffusion models by optimization, thereby allowing parameterized samples. Their divergence KL(h(x) ps (x(x)) reduces to the loss E., éØ ly; 1) log c(x, y). The squared error requires costly backpropagation through the diffusion model éø. unlike SDS. DDPM-PnP also uses auxiliary classifier c. while use CFG.
few other works have updates resembling Score Distillation Sampling for different applications. Gradients of the entropy of an implicit model have been estimated with an amortized score model at single noise level (Lim et al., 2020), though that work does not use our control variate based subtracting the noise from é. SDS could also optimization by using multiple noise levels. GAN-like amortized samplers can be learned by minimizing the Stein discrepancy (Hu et al., 2018; Grathwohl et al., 2020), where the optimal critic resembles the difference of scores in loss (Eqn. 3).
A.5 IMPACT OF SEED AND GUIDANCE WEIGHT
We find that large guidance weights are important for learning high-quality 3D models. Unlike image synthesis models that use guidance weights w [5, 30), DreamFusion uses weights up w 100, and works at even larger guidance scales without severe artifacts. This may be due to the constrained nature of the optimization problem: colors output by our MLP are bounded to [0, 1] by a sigmoid nonlinearity, whereas image samplers need clipping.
We also find that our method does not yield large amounts diversity across random seeds. This is likely due to the mode-seeking properties of LSDS combined with the fact that at high noise levels, the smoothed densities may not have many distinct modes. Understanding the interplay between guidance strength, diversity, and loss functions remains important open direction for future research.
A.6 EXTENDED QUALITATIVE COMPARISON WITH PRIOR WORK
In Figure 11, we compare DrcamFusion to Dream Fields on several prompts from the object-centric MS-COCO validation set.
18

---

Published as a conference paper at ICLR 2023
Guidance weight
50
75
100
250
Seed =
Seed = 5
Seed = 3
Seed =
Figure 9: A 2D sweep over guidance weights and random seeds for two different prompts ("a zoomed DSLR photo of robot couple fine dining" and "a DSLR photo of chimpanzee dressed like Henry VIII king of England")..
blue jug garden filled with mud
A pile of crab is seasoned and well cooked
there very colorful kite that is in the air
A couple boen built in suburbam backyards after
The rotted out bed of truck left in the woods
CLIP (ViT B/16)
17
SDS (Imagen)
Figure 10: Comparison of the CLIP loss from Dream Fields (top) and our SDS loss. For fair all of our proposed methods including view-
comparison of the loss functions in isolation, dependent prompts, shading, optimizing untextured renderings, and regularizers, with the same 3D NeRF representation. Qualitatively, 3D scenes generated with SDS (ours) are much more coherent than scenes generated with CLIP loss.
19

---

Published as a conference paper at ICLR 2023
Figure 11: Comparison between DreamFusion and Dream Fields (Jain et al., 2022) on object- centric MS-COCO prompts. Odd rows are results from DreamFusion, and even rows are results from Dream Fields. DreamFusion exhibits increased global coherence and quality.
20


Reviews:
[{"review_id": "zKFTuYm9eZz", "reviewer": "Reviewer_44Jy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel approach for text-to-3D generation based on pre-trained text-conditioned diffusion models (Imagen in this paper). More specifically, the authors learn a neural radiance field by rendering at random views and use the pretrained diffusion model as the objective function to back-prop gradients to the radiance field. The authors also analyzed the difficulties of directly applying the denoising objective as the loss function and proposed a \"simpler\" form related to \"probability density distillation.\"", "summary_of_the_review": "This submission creates a novel way of generating 3D content without needing 3D or 2D image data. Compared to similar existing work (e.g., dreamfield),  the usage of the diffusion model makes it possible to generate highly realistic samples. Also, the proposed fix of using \"probability density distillation\" loss for learning with a pretrained DM is technically sound, and also has great potential to be applied to other distillation tasks. Therefore, despite some possible weaknesses listed above, it should be a solid to be appear at ICLR.", "strength_and_weaknesses": "Strength: The method is pretty novel and effective. The analysis of the instability of the loss function is technically sound, and the solution makes the model effective in optimizing the high-quality 3D content based on the text prompt.\n\nWeakness:\n(1) the proposed text-to-3D generation is a bit tricky to evaluate. Since the model is per-scene optimization with a pretrained DM and text prompt, both the quality and text input will affect the synthesized output. Also, as it is difficult to have real 3D ground truth to compare the generation quality (in comparison to 2D generation) \n(2) while the generated results correspond to the text input well and sometimes generalize to very novel scenes, I think it is only thanks to the original power 2D diffusion model. The generated 3D, however, seems to have severe mode collapse issues compared to 2D models (blurry, dark, and unnatural high-contrast colors). Is it because the objective function encourages that? What might be the possible way to avoid such a generation further? How can we generate multiple 3D scenes with the same text?\n(3) the model seems to require a lot of computational resources to optimize for a single scene. Will the same approach be generalized to synthesize the radiance field without optimization directly?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WbHmSpr8gu", "reviewer": "Reviewer_UY2y", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a Text-to-3D algorithm by using a pre-trained 2D text-to-image diffusion model. \nIt modifies the denoising diffusion loss for better supervision signal. \nIt also introduces a probability explanation of the loss called density distillation.\nUsing this loss in a DeepDream-like procedure, the method optimizes a randomly-initialized NeRF via gradient descent such that its 2D renderings from random angles achieve a low loss. ", "summary_of_the_review": "The paper is well-written. The algorithm is novel. See more comments in Strength And Weaknesses.", "strength_and_weaknesses": "Strength:\n\n+ The method allows flexible control of the resulting 3D model of the given text, e.g. it can be viewed from any angle, relit by arbitrary illumination, or composited into any 3D environment.  \n\n+ The approach requires no 3D training data and no modifications to the image diffusion model.\n\n+ The 3D rendering process is well-considered such that many local minimums can be avoided.\n\n+ The paper is well written, and one can quickly grasp the main idea and many technical details.\n\n+ The paper presents a novel way to sample parameter space instead of pixel space.\n\n\nWeakness:\n\n- On P4, the paper says that it minimizes the diffusion training loss: $L_{Diff}$. As shown in Eq. 1, it is $\\epsilon_\\phi$ that is used to compare with $\\epsilon$. However, in Eq. 2, the variable becomes $\\hat{\\epsilon}_\\phi$. Please explain the discrepancy here.\n\n- On P6, above Eq. 5, the paper says that it performs alpha-composition from the back of the ray to the camera. This is quite unusual, as in NeRF we usually go the opposite way, i.e. from near camera to far, such that objects near the camera are first added, and the further objects can be occluded. \n\n- There seems no specular term in Eq. 7. However, some specular effects can be seen in Figure 4, e.g. on the armor. I wonder where those effects come from.\n\n- On P6, the description of the second MLP for scene generation is too brief. How important is it for the final synthesis performance? Why does this MLP only take ray direction as input? Why does it only produce RGB color instead of 3D geometry as well?\n\n- Some important results are missing from the ablation study: 1) instead of adding components one by one, the study should consider removing each component separately, such that it is easier to understand which component is the most relevant; 2) the comparison between the new SDS loss and the previous CLIP loss in DreamField is missing.\n\n- Minor issue: In Table 1, it is better to add a dash in the CLIP names, for example, CLIP-B/32.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Es_vpo2lo2", "reviewer": "Reviewer_nnv7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper uses a NeRF backbone with CLIP-based text guidance to synthesize 3D objects. Although most components are borrowed from its prior work, Dream Fields, the paper focuses on the sampling algorithm, and shows superior performance on both qualitative and quantitative results.", "summary_of_the_review": "This paper is a clear acceptance, but there are some minor issues in the writing. And I see the TPUs are used in this paper, despite unlikely, but it would be greatly appreciated if the authors can make the codes publicly available soon, as an effort towards an open and welcoming research community.", "strength_and_weaknesses": "Strengths:\n1. The visual quality is obviously phenomenal in generative modeling research, although the formulation is not generative modeling. Under the umbrella of creative content synthesis, it becomes interesting whether do we really need a real-time generative model, or simply a faster (1.5 hours in this work is still too long) inversion method.\n\nWeaknesses:\n1. This is personal, but I do not like the writing strategy of the paper. Many of the core concepts in this paper are borrowed from Dream Fields, but the abstract and introduction are written in a way as if the concept is first proposed in this paper. And the differentiation between DreamFusion and Dream Fields is written in obscure sentences. I would suggest the authors take a deeper look into how to present the differences.\n2. In Figure 5., the re-implementation version of Dream Fields is visually much worse than the original version, despite the quantitative scores seeming to be fine. I would suggest the authors adding more visual comparisons between these two methods in the Appendix.\n\nQuestions:\n1. Current 3D generative models mostly rely on a canonical coordinate to align objects, and therefore fail on categories without aligned canonical coordinates (such as trees or scenes). Does Dream Fusion have a similar limitation?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CZ2bttjxm-", "reviewer": "Reviewer_Ccog", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel method to generate 3D assets from text descriptions.\nThe proposed method utilizes a pretrained diffusion model to generate a denoised 2D image from text and noisy input.\nThe noisy input is rendered with a NerF model and known noise. The NerF model is trained by minimizing the noise residual between known noise and predicted noise.\nThe experimental results and the galleries in the supplementary material show the proposed method works quite well.\n", "summary_of_the_review": "The paper presents a novel solution for 3D assets generation from text, and the results show the proposed method work very well.\nSome descriptions can be improved to make readers a smoother understanding.", "strength_and_weaknesses": "++ The proposed method should be useful for 3D asset generation, and the results are very promising.\n\n++ The loss from Score Distillation Sampling opens a new way to use the 2D diffusion model as prior for NerF model optimization.\n\n++ The diffusion model is pretrained and can be directly used in the text-to-3D task. \n\nThere are also several concerns. \n-- In Fig. 1, some images are without a symbol indicating the prefix, and there is no \"wide angle zoomed out...\"\n\n-- CLIP model is not explained and cited in Sec. 1 while Sec. 4 shows the reference of CLIP.\n\n-- The ground truth noise $\\epsilon$ in Eq. 1 is not defined or explained before use. What does $\\epsilon$ look like?\n\n-- It is better to define the KL operation in the main text near Eq. 4, letting the reader have a smoother understanding.\n\n-- Why use Imagen in the proposed method?  How is the performance if using other diffusion models?\n \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%