Noise-free Score Distillation
Authors: Oren Katzir
Or Patashnik
Daniel Cohen-Or
Dani Lischinski
Venue: ICLR 2024 poster (2024)
Keywords: score distillation sampling
generative models
text to image
Primary area: generative models

Abstract: Score Distillation Sampling (SDS) has emerged as the de facto approach for text-to-content generation in non-image domains. In this paper, we reexamine the SDS process and introduce a straightforward interpretation that demystifies the necessity for large Classifier-Free Guidance (CFG) scales, rooted in the distillation of an undesired noise term. Building upon our interpretation, we propose a novel Noise-Free Score Distillation (NFSD) process, which requires minimal modifications to the original SDS framework. Through this streamlined design, we achieve more effective distillation of pre-trained text-to-image diffusion models while using a nominal CFG scale. This strategic choice allows us to prevent the over-smoothing of results, ensuring that the generated data is both realistic and complies with the desired prompt. To demonstrate the efficacy of NFSD, we provide qualitative examples that compare NFSD and SDS, as well as several other methods.

Published as a conference paper at ICLR 2024
NOISE-FREE SCORE DISTILLATION
Oren Katzir
Or Patashnik
Daniel Cohen-Or
Dani Lischinski
Tel-Aviv University
2 The Hebrew University of Jerusalem
ABSTRACT
Score Distillation Sampling (SDS) has emerged as the de facto approach for text- to-content generation in non-image domains. In this paper, we reexamine the SDS process and introduce a straightforward interpretation that demystifies the neces- sity for large Classifier-Fre Free Guidance (CFG) scales, rooted the distillation of an undesired noise term. Building upon our interpretation, we propose novel Noise- Free Score Distillation (NFSD) process, which requires minimal modifications to the original SDS framework. Through this streamlined design, we achieve more effective distillation of pre-trained text-to-image diffusion models while using a nominal CFG scale. This strategic choice allows us prevent the over-smoothing of results, ensuring that the generated data is both realistic and complies with the desired prompt. To demonstrate the efficacy of NFSD, we provide qualitative ex- amples that compare NFSD and SDS, as well as several other methods.
INTRODUCTION
Image synthesis has recently witnessed significant progress in terms of image quality and diver- sity (Yu et al. 2022 Ding et al. 2021 Gafni et al. 2022 Chang et al., 2022 Kang et al. 2023
Chang et al. 2023) Specifically, text-to-image models are rapidly improving. with diffusion-bar based
methods leading the way (Sohl-Dickstein et al., 2015: Ho et al. 2020: Dhariwal & Nichol 2021
Rombach et al., 2022 Balaji et al.) 2022] Saharia et al., 2022). Seeking to project the great power of such fiffusion-bbsed text-to-image models to other domains beyond images, Score Distillation
Sampling (SDS) was introduced. In their seminal work, DreamFusion, Poole et al. (2022) introduce the SDS loss which utilizes the strong prior learned by text-to-image diffusion model to optimize
a NeRF (Mildenhall et al. 2020)) based on a single text prompt. Other works have shown that this mechanism can also be used optimize other representations, such meshes (Chen et al. 2023) texture maps (Metzer et al. 2023). and SVG (Jain al.
2022 Tsalicoglou et al. 2023). fonts (lluz et al. 2023
Tanveer et al.
2023).
Despite the widespread adoption of the SDS loss in various domains and representations, there is still a gap in visual quality between images generated by the standard denoising diffusion process (ancestral sampling) and those resulting from an SDS-basee optimization process. Specifically, as noted by previous works (Poole al. 2022 Wang al. 2023b Zhu & Zhuang 2023). SDS tends to produce over-smoothed and over-saturated results, exhibiting limited ability to generate fine details, trait where modern text-to-image models typically excel. Furthermore, the SDS loss remains intriguing, as it is still not fully understood.
In this paper, inspired by SDS (Poole et al. 2022), , we present general framework that allows us- ing a pretrained diffuusin model to optimize a differentiable image renderer. Treating the diffusion model as a score function (Song et al., 2020), we propose a formulation that decomposes the score into three intuitively interpretable components: alignment with the condition, domain correction, and denoising. Based on insights gained by viewing the score function light our new decompo- sition, we introduce new Noise-Free Score Distillation (NFSD) loss, and show that it outperforms SDS without incurring any additional computational costs.
To demonstrate the general nature of our framework, we show that novel formulation sup- ports and provides a more concise and straightforward explanation for recent methods, such as VSD (Wang et al. 2023b and DDS (Hertz et al. 2023), which have shown improvements over SDS.
Denotes Equal Contribution

---

Published as a conference paper at ICLR 2024
Results obtained with our Noise Free Score Distillation (NFSD). Top: two learnt NeRFs (movies of these and many other examples are included in the supplementary materials). Bottom: gallery of images optimized with NFSD.
Figure 1:
We validate our formulation and approach by utilizing Stable Diffusion (Rombach et al. 2022 as our score function with focus on images and NeRFs as our representations. A few example re- sults are showcased in Figure [ Through careful design, our Noise-Free Score Distillation (NFSD) addresses some of the issues present in SDS and leads to improved visual results.
2 BACKGROUND
In this section, we provide the necessary background regarding diffusion models and the SDS loss (Poole et al. 2022) that enables text-to-3D) generation by optimizing the parameters differen- tiable image generation function.
Diffusion models Diffusion models (Sohl-Dickstein et al.) 2015 Ho et al., 2020) are family of generative models that are trained to gradually transform Gaussian noise into samples from target distribution Pdata- Starting from an initial noise model takes as input a noisy sample Z,, and predicts cleaner sample 2t-1, until finally obtaining
(0, I), at each diffusion timestep I, the
Zo Pdata Thus, such models effectively learn the transitions p( p(z,-1/z)) 1z.).
Commonly, diffusion models are parameterized by U-net (z, t) Ho et al. 2020), which predicts the noise that was used produce 2, from Zo. rather than predicting x 2 directly. This known as e-prediction. Previous works (Ho et al. [2020] Song et al., 2020) have also observed that <g(2,, 1) is proportional to the predicted score function (Hyvärinen & Dayan 2005) of the smoothed denisty Vz, log p. (z.), where Pt is the marginal distribution of the samples noised to time l. The score function intuitively, taking steps in the direction of the score function gradually moves the sample towards the data distribution.
vector field that points towards higher density of data at given noise level. Thus,
In this work, we focus on diffusion models that strive to generate samples aligned with given condition y (e.g., class, text prompt). To this end, the diffusion process is conditioned on y. This is typically achieved via classifier-free guidance (CFG) (Ho & Salimans, (022). where the condi- tioned prediction e c,(2;; y. I) of the noise extrapolated away from the unconditioned prediction (2(2; Ø,, !) by amount controlled by a scalar E IR:
E (2 es (z;; Ø0, 1)
(Ea (Z:: v, c (2, 20,
2

---

Published as a conference paper at ICLR 2024
where @ indicates a null condition (unconditioned). CFG modifies the score function to steer the process towards regions with a higher ratio of conditional density to the unconditional one. However, it has been observed that CFG trades sample fidelity for diversity (Ho & &allimans 2022).
Score Distillation Sampling (SDS) Over the last two years, text-to-image diffusion meth- ods (Rombach et al.) 2022] Saharia et al., 2022. Ramesh et al., 2022 Podell et al. (023) have achieved unprecedented image generation results by incorporating textual encoder outputs as a con- dition to the diffusion model. These powerful models are trained on billions of text-image pairs, and such extensive data is currently not available for other domains. The recent introduction of Score Distillation Sampling (SDS) (Poole et al., 2022 Wang et al.) 2023a) enables leveraging the priors of pre-trained text-to-image models to facilitate text-conditioned generation in other domains, particularly 3D content generation.
Specifically, given a pretrained diffusion model cg. SDS optimizes a set of parameters 0 of a differ- entiable parametric image generator y, using the gradient of the loss Lsns with respect to 0:
dx
VoLSDS = w(t) (z)((x
(2)
a0
where = =00) is an image rendered by 0, Ze (x) is obtained by adding a Gaussian noise to corresponding to the t-th timestep of the diffusion process, and y is a condition to the diffusion model. In practice, at every optimization iteration, different values of t and Gaussian noise randomly drawn. The parameters l are then optimized by computing the gradient of CSDS with respect and backpropagating this gradient through the differentiable parametric function g.
Poole et al. (2022) formally show that LSDS minimizes the KL divergence between a family of Gaussian distributions around x and the distributions p(zz, y.t) learned by the pretrained diffusion model. Intuitively, Equation _ can be interpreted as follows: since x = g(0) is clean rendered image, Gaussian noise is first added to it in order to approximately project it the manifold of noisy images corresponding to timestep f. Next, the score e (2)(;: t) provides the direction in which this noised version of x should be moved towards a denser region in the distribution of real images (noised to timestep t and aligned with the condition y). Finally, before the resulting direction can be used to optimize 0, the initially added noise e is subtracted. We interpret this last step as an attempt to adapt the direction back to the domain of clean rendered images.
While SDS provides an elegant mechanism for leveraging pretrained text-to-image models, SDS- generated results often suffer from oversaturation and lack of fine realistic details. These issues were, in part, attributed to the use of a high CFG value (Wang et al. (023b). which |Poole et al. (022) empirically found to be necessary to obtain their results. Several derivative approaches have
emerged to address these challenges (Metzer et al., 2022 Lin et al. 2023 Chen et al. 2023 Wang
al. 20236 Huang et al. 2023).
One effective approach for improving the generation quality is time annealing, which gradually reduces the diffusion timesteps / drawn by the optimization process, as progresses (Lin et al,
2023 Zhu & Zhuang, 2023, Wang et al. (023bb [Huang et al., (023). Recently, VSD (Wang et al. 2023b> and HiFA (Zhu & Zhuang] 2023) reformulated the distillation loss. HiFA uses a denoised image version instead of the noise prediction. while VSD offers variational approach, matching the prediction of noisy real images with that of the noisy rendered images via an additional fine-tuned diffusion model. In image editing, DDS (Hertz al. 2023) observed artifacts when applying SDS to edit real images, which was attributed to bias in SDS. To mitigate this bias, DDS employs a subtraction of two SDS terms.
In the next section we propose our novel interpretation of SDS via decomposition of the predicted score function into three interpretable components. The insights gained from this decomposition lead us to propose a simple yet effective improvement to SDS, which we call Noise-Free Score Distillation (NFSD), in Section 4 Furthermore, this decomposition enables a simple and unified interpretation of the recent progress in SDS, as discussed in Section
3 SCORE DECOMPOSITION
As discussed earlier, the noise predicted by a trained diffusion model aims to be proportional to the score function Vz, log pe (Z.), where Pt is the marginal distribution of the samples noised to time t.
3

---

Published as a conference paper at ICLR 2024
t= 100
( = 200
1 300
t 500
t= 700
i 1000
Figure 2: Visualization of dc. The images in the left column are generated by Stable Diffusion (SD version 2. 1-base) with the prompts "A photo of horse in a meadow" and "A statue of Buddha". The other columns visualize &c for added noise e with magnitude corresponding to different diffusion timesteps t. As can be seen, òc is fairly clean and concentrated around the main object in the image. (The visualization is done by decoding each & using the VAE decoder of SD, please refer to Appendix A.T for more details).
In order to gain a better understanding of SDS, it is helpful examine the decomposition of the score direction into several intuitively interpretable components.
First, consider the difference &c < y, t) Co 00,1) in Equation While ca (ze; y, t) ideally points towards local maximum in the probability density of noisy real images conditioned on y, 6 (2(2; Ø, t) points towards a denser region in the distribution of unconditioned noisy images. Thus, the difference & between the two predictions may be thought of as the direction that steers the generated image towards alignment with the condition y, and we henceforth refer to it as the condition direction.
The condition direction & is empirically observed to be uncorrelated with the added noise e and hav- ing its significant magnitudes around the condition-specifie image regions. As demonstrated in Fig- ure 2. óc is consistently aligned with the condition for noise corresponding to different timesteps t of the diffusion process. This observation is consistent with the inspiration behind CFG (Ho & Salimans 2022 classifier-guidance of an implicit classifier Vz, log (y/Zz). Extending this ratio- nale, such classifier, trained noisy data 2,, should be invariant the additive noise e, and its gradients with respect to the input image z, should focus on details in z that are most relevant to y.
Rewriting Equation I using the condition direction &c defined above, we obtain:
g, eØ cσ(zz; 0,t) + S(E (zr;y, t) Ez( Ø, f)) es (zei 2, + sòc.
By the nature of its training, the unconditional term e (2;; Ø () is expected to predict the noise e that was added to an image noise to an ofi-o--diitribution (OOD) rendered image x = g(o). which is not sampled from Pdata- Thus, we can think of (2;2; 20,t) as combination of two components, (2,; (0,t) åp 6n. where ōp is the domain correction induced by the difference between the distributions of rendered and real images, while ON is the denoising direction, pointing towards cleaner image. Intuitively, we expect 6p to be correlated with the content of x(), while no such correlation is expected for dN.
Pdata produce z₁. However, in SDS, is obtained by adding
We are aware of any general way to explicitly separate e (Z,; Ø, () into these two compo- nents. Nevertheless, we attempt to isolate the two components for visualization purposes in Fig- 3 The idea examine the difference between two unconditional predictions es (xıx);;; Ø, t) and Eo (z, (XooD); Ø,t), where z, (XD)) and z, (XooD) are noised in-domain and out-of-domain im- ages, respectively, that depict the same content and are added the same noise c. Intuitively, while (Xoop): Ø, f) both removes noise (Ōx) and steers the sample towards the model's domain (õp), the prediction es (xpD) mostly just removes noise (Ōx), since the image already in-domain. Thus, in Figure 3 we use the latter to visualize Ex (column (c)) and the difference between the two predictions to visualize óp (column (d)), effectively assuming that ōg is shared between XID XOOD- As can be seen, dx indeed appears consist of noise uncorrelated with the image content, while åp large in areas where the distortion is most pronounced and adding óp to XooD effectively enhances the realism of the image (column (e)). More details about this process can be found in the appendix.
4

---

Published as a conference paper at ICLR 2024
(a)
(b) XOOD
(c) 6N
(d) 5D
(c) XOOD + 8p
Figure 3: Visualization of 6x and 6D (at t= 400). Columns (a) and (b) show a pair of in-domain (xm) and out-of-domain (XooD) images, both depicting the underlying content. We add the same noise to both images, and use the pre-trained diffusion model to predict the score. Intuitively, the noised XID image requires no domain correction, and thus the predicted score consists of only 5x. shown in (c). Subtracting Ey from the prediction for the noised XOOD image gives us the domain correction dp, shown (d). Indeed, adding dD to XOOD produces more realistic image (e).
f=1 t= 100
t = 200
= 300
500
f 700
f 1000
Figure 4: Visualization of 6N e. Top row: noise corresponding different diffusion timesteps t is added to an in-domain image of a horse (as indicated below each column). Bottom row: the residual (z, Ø, between the network prediction and the actual noise. Since the original image is in-domain (generated by SD), âp s 0, and therefore, c (2,; 23, () R 6g- For visualization purposes, the residual is decoded and clamped between and 1. Although we do not expect the residual ON to be correlated with the image, it may be seen that some correlation in fact exists, and furthermore, the residual becomes progressively noisier at smaller timesteps
To summarize so far, using the components discussed above, we can rewrite the CFG score as:
E (2:; y, 1) 6p + ON - sōc-
(4)
Poole et al. (2022) define the SDS loss using the difference between the CFG score and the noise e that was added the rendered image x to produce 2,, i.c.,
dx
8x
VoLSDS = w(t) (z.: t)
= wt) + EN sōc - 00
(5)
00
Note that while both ōp and &e are needed to steer the rendered image towards an in-domain image aligned with the condition y. the residual 6n e is generally non-zero and noisy, and this issue becomes increasingly pronounced when smaller time steps, responsible for the formation of fine details, are employed, as visualized in Figure This residual may explain, in part, the lower quality images generated using SDS, compared ancestral sampling: at each optimization step, the opti- mized parameters are guided at random direction depending on ôN - E, resulting in an averaging effect. While higher level semantics lower diffusion times tend to be over-smoothed. Previous works (Hertz et al. 2023 Wang et al. 2023b) have also observed that the subtraction of indeed leads to blurry results.
roughly less affected, fine and medium level details, from
Importantly, our decomposition in Equation 5 explains the need for using large CFG coefficient in SDS (e.g., 100), as this enables the image-correlated sŌc term dominate the loss, making
5

---

Published as a conference paper at ICLR 2024
8= 7.5
15
30
100
Figure 5: The impact of CFG on ancestral sampling. We generate all images using the same prompt "A photo of horse in meadow" with the same seed and different values of the CFG parameter As can be seen, large values of lead to over-saturated and less realistic results.
the noisy residual 8x c relatively negligible. However, high CFG coefficients are known to yield less realistic results as demonstrated in Figure 5 typically leading to over-saturated images and NeRFs. Although high CFG coefficients have also been held responsible for lack of result diversity, we demonstrate in Figure 11 that this assertion is not accurate.
NOISE FREE SCORE DISTILLATION
As discussed above, ideally only the soe and the 6D terms should be used to guide the optimization of the parameters O. While &e is simply the difference between the conditioned and the null-conditioned predictions, 6D is more challenging to separate from óx. as they are both part of the predicted noise (2(2;; t)..
To extract op. we distinguish between different stages in the backward (denoising) diffusion process. First, note that the noise variance, i.e., the magnitude of the noise to be removed, is monotonically decreased in the backward process. Thus, for sufficiently small timestep values t, 6N becomes small enough to neglected, and the score es (z;; Ø0, 1) = ON + &p approximately dp. Specifically, we apply this approximation for 1. < 200.
As for the larger timestep values, 2 200, we propose to approximate dp by the difference Eg(Ze; t) E (Zi Poeg, t), where Pneg "unrealistic, blurry, low quality, out of focus, ugly, low contrast, dull, dark, low-resolution, gloomy". Here, we are making the assumption that 8c=pus = cdp and thus c (z(; 0,t) cs (2(; Pusg, t) -Cpppa dp.
To conclude, we approximate dp by
6p = (z) t, (z) B, Ea <z Pneg t), otherwise,
if < 200
(6)
and the resulting & and ån to define an alternative. noise-free score distillation loss LNTSD- whose gradients are used to optimize the parameters 0, instead of VoLsps:
0x
VoLNESD = wt) (6p + sōc)
00
In Figure Figure we show that these seemingly small changes in the definition of the loss lead to noticeable improvement in the quality of generated images, well as NeRFs. Note, that while in the results of SDS we use s = 100, in our results we use the commonly used value of s = 7.5. The reason for this is that by taking the measures described above to approximately eliminate the EN component, it is no longer necessary to resort to a large value to make the de term dominant.
5 DISCUSSION
Our score decomposition formulation can be used to explain previous works that were proposed to improve the SDS loss. This demonstrates the wide scope and applicability of our formulation.
Hertz al. 2023) propose adaptation of the SDS loss for image editing task. Specifi- cally, instead of randomly initializing the optimization process as in SDS, it is initialized with the (in-domain) input image. DDS optimizes the input image according the text condition, while pre- serving image attributes that are irrelevant to the edit task guided by the input prompt. The gradients used by DDS are defined by VoLons Vo (LL (x), Vo (2. (2),
DDS.
(8)

---

Published as a conference paper at ICLR 2024
Figure 6: NeRFs optimized with NFSD.
where (x). I denote the noisy original imput image and its corresponding prompt, respectively. Here y denotes the prompt that describes the edit, and x, x are noised with the same noise e. Incor- porating our score decomposition into Equation yields
VoLDDs (ō) N& sóc,r e) (6 ON sÓc... - = w())s((cc.. dc..)) (9)
Our formulation helps to understand the high-quality results achieved by DDS: the residual compo- nent which makes the results in SDS over-smoothed and over-saturated is cancelled out. Moreover, since the optimization is initialized with an in-domain image, the óp component is not effectively needed and cancelled out. The remaining direction is the one relevant to the difference between the original prompt and the new one.
ProlificDreamer. Wang al. (2023b) tackle the generation task, and propose the VSD loss. which successfully alleviates the over-smoothed and over-saturated results obtained by SDS. In VSD, alongside the pretrained diffusion model Cg. another diffusion model CLORA trained during the optimization process. The CLoRA model is initialized with the weights of Co. and during the opti- mization process it is fine-tuned with rendered images x = g(0). Effectively, the rendered images during the optimization are out-of-domain for the original pretrained model distribution, but are in-domain for ÉLRAA Hence, the gradients the VSD loss are defined as
dx
VgLysD = wt) (<)(2((x); - CLORA (LoRA (2,(x);
(10)
00
where is another condition that is added <LARA and represents the camera viewpoint of the rendered image Viewed in terms of our score decomposition, since ELORA is fine-tuned on x, both dc and dp are approximately O, thus it simply predicts oN. Therefore, VeLVSD can be written
0x (()((ōp + sôc)
dx
VeLvsD = (() + 8N + sdc ON) 30
(11)
00
i... it approximates exactly the same terms as our NFSD. It should be noted that unlike our approach, VSD has a considerable computational overhead of fine-tuning the additional diffusion model during the optimization process.
EXPERIMENTS
We implement NFSD using the threestudio (Guo al. 2023 framework for text-based 3D gener- ation. Unless stated otherwise, all 3D models are optimized for 25, 000 iterations using Adam W optimizer (Loshchilov & Hutter 2017) with learning rate of 0.01. The initial rendering resolution of 64 x 64 is increased to 512 x 512 after 5, 000 iterations; at the same time we anneal the maximum diffusion time to 500 as proposed by Lin et al. (2023); Wang et al. (20235). The implicit volume is initialized according to the object-centric initialization (Lin et al. 2023 Wang et al. 2023b). We alternate the background between random solid-color and a learned neural environment map. The pre-trained text-to-image diffusion model for all experiments is Stable Diffusion b-ase (Rombachl ct al. 2022), a latent diffusion model with e-prediction.
3D generation. Figure G showcases several NeRFs optimized using NFSD. As can be the rendered images are sharp and contain highly intricate details. The prompts used and additional examples can be found in Appendices A.2 and A.7 respectively.
Comparison with SDS. We compare our NFSD with SDS under different parametric generators and different configurations. In each comparison, the same seed is used by both methods. Specifically,

---

Published as a conference paper at ICLR 2024
SDS 7.5
SDS 100
NFSD 7.5
"A metal lying Buddha"
"A cake made of icc"
"A photo of sunset, winter, river"
"A white dog sleeping"
"A fire with smoke"
Figure 7: 2D image generation with SDS and NFSD. We directly optimize the latent space of SD- 2.1-base (Rombach et al., 2022). Top row: SDS with CFG of 7.5 generates overly-smooth images that severely lack detail. The common solution to increase the CFG scale to 100 (middle row). The high CFG enables reasonable distillation of the score, but some artifacts remain (e.g., the dog's purple eyes, the frosting in the cake) and realistic details are still lacking (e.g., the fire). Bottom row: using the standard CFG of 7.5, our NFSD succeeds in distilling finer details, such the frosting on the cake or the fire example.
we set the same seed for the guidance process, i.e., the noise and the diffusion time are the same for both methods, every step of the optimization process.
2D image generation: Here, we directly optimize the latent code of Stable Diffusion, a 64 x 64 x 4 tensor. In the notations defined in Section 2 0 E R64x64x4 and g() 0. To initialize the op- timization process, 0 is sampled from Gaussian Distribution. We then use either LNESD LSDS as the only loss for 1, 000 iterations. As illustrated CFG scale (7.5) yields over-smoothed images, while using a high CFG scale (100) generates the main object but lacks background details and occasionally introduces aritfacts. In contrast, NFSI) optimization is able to produce more pleasing results in which the object is clear, the background detailed, and the image looks more realistic, even when using a CFG value of 7.5. For example, observe the fur of the dog which contains artifacts in the SDS-100 configuration, and looks more alistic with NESD-7.5, 55. In addition, observe the fire flames which feature fewer details and therefore look painted with SDS-100 100, and exhibit more detail and realism when using NFSD-7.5.
Figure 7 SDS optimization with a nominal
Text-to-NeRF synthesis: Figure g we use SDS and NESD to optimize NeRFs given text prompts. Here do not preform time annealing of the diffusion, which hinders bit the quality of the output model, but allows better emphasis on the differences between SDS and NESD. As can be seen, although SDS succeeds in generating plausible 3D objects, typically generates fewer fine details. Note for example the wings of the eagle and the mane of the horse. Note that, unlike images, the NeRF representation is inherently smooth; furthermore, additional losses, such shape density, are applied. These regularizations, combined with a high CFG value, enable SDS to produce plausible results, despite the unwanted noise distillation. In contrast, NFSD produces more detailed results with ordinary CFG values by attempting to eliminate the noise and explicitly approximate op-
Comparison with related methods. In Figure g we compare our method with recent approaches including DreamFusion (Poole et al. (002)) Magic3D (Lin et al. 2023), Latent-NeRF (Metzer et al.) 2022), Fantasia3D (Chen et al. 2023), and ProlificDreamer (Wang et al. 2023b), Following previ- ous works (Metzer et al.) 2022 Wang et al. 2023b Chen et al. 2023 Lin et al. 2023), for each of these methods we compare to results that were reported by the authors. As can be seen, our method achieves comparable better results, while being significantly simpler than most of these methods. For example, observe the roof the cottage, which is highly detailed in our method and Pro- lifieDreames but exhibits less detail in the other methods. Unlike ProlificDreamer which requires
8

---

Published as a conference paper at ICLR 2024
SDS 100
NFSD 7.5
Figure 8: Comparison of NeRFs optimized by SDS (top) and NFSD (bottom). NFSD allows better distillation: e.g., observe the wings of the eagle (leftmost), the angel's 's stomach (second from left), and the horse's s mane and tail (third from right). Furthermore, NFSD creates fewer sporadic features, as in the gargoyle (third from left). The prompts are reported in Appendix A.2
DreamFusion
Latent-NeRF
Magic3D
Fantasia3D
ProlifiDDreamer NFSD (ours)
Figure 9: Comparison of NFSD with different methods. Our NFSD is of high resolution and ex- hibits detailed features. In the top row we used the prompt "A plate piled high with chocolate chip cookies", and in the bottom one we used "A 3D model of an adorable cottage with thatched roof".
optimizing a diffusion model in parallel to the NeRF optimization, our method is simpler to imple- ment and the optimization process is much faster. Please note that the original methods differ in their implementation details, including the diffusion model that guides the optimization. Therefore, Appendix A.8, we provide comparison using threestudio (Guo al. 2023) for all method results.
CONCLUSION AND FUTURE WORK
In this paper, have revisited the SDS process and introduced novel interpretation, dissecting the score into three distinct components: the condition, the domain, and the denoising components. Through this novel perspective, proposed simple distillation process, which refer to as Noise-Free Score Distillation (NFSD). NFSD was developed with the explicit goal of preventing noise distillation during the optimization process. Notably, NFSD requires only minimal adjust- ments to the SDS framework, all while operating with nominal Classifier-Free Guidance (CFG) scale. Despite its simplicity, NFSD has shown promising potential in advancing the generation 3D objects, demonstrating notable improvements when compared to both SDS and existing approaches.
While NFSD enables better score distillation compared with SDS, two main drawbacks of the SDS process still exist, namely the well known Janus problem (multi-face) and low diversity. We be- lieve that the latter direct result the distillation process mechanism: the diffusion scores that guide the optimization climinated. While annealing the diffusion time as the optimization progresses is helpful, designing more principled noise scheduling might prove more effective, and lead improved diversity.
uncorrelated across successive iterations, even if noise is successfully
Additionally, we recognize the challenging nature of evaluating NeRFs generated from text, with a current absence of suitable metrics and benchmarks to assess NeRF quality and enable comprehen- sive comparisons across various methodologies. We believe that the development of such metrics is essential for advancing research in this domain and and would like investigate this in the future.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Rinon Gal and Yuval Alaluf for their feedback and helpful suggestions. This work was supported in part by the Israel Science Foundation (grants no. 2492/20, 361 1/21 and 3441/21), Len Blavatnik and the Blavatnik family foundation.
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Mila Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu. ediff-i: Text- to-image diffusion models with ensemble expert denoisers. arXiv preprint 1.X0322211.01324, 2022.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 11305-1131,, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- -ssan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishman. Muse: Text-to-image generation via masked generative transformers, 2023.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), October 2023.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. ArXiv, abs/2105.05233, 2021.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cogview: Mastering text-to-image generation via transformers. In Neural Information Processing Systems, 2021.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. ArXiv, abs/2203, 13131, 2022.
Yuan-Chen Guo, Ying-Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia- Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang. threestudio: A unified framework for 3d content generation. https://githhbt wwwp//heeeeettttiioooooooooooooo threestudio/ 2023.
Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. arXiv:2304.07090, 2023.
Delta denoising score.
arXiv preprint
Jonathan Ho and Tim Salimans. arXiw:2207.12595 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.
Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi, Zueng-Jun Zha, and Lei Zhang, Dream- time: An improved optimization strategy for text-to-3d content creation. arXiv preprint arXiv:2306. 12422, 2023.
Aapo Hyvärinen and Peter Dayan, Estimation of non-normalized statistical models by score match- ing. Journal of Machine Learning Research, 6(4), 2005.
Shir lluz, Yael Vinker, Amir Hertz, Daniel Berio, Daniel Cohen-Or, and Ariel Shamir. Word-as- image for semantic typography. arXiv preprint arXiv:2303.018/83 2023.
Ajay Jain, Amber Xie, and Pieter Abbeel. Vectorfusion: Text-to-svg by abstracting pixel-based diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 191 1-29 2023.
10

---

Published as a conference paper at ICLR 2024
Minguk Kang, Jun-Junn-a Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d con- tent creation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 300-309, 2023.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference Computer Vision, pp. 423-439. Springer, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or Or. Letent-nerf for shape-guided generation 3d shapes and textures. arXiv preprint NXi:22211.07600 2022.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhor, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01955, 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. arXiv preprint arXiv:2209. /4988, 2022.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. ArXiv, abs/2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205.11448) 2022.
Jascha Narain Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. ArXiv, 055/1503303585 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 20t.::0011.13456. 2020.
Maham Tanveer, Yizhi Wang, Ali Mahdavi-Amiri, and Hao Zhang. Ds-fusion: Artistic typography via discriminated and stylized diffusion. arXiv preprint arXiv:2303.09604, 2023.
Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. arXiv preprint arXiv:2304.12339, 2023.
Kevin Turner. Decoding latents to rgb without upscaling. httpss//dtiiusss.gggggggccccooot heeerss-o--gghiiiiioouuuppaa/nnp 2022.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jaco- bian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22619-12629, 2023a.
Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- Dreamer: High-fidelity and diverse text-to-3D generation with variational score distillation. arXiv preprint arXiv:2305. 162/3, 2023b.
11

---

Published as a conference paper at ICLR 2024
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasude- van, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, Benton C. Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. ArXiv, abs/2206. 10789, 2022.
Joseph Zhu and Peiye Zhuang. Hifa: High-fidelity text-to-3d with advanced diffusion guidance. arXiv preprint arXiv:2305. 2023.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 VISUALIZATION OF SCORE COMPONENTS
In Figures 2 and we visualize the score components, or manipulation of them. Stable Diffusion operates in the latent space of a pretrained autoencoder, and there- fore we use the decoder to obtain these images. At a first glance may be surpris- ing that the decoder can decode such latents, they are out of the distribution that was trained on. However, as shown in (Tunnrr 2022) and discussed by (Metzer) al. 2022). the RGB value at certain pixel, can be approximated by applying fixed linear transformation on the corresponding pixel of latent code. Hence, conclude that the latent space of Stable Diffusion behaves similarly to the RGB space, and decoding to RGB latent codes that are out distribution is still mean- ingful. We also note that when decoding the zero latent code, get the brown RGB image shown on the right. Therefore, this color our visualizations indicates a value of the latent code.
Additionally, in Figure B visualize dp and ÖN by generating pairs of X (in-domain) and XOoD -outiof-domain images. Additional components are visualized Figure [0 The two ID images were generated using DDIM sampling with the prompts "a pretty on sand" and "a cow field". Subsequently, generate the corresponding OOD images by applying a Gaussian filter to the cat image and a bilateral filter to the cow image.
200D
6M
6N 6p
60
+ 6p
MooD + by
KOOD + AN 60
+ 87.5
Figure 10: Extension of Figure in the main paper. Top row: we generate a pair of in-domain (xID) and out-of-domain (XooD) images, both depicting the underlying content. The noise is added to both images yielding z.D = (xi)) and 2.00 O0D z, (Xoo)) for ID and OOD respectively. We then calculate the predicted score of the pre-trained diffusion model, <g eø (XID); Ø, t) and coD = es (z (XOOD): Ø, t). Bottom row: we first calculate óp by subtracting the OOD score prediction from the ID score prediction, and then show that by adding this component to the OOD image, the quality of the resulting image, XOOD + 6p. is significantly improved. Conversely, when adding both ōp and dx to XOOD (resulting in XOOD + 8x + ōp), the noise is clearly apparent. When subtracting from XOOD + 6N 5p. in manner similar to the procedure in SDS, the resulting image (XooD ON + &p e) looks blurry. We also show gneg as defined in Equation 6 and demonstrate that its addition XoOD similar the result of adding the åp defined in the lefimost column.
A.2 PROMPTS USED IN THE PAPER
Here report the prompts used in several of the figures, in a left-to-right, top-to-bottom fashion.
In Figure "A huge Hedgchog", "a soldier iron decor pen holder", "a trunk up statue of an Elephant with Thailand Decoration, side view", "a photo of vase with sunflowers". "a brass statue of dragonfly" and "a rainbow colored wings spread parrot".
In Figure 8 "a cagle catching snake", "a golden statue of fairy angel with white wings", "a metal gargoyle statue with white wings, "a silver metal running horse on a table glass", "a huge Hystrix" and "a phoenix in golden cage".
13

---

Published as a conference paper at ICLR 2024
A.3 DIVERSITY OF RESULTS
We explore the diversity of generated results achieved by our method compared to SDS-100. We follow procedure similar the one outlined in Sec. by training a 2D-Jatent image, albeit without implementing any annealing of the diffusion time. As illustrated in Figure IT] both methods exhibit limited variation, portraying similar objects of comparable sizes positioned centrally within the im- age. Therefore, we infer that the limited diversity observed in SDS cannot be attributed solely to the use of high CFG scale.
SDS 100
NFSD 7.5
Figure 11:
Diversity in NFSD and SDS Results. We optimize 2D-latent images using the prompt "An astronaut riding horse" for both SDS (top) and NFSD (bottom) with each column originating from a different seed.
A.4 NEGATIVE PROMPTS
Here we show the relation between the term dd presented in the paper to the technique of using negative prompts. Specifically, the usage negative prompts, namely replacing the empty un- conditional prompt (""") with a selected negative prompt, is well-known to improve the results of ancestral-sampling (Liu et al., 2022). the context of SDS, it is also possible to use a negative prompt instead of the empty prompt, yielding
dx
VgLSns = (1) + a(ea(Zri v, 1) - c Pneg; 1)) E)
(12)
00
Since
(a(zii,,t ca(zi Poog t) = v, es (z1: t)) (Eo(Ze) Pacg t) - e (z))) (13)
then, Lsns can also written as
8x
Vo ney (1) + NN + s6c + 6p) c) 80
(14)
since 6p is defined as (Z; Pueg t) - c (z,; Ø, t)) in Equation prompt in SDS, mimics the addition of amplified ån term to the SDS loss. This further strengthens the importance of the term ân. and supports its definition with the negative prompt. As can be seen in Figure 13 the negative prompt (NP) usually adds details to both SDS and NFSD (e.g., the smoke). However, The results obtained by SDS-100+NP sometimes exhibit some artifacts such as the dark spots on the cake's frosting, and above the dog's eyes. NFSD-7.5+ obtains highly detailed results (e.g., the ice on the berries), but sometimes slightly deviates from the prompt (e.g., the cake does not look made of ice with NP).
6 Interestingly, using a negative
A.S ABLATION STUDIES
Time Threshold We empirically validate our decision to set t, 200 the sufficiently small time Equation for estimating es (z;; 1) 2 åp. We follow a similar procedure to the one outlined in Sec. 6 without time annealing. As depicted in Figure 12] time interval that is too small fails to regulate the generated image, resulting noticeable artifacts, while too large values i lead to overly smoothed result. Note, that in all the visualized examples 7.5, and therefore the results for t2 = 1000 look worse than those of SDS-100.
CFG scale of
14

---

Published as a conference paper at ICLR 2024
L. 0
100
1, = 200
t, = 300
La 400
4. = 500
L, 1000
Figure 12: Different time thresholds &.. For every t < tz. dD is estimated via (2,; Ø, t). The prompts used are: "An astronaut riding a horse" and "Panda snowboarding".
Ablating Sp better understand the role of ón in NFSD, we remove from our loss, keeping only the &c term. In the second row of Figure 13 we show results where we optimize 2D-latent image only with dc. As can be seen, optimizing with & results in an image that follows the prompt, but has many artifacts.
"A motal lying Buddha"
ice"
"A photo of winter, river"
"A white dog sleeping"
"A fire with smoke"
SDS-7.5
7.56c
SDS-100
SDS-100+NP
NFSD-7.5
NFSD-7.5-
Extension of Figure 7 the second column we present results obtained by using only the de component in NFSD. As shown, this results in many artifacts. Using the negative prompt adds details to both SDS-100 and NFSD-7.5 as shown in the fourth and rightmost columns for SDS and NESD, respectively. However, SDS-100+ND exhibits some artifacts (e.g., the cake frosting, the dog's eyes). NFSD-7.5+NF produces sharp and highly detailed results, but sometimes slightly deviates from the prompt (e.g., the cake).
Figure 13:
A.6 QUANTITATIVE EVALUATION
FID We quantitatively evaluate both our NFSD and SDS for the task of 2D-image generation using the COCO2014 caption dataset. We have sampled 5K captions and images from the validation dataset. The FID scores between the generated data of each method and the COCO dataset shown in Table As can be seen the table, NFSD achieves better FID compared to SDS- 100, confirming the better image quality achieved by our method.
15

---

Published as a conference paper at ICLR 2024
SDS-100
NFSD (ours)
FID
39.04
35.18
Table I: FID-5K results on COCO2014, NFSD achieves lower FID score compared to SDS.
User Study Following Magic3D (Lin et al. 2023) and ProlificDreamer (Wang et al. 2023b). we conducted user study comparing rendered NeRF images obtained by DreamFusion (Poole et al. 2022], Fantasia3D (Chen al. 2023), Magic3D (Lin et al. 2023). ProlificDreamer (Wang et al, used 15 prompts from previous works, and for each prompt, we displayed images corresponding to this prompt obtained by methods that provide a result for this prompt. Respondents were asked to choose the image that is most aligned with the given prompt, and the highest quality image. Deciding between methods may be very difficult certain examples, and therefore we allowed to choose results from more than a single method. The final score of each method is defined as the percentage of times the results of this method were selected. Note that since allow respondents scores are not summed to 100. The results are summarized in Table 2 As can be seen, all methods provide results that are aligned with the prompt and our method attains the highest score. In terms of quality, our method obtains significantly better results compared with the competing methods.
2023b), and our NFSD method. Specifically,
choose more than single method in each question, the
DreamFusion
Fantasia3D
Magic3D
PreaificDreamer NFSD (ours)
prompt alignment image quality 3.19
46.09
61.74
42.03
41.55
65.51
27.83
17.75
37.68
70.14
Table 2: User study results. The highest possible score for each method is 100.
A.7 ADDITIONAL RESULTS
In Figures and
15
we present additional results for text-to-NeRF generation using our NFSD.
"Michelangelo style statue of dog reading news on a collphone"
"pink flamingo in paradise"
"Orangutan cating banana"
"["] baby bunny sitting on top to a stack pancakes"
"A rat driving car"
"A two headle fire dragon with small wings"
Figure 14: NeRFs optimized with NFSD. [*] "A zoomed out DSLR photo of". [...--'A wide angle zoomed out DSLR photo of zoomed out view of".
16

---

Published as a conference paper at ICLR 2024
"A bighly detailed of dinosaur made of leaves"
"Decorative mythological Chinese dragon"
[[...] Tower Bridge made gingerbread and candy"
camel with colorful saddle"
ceramic lion"
"A golden with white wings"
"A sculpture of cleopatra with ccremonial decoration"
"A ripe strawberry"
"A rohotic butterfly metal flower"
"A DSLR photo of classic Packard car"
"A small saguaro cactus planted clay pat"
"A white cat sleeping"
"Ruiling langga mewah"
"A lumbuga pendant depicting a cal"
Figure 15: NeRFs optimized with NFSD. [*] "A zoomed out DSLR photo of". [..]-"AA wide angle zoomed DSLR photo of zoomed out view of".
17

---

Published as a conference paper at ICLR 2024
A.8 MORE COMPARISONS WITH RELATED METHODS
As mentioned in the main paper, methods in different papers were implemented differently. For example, the diffusion model used in each of the methods if different, the NeRF implementation may be different, etc. Hence, we present results obtained by running the unified threestudio (Guo al., 2023) framework implementation for all the methods. For all the methods we use Stable Diffusion 2.1-base and run them on single GPU. As can be seen in Figure T6 for some of the methods the results look better with threestudio (e.g.. DreamFusion), but for other methods the results may seem worse. For Magic3D (Lin et al. (023) the gap may be caused by the diffusion model, while for Fantasia3D (Chen et al. 2023) may be caused by the amount of resources. The results reported by the authors of Fantasia3D were obtained by optimizing on 8 GPUs while use a single one. Overall, our method is comparable or better than all the methods, exhibiting high resolution and detailed features.
DreamFusion
Latent-NeRF
Magic3D
Fantasia3D
ProlificDreamed
NFSD (ours)
Figure 16: Comparison of NESD with other methods using the threestudio (Guo et al. 2023 imple- mentation.
Additionally, in Figures 17 18 19 20 and 21 we show additional comparisons with results obtained by other methods, as reported their respective original papers.
18

---

Published as a conference paper at ICLR 2024
"A marble bust of a mouse"
DreamFusion
Magic3I
Fantasia3D
ProlifiDDreamer
NFSD (ours)
"A car made sushi"
DreamFusion
Magic3D
Fantasia3D
ProlificDreamer
NFSD (ours)
Figure 17: Comparison of NFSD with other methods using results obtained from the original papers.
"A ripe strawberry"
Fusion
Magic3I
Fantasia31D
NFSD (ours)
"A small saguaro cactus planted in clay pot"
DreamFusion
ProlificDreamer "A rabbit, animated movie character, high detail 3d mode"
Magic3D
NESD (ours)
DreamFusion
Magic3D
ProlifiDDreamer
NFSD (ours)
Figure 18: Comparison of NFSD with other methods using results obtained from the original papers.
19

---

Published as a conference paper at ICLR 2024
stack of pancakes covered in maple syrup"
DreamFusion
Magic3I
Fantasia31)
NFSD (ours)
"A delicious croissan"
DreamFusion
Fantasia3D
ProlificDreamer
NFSD (ours)
"An ice cream sundae"
DreamFusion
Magic3D
Fantasia3D
NFSD (ours)
"A baby bunny sitting on top of stack of pancakes"
DreamFusion
Magic3D
ProaifieDreamer
NFSD (ours)
"A hamburger"
DreamFusion
LatentNerf
Fantasia
NFSD (ours)
Figure 19: Comparison of NESD with other methods using results obtained from the original papers.
20

---

Published as a conference paper at ICLR 2024
"Baby dragon hatching out of a stone egg"
DreamFusion
Magic3D
NFSD (ours)
"An iguana holding balloon"
DreamFusion
Magic3D
NFSD (ours)
blue tulip"
DreamFusion
ProlificDreamer
NFSD (ours)
"a cauldron full of gold coins"
DreamFusion
PromificDreamer
NESD (ours)
Figure 20: Comparison of NFSD with other methods using results obtained from the original papers.
21

---

Published as a conference paper at ICLR 2024
"Bagel filled with cream cheese and lox"
DreamFusion
Magic3D
NFSD (ours)
"A plush dragon toy"
DreamFusion
ProlifiDDreamer
NFSD (ours)
"A ceramic lion"
DreamFusion
Magic3D
NFSD (ours)
"Tower Bridge made out of gingerbread and candy"
DreamFusion
Magic3D
NFSD (ours)
Figure Comparison of NESD with other methods using results obtained from the original papers.
22


Reviews:
[{"review_id": "dDx0OjKT6j", "reviewer": "Reviewer_eQLa", "summary": "This study proposes a simple yet effective method, Noise-Free Score Distillation (NFSD), to improve the conventional score distillation using a minimal modification. This study decomposes the score with classifier-free guidance (CFG)  into three terms, the condition, the domain, and the denoising components. Then, they remove the prediction error on unconditional samples between the estimated scores and and injected noises, since the score prediction error on unconditional samples is noisy. The domain score is estimated by a text prompt for a text-to-image model. The experimental results show that extremely high scale of CFG in score distillation is unnecessary, and NFSD can improve fine-grained details of generated images or neural fields.", "questions": "Q1. Although the authors discuss the low diversity of NFSD, I wonder the detailed reason why the reduced CFG scale cannot produce diverse visual contents. In addition, can the authors provide the samples with different seeds and the same text prompts to show the diversity of generated contents?\n\nQ2. In Figure 3, what is the diffusion timestep? In addition, I think that the authors should show the results of \n$x_{\\text{OOD}} + \\delta_D + \\delta_N^{\\text{OOD}}$\n, where $\\delta_N^{\\text{OOD}}$ is the denoising score of $x_\\text{OOD}$, not $x_\\text{ID}$. I also suggest clarifying the notation of $\\delta_N$ and $\\delta_D$ in Figure 3, since the two scores are from different samples. \n\nQ3. Why do the prediction errors in Figure 4 (the second row) show a less-noisy map at t=1000? I think that the results are unintuitive, since they indicate that the score prediction at t=1 is difficult, while the score prediction at t=1000 is conducted almost perfectly except for the central region. \n\nQ4. In Section 4, the authors claim that the magnitude of the noise to be removed is monotonically decreased in the backward process. I wonder how we can assume that the scale of the domain score is preserved? Is there any rationale that only $\\delta_N$ decreases over the backward process, while $\\delta_D$ preserves its scale?\n\nQ5. How about the results of SDS, where its CFG adopts the same negative prompts as NFSD, described in Section 4?\n\nQ6. The authors have discussed that ProlificDreamer’s LoRA adaptation has a similar role with NFSD to exclude the prediction error of the denoising term $\\delta_N$. Then, can the LoRA of ProlificDreamer be replaced with NFSD, while variational particle optimization is used? It would be interesting to show the compatibility of NFSD with ProlificDreamer.\n\nQ7. Since NFSD requires additional inference at each training iteration due to negative prompting, I think that comparing the results of NFSD with those of SDS in terms of the number of function evaluations (NFEs) of diffusion models. \n\nQ8. In Section 4, how can we assume that the score prediction on text conditions is also composed of $\\delta_D + \\delta_N + \\delta_C$, where $\\delta_D + \\delta_N$ is equal to the unconditional prediction? I think that it is a technical flaw, since Eq.(3) just implies $\\epsilon_\\phi (z_t ; t) - \\epsilon_\\phi (z_t  ; y=p_\\text{neg}, t) = \\delta_{C=p_\\text{neg}}$. That is, $\\delta_C$ is defined with both conditional and unconditional scores, not solely on the conditional score term. \n\nQ9. How is the negative prompt to estimate the domain term defined? I wonder whether the negative prompt is universal regardless of the image renderer. In addition, it assumes that the domain score can be estimated by the text prompts. However, how can we say that the image is from out-of-distribution, when the image can be estimated by text prompts of text-to-image models?\n\nQ10. Can be simply using $s \\delta_C$ for the score distillation possible without $\\delta_D$? That is, using $\\delta_D$ is necessary?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The proposed method, NFSD, is simple yet effective. In addition, the qualitative results support and demonstrate the effectiveness of NFSD.\n\nS2. The paper is well-organized and easy to understand.\n\nS3. The analogical decomposition of scores into three terms is interesting and makes sense.", "weaknesses": "W1. Despite the interestingness of score decomposition, the proposed method stems from numerous assumptions based on empirical findings without a principal approach.\n\nW2. Thorough experiments to validate the effectiveness of NFSD are absent. Although the qualitative results show improved quality of text-to-NeRF than conventional SDS-based approaches, there is no ablation study and quantitative result. \n\nW3. Some technical parts lack enough rationales. For example, estimating the domain score by negative text prompts lacks the rationales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SEUdcpZHpw", "reviewer": "Reviewer_opTp", "summary": "This paper first revisits Score Distillation Sampling (SDS) and proposes to decompose the updates generated by SDS into three components: domain correction, noise estimation, and condition direction. Through this approach, the authors provide an explanation for why SDS accommodates a high Classifier-Free Guidance (CFG) coefficient and introduce Noise-Free Score Distillation (NFSD). NFSD re-estimates the unconditional score using the negative prompt trick. As a result, NFSD can employ a standard CFG weight to alleviate the over-smoothing/saturation problem and enhance the quality of text-guided image editing and 3D asset generation.", "questions": "1. Furthermore, it is not evident how steering the update of SDS could alter the optimization objective. Providing a more rigorous and formal argument would deepen the contribution of this work.\n\n2. The authors introduce Eq. 6 to estimate $\\delta_D$. Can the authors offer a rationale or justification for this approximation? Additionally, including visualizations that align with Fig. 3 would enhance the clarity and understanding of this proposal.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "+ The paper is well structured and organized. The method introduced in this paper is intuitive and straightforward to implement. The motivations behind the approach are vividly conveyed through clear formulations and effective visualizations.\n\n+ The decomposition of SDS is both novel and intriguing. It not only offers a compelling interpretation of the large CFG weight selection in DreamFusion but also offers valuable insights into DDS [1] and VSD [2].\n\n+ The empirical results clearly demonstrate a significant enhancement in 3D generation through the simple modifications initiated by NFSD.\n\n[1] Hertz et al., Delta Denoising Score, 2023\n\n[2] Wang et al., ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, 2023", "weaknesses": "- While the explanation is intuitively presented, it remains somewhat challenging to discern the fundamental distinction from the negative prompt trick.\n\n- In Sec. 5, the paper asserts that NFSD is notably more efficient than VSD, despite sharing a similar working mechanism. Although this claim appears obvious, I would recommend providing quantitative evidence to substantiate this advantage when compared to other baseline methods. It is conceivable that dropping the noise term could even speed up the convergence of ancestral sampling by using fewer optimization steps.\n\n- Further ablation studies are needed to validate the assertions put forth in this paper. In comparison to SDS, two terms have been omitted according to Eqs. 5 and 7: the noise prediction $\\delta_N$ and the noise ground truth $\\epsilon$. However, it remains unclear which of these terms plays the most pivotal role in improving the final results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MoWY0RH56d", "reviewer": "Reviewer_wypJ", "summary": "This paper reexamined the Score Distillation Sampling and proposed Noise-Free Score Distillation. The details of the images generated using SDS are more blurred, due to the slightly different distribution between the images generated by the generator(x(\\theta)) and the original image x. This paper found a decomposition to counteract this effect and the authors use this decomposition to explain why previous methods have improved SDS. Adequate experimental results also demonstrate the effectiveness of the methodology.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper proposed a decomposition method to solve the problem of ambiguous results caused by the different distribution of the images generated by the generator and the original images; and uses this decomposition method to explain why previous methods have improved SDS. The experimental results are intuitive.", "weaknesses": "I'm concerned about whether p_{neg} = “unrealistic, blurry, low quality, out of focus, ugly, low contrast, dull, dark, low-resolution, gloomy” is generalizable across situations and able to cancel out \\delta_{N}. Would a better generator g(\\theta) be able to achieve the same effect, or train a model to estimate the bias \\delta_{N}?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SZsgqBnBTH", "reviewer": "Reviewer_EnYB", "summary": "This work tackles the blurry results from Score Distillation Sampling (SDS) for text-to-3D generation. The score is decomposed into the condition, domain, and noise residual terms; the proposed method is designed to reduce the effect of the undesired noise component, by heuristically estimating the domain term with a negative prompt.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Text-guided 3D generation by leveraging pretrained text-to-image models is a hot, timely topic. The authors propose to improve the famous SDS-based framework with a small modification.\n- The proposed decomposition can be also used to understand existing works (DDS and VSD), which seems like a valuable contribution.\n- The paper is well-written; the terms and derivations are clearly presented.", "weaknesses": "- Although a key value of this work seems to be the decomposition of the SDS loss, I have a few questions on designing the proposed ~NFSW~ NFSD method (<- apologize for this big typo at the initial review):\n  - How did the authors separate the small and large timestep values based on t=200? Why not t=100, 300, or 400?\n  - Is it valid to assume δ_{C=p_neg} ≈ −δ_{D}? Did the choice of the negative prompt affect the performance?\n  - How about changing (6) into just using the second part of (6) for all the time steps (i.e., unconditional term - negative-prompt-induced term, for all the time steps)?\n- I appreciate the effort for many visual results; however, the lack of any quantitative results concerns me a lot. Is it possible to include the comparison using CLIP R-Precision of Table 1 in the DreamFusion paper? Furthermore, leveraging the MS-COCO text-to-image benchmark with FID/IS/CLIP score metrics may be worth trying to justify the results of 2D image generation in Figure 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)