Score Distillation via Reparametrized DDIM
Authors: Artem Lukoianov
Haitz Sáez de Ocáriz Borde
Kristjan Greenewald
Vitor Campagnolo Guizilini
Timur Bagautdinov
Vincent Sitzmann
Justin Solomon
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D generation
diffusion
score distillation
Primary area: diffusion_based_models
TL;DR: We bridge a gap between 3D Score Distillation and 2D image generation both in terms of theory and qualitative results

Abstract: While 2D diffusion models generate realistic, high-detail images, 3D shape generation methods like Score Distillation Sampling (SDS)  built on these 2D diffusion models  produce cartoon-like, over-smoothed shapes. To help explain this discrepancy, we show that the image guidance used in Score Distillation can be understood as the velocity field of a 2D denoising generative process, up to the choice of a noise term. In particular, after a change of variables, SDS resembles a high-variance version of Denoising Diffusion Implicit Models (DDIM) with a differently-sampled noise term: SDS introduces noise i.i.d. randomly at each step, while DDIM infers it from the previous noise predictions. This excessive variance can lead to over-smoothing and unrealistic outputs. We show that a better noise approximation can be recovered by inverting DDIM in each SDS update step. This modification makes SDS's generative process for 2D images almost identical to DDIM. In 3D, it removes over-smoothing, preserves higher-frequency detail, and brings the generation quality closer to that of 2D samplers. Experimentally, our method achieves better or similar 3D generation quality compared to other state-of-the-art Score Distillation methods, all without training additional neural networks or multi-view supervision, and providing useful insights into relationship between 2D and 3D asset generation with diffusion models.

Score Distillation via Reparametrized DDIM
Artem Lukoianov!
Haitz Sáez de Ocáriz Borde
Kristjan Greenewald
Vitor Campagnolo Guizilini
Timur Bagautdinoy
Vincent Sitzmann'
Justin Solomon'
Massachusetts Institute of Technology ?University of Oxford
MIITIIBM Watson AI Lab, IBM
Research *Toyota Research Institute
'Meta Reality Labs Research
httpa://lukoinovv/ com/ sds//
Abstract
While 2D diffusion models generate realistic, high-detail images, 3D shape genera- tion methods like Score Distillation Sampling (SDS) built on these 2D diffusion models produce cartoon-like, over-smoothed shapes. To help explain this dis- crepancy, we show that the image guidance used in Score Distillation can be understood as the velocity field of 2D denoising generative process, up the choice of noise term. In particular, after a change of variables, SDS resembles a high- high-variance version of Denoising Diffusion Implicit Models (DDIM) with a differently-sampler noise term: SDS introduces noise i.i.d. randomly at each step, while DDIM infers from the previous noise predictions. This excessive variance lead to over-smoothing and unrealistic outputs. We show that a better noise approximation can be recovered by inverting DDIM in each SDS update step. This modification makes SDS's generative process for 2D images almost identical detail, to DDIM. In 3D, it removes over-smoothing, preserves higher-frequent and brings the generation quality closer to that of 2D samplers. Experimentally. our method achieves better or similar 3D generation quality compared to other state-of the-art art Score Distillation methods, all without training additional neural networks or multi-view supervision, and providing useful insights into relationship between 2D and 3D asset generation with diffusion models.
DDIM (CFG-7.5)
b) 2D SDS (CFG-7.5) 2D ours (CFG -0--7.. 7.5) d) 3D SDS (CFG-100) 3D ours (CFG-7.5)
Figure 1: Score Distillation Sampling (SDS) "distills" 3D shapes from 2D image generative models like DDIM. While DDIM produces high-quality images (a), the same diffusion model, yields blurry results with SDS in the task of 2D image generation (b); in 3D, SDS yields over-saturated and simplified shapes (d). By replacing the noise term in SDS to agree with DDIM, our algorithm better matches the quality of the diffusion model in 2D (c) and significantly improves 3D generation (e).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

"a DSLR photo of an old man"
"a DSLR photo of white fluffy cat"
"pumpkin head zombie, skinny, highly detailed, photorealistic" "
"photograph of black leather backpack"
"bagel filled with cream cheese and lox"
"a DSLR photo of Cthulhu"
Figure 2: Examples of 3D objects generated with our method.
Introduction
Image generative modeling saw drastic quality improvement with the advent of text-to-image diffusion models [1] trained on billion-scale datasets [2] with large parameter counts [3]. From short prompt, these models generate photorealistic images, with strong zero-shot generalization to new classes [4], Efficient training methods for image data, combined with Internet-scale datasets, enabled the development of these models. However, applying similar techniques to domains where huge datasets scarce, such as 3D shape generation, remains challenging.
The need for 3D objects in downstream applications like vision, graphics, and roboties motivated methods like Score Distillation Sampling (SDS) [5] and Score Jacobian Chaining (SJC) [6], which optimize volumetric 3D representations [7, 9] using queries to a 2D generative model [10]. In every iteration, SDS renders the current state of the 3D representation from random viewpoint, adds noise to the result, and then denoises using the pre-trained 2D diffusion model conditioned on text prompt. The difference between the added and predicted noise is used on the rendered images, which is propagated to the parameters of the 3D model. The underlying 3D representation helps make the generated images multi view consistent, and the 2D model guides individual views towards a learned distribution of realistic images.
gradient-style update
In practice, however, noted in [11, 12, 13], SDS often produces 3D representations with over- saturated colors and over-smoothed textures (fig. Id), not matching the quality of the underlying 2D model. Existing approaches tackling this problem improve quality at the cost of expensive re-training or fine-tuning of the image diffusion model [11]. complex multi-stage handling 3D representations like mesh extraction and texture fine-tuning [14, 15, 11], altering the SDS guidance [13, 12, 16).
As an alternative to engineering-basee improvements to SDS, in this paper we reanalyze the vanilla SDS algorithm to understand the underlying source of artifacts. Our key insight is that the SDS update rule steps along an approximation of the DDIM velocity field. In particular, we derive Score Distillation from DDIM with a change of variables to the space of single-step denoised images. In this light, SDS updates are nearly identical to DDIM updates, apart from one difference: while DDIM samples noise conditionally on the previous predictions, SDS resamples noise i.i.d. in every iteration. This breaks the denoising trajectory for each independent view and introduces excessive variance. Our perspective unifies DDIM and SDS and helps explain why SDS produce blurry and

---

over- wwtrrsaturated results: the variance- boosstng effect of noisy guidance is usually mitigated with high Classifier Free Guidance (CFG) [17] to reduce sample diversity the cost of over-saturation [18].
Based on our analysis, we propose an alternative score distillation algorithm dubbed Score Distillation via Inversion (SDI), closing the gap DDIM. We obtain the conditional noise required for consistency of the denoising trajectories by inverting DDIM on each step of score distillation (fig. 5). This modification yields 3D objects with high-quality textures consistent with the 2D diffusion model (fig. le). Moreover, in 2D, our method closely approximates DDIM while preserving the incremental generation schedule of SDS (fig. lc).
Our key contributions are as follows:
We prove that guidance for each view in the SDS algorithm is a simplified reparameterization of DDIM sampling: vanilla SDS samples random noise at each step, while DDIM keeps the trajectories consistent with previously-predicee
noise.
We propose a new method titled Score Distillation via Inversion (SDI), which replaces the problem- atic random noise sampling in SDS with prompt-conditioned DDIM inversion and significantly improves 3D generation, closing the quality gap to samples from the 2D model.
We systematically compare SDI with the state-of-the fit-hhherttt Score Distillation algorithms and show that SDI achieves similar or better generation quality, while not requiring training additional neural networks or multiple generation stages.
2 Related work
3D generation by training on multi-view data. Recent 3D generation methods leverage multi-view or 3D data. Zero123 [19] and MVDream [20] generate consistent multi-view images from text; 3D radiance field is then obtained via score distillation. Video generative models can be fine-tuned on videos of camera tracks around 3D objects, similarly yielding model that samples multi-view consistent images to train a 3D radiance field [21, 22]. Diffusion with Forward Models [23] and Viewset Diffusion [24] directly train 3D generative models from 2D images. While these methods excel at generating multi-view consistent, plausible 3D objects, they depend on multi-view data with known camera trajectories, limiting them synthetic or small bundle-adjusted 3D datasets. We instead focus on methods that require only single-view training images.
Distilling 2D into 3D. Score Distillation was introduced concurrently in Dreamfusion SDS [5], Score Jacobian Chaining (SJC) [6], and Magic3D [14]. The key idea is to use a frozen diffusion model trained on 2D images and "distill" it into 3D assets. A volumetric representation of the shape is rendered from a random view, perturbed with random noise, and denoised with the diffusion model; the difference between added and predicted noise is used improve the rendering. These works, however, suffer from over-smoothing and lack of detail. Usually a high value of classifier free guidance (CFG 100) [17] is used to reduce variance the cost of over-saturation [18].
PremificDreamer [11] generates sharp. detailed results with standard CFG values (s= 7.5) and without over saturatinn. The key idea is to overfit an additional diffusion model to specifically denoise the current 3D shape estimate. Fine-tume minittnnngg the second model, however, is cumbersome and theoretical justification for this change is still unclear. Recent papers further improve on ProlifiDrremme''s 's results or try to explain its behavior. SteinDreamer [25], for example, hypothesizes that ProlificDreamer" s improvements come from variance reduction of the sampling procedure [26].
Other papers propose heuristics that improve SDS. [12, 16, 27] decompose the guidance terms and speculate about their relative importance. Empirically, visual quality can be improved by suppressing the denoising term with negative prompts [12] or highlighting the classification term [16]. [14, 15, 11, 28] use multi-stage optimization: they first train volumetric representation and then extract mesh or voxel grid to fine-tune geometry and texture. HiFA [13] combines ad-hoc techniques: additional supervision in latent space, time annealing, kernel smoothing, z-variance regularization, and pretrained mono-depth estimation network to improve the quality of single-stage NeRF generation.
LueidDreamer, or Interval Score Matching (ISM). hypothesizes based on empirical evidence that in SDS, the high-variance random noise term and large reconstruction error of a single-step prediction togehter cause over-smoothing [29]. Based on these observations, the authors replace the random noise term SDS with noise obtained by running DDIM inversion and introduce multi-step denoising to improve reconstruction quality. As we will show in section 4, the update rule of ISM can be seen
3

---

Prompt- "colored photograph of an old man" CFG Value
5.
30.
100.
Figure 3: The effect of CFG values 2D generation with StableDiffusion 2.1 [1]. For small values, the model tends to ignore certain words in the prompt. For high values, images become over-saturated
as a special case of our formulation. Moreover, our analysis reveals that the added noise should be inferred conditionally the text prompt y. which further improves quality.
In this work, rather than augmenting the SDS pipeline or relying on heuristics, we derive Score Distillation through the denoising process of DDIM and propose simple modification of SDS that significantly improves 3D generation.
3 Background
Diffusion models. Denoising Diffusion Implicit Models (DDIM) generate images by reversing a diffusion process [30, 31, 32]. After training denoiser e/ and freezing its weights 0, the denoising process can be seen as an ODE on rescaled noisy images z(t) = zxt)/ /a(t). /a(t) Given a prompt y and current time step E [0, IJ, the denoising process satisfies:
df((t) <b (Va(1)((1).)) dt
do(() dl
where ([1) sampled from a Gaussian distribution, o(t) v a(t)/ (C1)//alf at), and a(t) are scaling factors. When discretized with forward Euler, this equation yields the following update to transition from step less noisy step t
a(t -) =(t) - cb( (t -t)]
The DDIM ODE can also integrated reverse direction to estimate æ(t) for any E [0, 1] from a clean image Fo. This operation is called DDIM inversion and is studied in multiple works [33, 34).
Classifier-free guidance. Classifier free guidance (CFG) [17] provides high-quality conditional samples without gradients from auxilary models [35]. CFG modifies the noise prediction é (score function) by linearly combining conditional and unconditional predictions:
(=()),)) = (x(1), Ø) + + (E)((()) (x(1),
where the guidance scale is a scalar, with 7 = corresponding to unconditional sampling and y to conditional. In practice, larger values 7 are necessary to obtain high-quality samples a cost of reduced diversity and extreme over-saturation We demonstrate the effect of different CFG values in fig. 3. For the rest of the paper use the modified CFG version of the denoiser ca (x((), v).
Score Distillation. Diffusion models efficiently generate images and can learn to represent common objects from arbitrary angles [36] and with varying lighting [5]. Capitalizing on this success, Score Distillation Sampling (SDS) [5] distills pre-trained diffusion model e to produce a 3D asset. In practice, the 3D shape is usually parameterized by NeRF [7], InstantNGP [8], Gaussian Splat- ting [9]. Multiple works additionally extract an explicit representation for further optimization [14, 15, 11, 28]. We use InstantNGP [8] to balance between speed and case of optimization.
Denote the parameters of a differentiable 3D shape representation by E Rd and differentiable rendering by function (1 c) Rd x C - IR Nx.V that returns an image given camera parameters e E C. Intuitively, in each iteration, SDS samples e, renders corresponding (image) view g(v), c), Ao, /) to level
perturbs it with the true and predicted noise propagated to the parameters of the 3D shape. More formally, after sampling the camera view and randomly drawing a time SDS renders the volume and adds
1]. and denoises with e the difference between

---

To (o(12)
zo (ts)))
(t g(vic) x01
S. wt..n............. w.m.n.n...... m.n.n.
DDIM
Single-step denoising
((1)
Noising
a(b2) i13
DDIM in To space
Figure 4: Left: Evolution of variables in Score Distillation with time. The top row depicts how noisy images x(t) evolve during 2D generation; the middle row shows evolution of a NeRF for 3D generation; and the bottom row shows how the single step denoised variable xo (t) changes with t. Right: Each step of DDIM steps toward a denoised image. This can be seen as a step to ro(() and step back to a slightly less noisy image. Through a change-of-variables we obtain a process on (1).
Gaussian noise с to obtain a noisy image xtt) = (a(t)(()) c) + v1 - a(t)c. Then, SDS improves the generated volume by using gradient(- direction to update its parameters s::
0g
V WsDS = (t) [e)(((()) (x(t), 3) - c]
(4)
Oup'
We refer to the term (=()),)) - []]
as guidance in score distillation, it 'guides" the views of the shape. In theory, this expression may not correspond the true gradient of function and there are many hypotheses about its effectiveness [5, 25, 12, 16, 26]. In this work, we show that instead it can be seen as a high-variance version of DDIM after a change of variables.
Linking SDS to DDIM
Discrepancy in image sampling. Beyond the lack of formal justification of eq. (4), in practice SDS results are over-saturated and miss details for high CFG values, while they are blurry for low CFG values. To illustrate this phenomenon, fig. 1 shows simple experiment, inspired by [11]: We replace the volumetric representation in eq. (4) with an image gan (122,, c) == 2/ E RNNN In this case, SDS becomes an image generation algorithm that can be compared other sampling algorithms like DDIM [37]. Even in this 2D setting, SDS fails to generate sharp details, while DDIM with the same underlying diffusion model produces photorealistic results, motivating our derivation below.
Why not use DDIM as guidance? Given the experiment above, a natural question to ask is if is possible to directly use DDIM's 's update direction from eq. (1) SDS guidance in eq. (4) to update the 3D representation. The problem with this approach lies in the discrepancy between the training data of the denoising model and the images generated by rendering the current 3D representation. More specifically, the denoising network expects an image with a certain level of noise corresponding to time as defined by the forward (noising) diffusion process, whereas renderings of 3D representations g(b, c) evolve from a blurry cloud to a well-defined sample (fig. 4 left).
Evolution of zo)(). Instead of seeing DDIM as denoising process defined on the space of noisy images zt), we reparametrize to a new variable:
zo z(t) - (
(5)
In words, Fu (f) is the noisy image at time t denoised with a single step of noise prediction. Empirically, the evolution of (t(() is similar to the evolution of g(,, c)-from blurry to sharp. The left side of fig. 4 compares these processes. This similarity motivates us to rewrite eq. (1) in terms of xt()), and to understand SDS as applying similar updates to the renderings of its 3D representation.
Reparametrizing DDIM. Figure 4 (right) shows schematically how the DDIM update to Iit) alternates between denoising to obtain ro(() and adding the predicted noise back get cleaner a(t-r). Based on the intuition above, we reorder the steps, adding noise to ro(() and then denoising to estimate zo(t - 7). Consider neighboring time points and r < t in discretized DDIM eq. (2) (lower time means less noise). We rewrite eq. (2) using the definition of zu(() from eq. (5) to find
zo(t T) zo(() a(t 7) [ (x(t v),v) - (=((),)))
(6)
5

---

Rendering 2o (t)
Noisy 2t()
Target Fo 7)
DDIM inversion
Denoising
Image Loss [|ro(t) zo(t - -)]
Figure 5: Overview of SDL. At each training iteration, SDI renders a random view of the 3D shape, runs DDIM inversion up to the noise level t, and denoises the image with a pre-trained diffusion model for noise level t - T. Finally, the denoised image is back-propagated into the 3D shape.
which is consistent with the intuition behind SDS: improving an image invol ves perturbing the current image and then denoising it with a better noise estimate. We cannot directly apply eq. (6) to SDS in 3D, since it depends on 2 ()); if we think of so (L) as similar to a rendering of the 3D representation for some camera angle, it is unclear how obtain a consistent set of preimages xx(t) at each step of 3D generation. From eq. (5), however, x²t) should satisfy the following fixed point equation:
xxt) Va(t)zo(t) = [xx(t) (a(0aa(0))//
a(t)
or rewritten terms of noise
6 = ==(((((())(())) + v1 a()e,
(8)
Define Ni, (=0(1)) = as a solution of this equation given (1). Then, we can write: (x()),)) = (zo(t)) and =( = Va - -))o(t) + v alt - ()) (zo(t))
(9)
Thus, eq. (6) turns into:
to noised with n.. to time -
zu(t-r) = zu(t)-o(t-T)(e (Valt=VVaaarrtlf(( V1-/aiattttt
nnn (2o (z) (t)),
(Zo(t)) (10)
predicted moise
noise sample ie)
We can already see that the structure of eq. (10) is very similar to the SDS update rule in eq. (4). Note that the update direction in eq. (10) is the same as in the SDS update rule eq. (4), where xi, plays the role of the random noise sample e. We could use it as a guidance for the 3D generative process in SDS by replacing in eq. (4) with Ki, (t)) In practice, however, it is hard to solve eq. (8), as eg is nigh-dimensional and nonlinear. In an unconstrained 2D generation, kt, can be cached from previous denoising step, matching the update step to DDIM exactly as in fig. 1c. In 3D), however, this impossible due to the simultaneous optimization of multiple views and projections to the space of viable 3D shapes. Below we show that naïve approximation replacing ki with Gaussian yields SDS, and we will propose alternatives that are more faithful to the derivation above.
SDS as a special case. From eq. (10), to get a cleaner image, we need to bring the current image to time with noise sample x.y, denoise the obtained image, and then subtract the difference between added and predieted noise from the initial image. coarse approximation of kc, uses i.i.d. random noise Kspss (t) ~ NN(0, I). matching the forward process by which diffusion adds noise. This choice of KSDs precisely matches the update rule eq. (10) to the SDS guidance in eq. (4).
ISM special case. The main update formula in Interval Score Matching (ISM) [29] is a particular case of eq. (10), where Ki, is obtained via DDIM inversion without conditioning on the text prompt As demonstrated in section 5, DDIM inversion approximates solution of eq. (10), explaining the improved performance of ISM. However, our analysis suggests that an even better- perterrpprforming ky should incorporate the text prompt y, which further improves the results and avoids over-saturation.
5 Score Distillation via Inversion (SDI)
As we have shown, SDS follows the velocity field of reparametrized DDIM in eq. (10), when KSDS (sDs(2o(t)) (t)( is randomly sampled in each step. Our derivation, however, suggests that ksps(zo

---

could be improved by bringing it closer to a solution of the fixed- equation in eq. (8). Indeed, randomly sampling ksns as Dreamfusion yields excessive variance and blurry results for standard CFG values, while using higher CFG values leads to over-saturation and lack of detail. On the other hand, solving eq. (8) exactly is challenging due to its high dimensionality and nonlinearity.
Like ISM [29], we suggest to obtain Ki by inverting DDIM, that is. by integrating the ODE in eq. (1) with evolving backwards (from images to noise) as in [37, 33, 34]. As we can see in eq. (8), ki should be function of the text prompt y, leading us to perform DDIM inversion conditionally on y- unlike ISM. This process approximates but is not identical to the exact solution for kl. fixed points of eq. (8) invert a single large step of DDIM, while running the ODE in reverse inverts the entire DDIM trajectory. We ablate alternative choices for ky in section 6.2 and conclude that in practice DDIM inversion offers the best approximation quality. Additionally, to match the iterative nature of DDIM, we employ a linear annealing schedule of t. We refer to our modified version of SDS as Score Distillation via Inversion, or SDI.
Score Distillation Sampling
Score Distillation via Inversion
20-
1.0
0,8
0.6
0.4
0.2
1.0
0.8
0,6
0.4
0,2
Figure 6: Comparison of intermediate variables in SDS and SDI (ours) for different timesteps I. Starting with rendering of 3D shape we demonstrate how each algorithm perturbs it (s:(1) variable on the top row) and how it is denoised with a single step of diffusion (zo(t) variable on the bottom row). The prompt used is "Pumpkin head zombie, skinny, highly detailed, photorealistic, side view."
Figure shows the effect of inferring the noise via DDIM inversion instead of sampling it randomly. The special structure of the improved K² results in more consistent single-step generations and produces intricate features at earlier times. When inverted and not sampled, the noise appears 'in the right place': in SDD the noise covers the whole view, including the background, whereas in ours the noise is concentrated on the meaningful part of the 3D shape. This improves geometric and temporal coherency for zo predictions even for large f. The reduced variance drastically increases sharpness and level of detail. Moreover, it allows to reduce CFG value of generation "fiwd to the standard 7.5, avoiding over-saturation. Another interesting finding is that DDIM inversion works best when the reverse integration is performed with negative CFG Tine -Yiw = -7.5. The overview of our method is presented in fig. 5, the details about inversion algorithm are presented in section 6.2, and the implementation details are discussed in appendix A.
6
Experiments
6.1 3D generation
We demonstrate the high-fidelity 3D shapes generated with our algorithm in fig. 2 and provide more examples of 360 views in appendix H.2. A more detailed qualitative and quantitative comparison of our method with ISM [29] is provided in appendix B. Additionally, we report the diversity of the generated shapes in appendix C.
Qualitative comparisons. Figure 7 compares 3D generation quality with the results reported in past work using a similar protocol to [11, 12]. For the baselines we chose: Dreamfusion [5] (the work we build on). Noise Free Score Distillation (NFSD) [12] (uses negative prompts in SDS), PromificDreamer [11] (fine-tunes and trains a neural network to denoise the 3D shape), Interval Score Matching (ISM) [29] (obtain the noise sample by inverting DDIM and perform multi-step denoising), and HiFA [13] (guides both image and latent spaces, regularizes the NeRF, and supervises the geometry with mono-depth estimation). Our figures indicate that Score Distillation via Inversion (SDI) yields similar or better results compared with ofate-of-the-art. Appendix H.3 presents more extensive comparison.
7

---

Dreamfusion
NFSD
PreamficDreamer
ISM
HiFA
Ours
Figure 7: Comparison of 3D generation with other methods using their reported results. The prompts "An ice cream sundae" and "A 3D model of an adorable cottage with thatched roof".
Quantitative comparison. We follow [5, 16, 25] to quantitatively evaluate generation qual- ity. Table 1 provides CLIP scores [38] to measure prompt-generation alignment, computed with torchmetrics [39] and the VIT-B/32 model [40]. We also report ImageReward (IR) [41] to imitate possible human preference. We include CLIP Image Quality Assessment (IQA) [42] to measure quality ("Good photo" vs. "Bad photo"), sharpness ("Sharp photo" vs. "Blurry photo"), and realism ("Real photo" vs. "Abstract photo"). For each method, test 43 prompts with 50 views. For multi-stage baselines, we run only the first stage for fair comparison. We report the percentage of generations that out-of-memory as well as mean run time and VRAM usage. For VRAM, we average the maximum usage of GPU memory between
generate empty volume as diverged ("Div." in the table).
As many baselines not open-source, we use their implementations in threestudio [43]. SDI outperforms SDS and matches or outperforms the quality state-of-the-art methods, offering simple fix to SDS without additional supervision multi-stage training.
Table Quantitative comparisons to baselines for text-to-3D generation, evaluated by CLIP Score and CLIP IQA. We report mean and standard deviation across 43 prompts and 50 views for each.
Method
CLIP Score (t)
CLIP IQA (%)
IR (?)
Div. (%) J
Time
VRAM
"quality"
"sharpaess"
"real"
SDS [S]. 10k steps
29.81 2.49
76 6.6
99 + 1.2
98 2.4
-1.51 = 0.83
18.6
66min
6,2GB
[6], 10k steps
30.30 :: 1.98
7.4
99
98 1.1
-1.76
11.6
13min
13.1GB
VSD [11], 25k steps
33.31
2.39
77 6.7
98 + 1.3
96 4.4
-1.17 + 0.58
23.2
334min
47.9GB
ESD [44), 25k steps
32.79
2.15
77 7.2
98 1.2
97
-1.20
14.0
33
46.8GB
HIFA [13], 25A steps
32.80
6.5
98 084:3
9.2
-1.16
4.7
235min
46.4GB
SDI(ours), 10k steps
33.47 + 2.49
82
98 + 1.3
1.2
-1.18 0.59
4.7
119min
39,2GB
6.2 Ablations
Dreamfusion
+12 rendering
annealing
+ noise inversion
w/o 512
(ours)
rendering
anncaling
Figure 8: Ablation study proposed improvements.
Proposed improvements. Figure 8 ablates the changes we implement on top of SDS. Starting from Dreamfusion [5] with CFG 7.5 we incrementally add: higher NeRF rendering resolution (64 x 64 to 512 x 512), linear schedule on t, and our core contribution DDDM inversion. The results clearly demonstrate that the main improvement quality comes from the inferred noise.
Choice of (x). The key component our algorithm is the choice of inferred noise xi) In theory, < (z) should solve eq. (8), but it is impractical to do so. Hence, in fig. 9 we compare the following choices of n.y (x)) and report their numerical enrors:

---

Random, resampled: Sample Ki (z) in each new update step from (),
Random, fixed:
Sample k. (x) from from.N (0, I) once, and keep it fixed for each iteration:
Fixed point iteration: Since the optimal solution is a fixed point of eq. (8), initialize (x)
(0, 1) and run fixed point iteration [45] for 10 steps (in practice, more steps did not help).
SGD optimization: Optimize n.y (z) via gradient decent for 10 steps, initializing with noise.
Run DDIM inversion for int( (fewer steps for smaller t) steps to time t, with negative CFG Tivv 77.5 5 for inversion and positive "Iwd = 7.5 for forward inference.
DDIM inversion:
The left side of fig. 9 compares the choices for 3D generation qualitatively; the right side plots error induced in eq. (8) (rescaled to Tg variable due to its ambiguity around 0). As can be seen, the regressed noise has big impact on the final generations. Both fixed point iteration and optimization via gradient descent fail improve the approximation of kiy, while fixed point iteration diverges in 3D. On the other hand, DDIM inversion yields a reasonable approximation ky and significantly improves 3D generation quality. We provide visual comparison of the obtained noisy images for each baseline in appendix D.
Random, resampled
Random, fixed
SGD Optim.
DDIM
Inversion
MSE
0.8
0.6
0.4
0.2
Prompt "an iguana holding balloon"
Fixed point iterations diverged
Figure 9: The ablation study of different xy choices in our algorithm. We show the obtained 3D generations the left (Fixed Point Iteration diverges), and the numerical error in eq. (8) induced for each timestep on the right. The resampled and fixed noise strategies produce the same error.
CFG for inversion. [33, 34] report that DDIM inversion accumulates big numerical error for CFG Y O. Surprisingly, we find that DDIM inversion for CFG "Ywdd > 0 can be adequately estimated by running the inversion with negative CFG Vinv -Yfwd Figure 10 compares 3D inversion strategies qualitatively and quantitatively. Naively taking "iivv = "iudd 7.5 yields the biggest numerical error, while other strategies perform on par. 3D generations, illustrated on the right, show that Tiw 7fwd 7.5 introduces excessive numerical errors, causing generation drift in random direction. The best parameters (as we demonstrate in appendix E) for 2D inversion (?iv Nww = 0) fail to converge in 3D as there is not enough guidance toward class sample. Introducing guidance only on the forward pass with "/nv 0, ?Ywdd = 7.5 solves the problem, and the algorithm generates the desired 3D shape, but constantly adding CFG on each step over-saturates the image. Note in that configuration the inversion is performed unconditionally from the text prompt, matching ISM [29]. As we can see, prompt conditioning an important component eq. (8), and using 77.5, Iffd = 7.5 cancels the vatursatuatiinn and produces accurate 3D generations.
Tiw
fiwd 7.5
Twd = 0
Ywd 7.5
Hww 7.5 Tinv -755
Vinv = 7.5
Tinv = 0
Tinv 0
MSE
0.8
0.6
0.4
0.2
Prompt DSLR photo of hamburger"
Figure 10:
Comparing DDIM inversion strategies. Left: Numerical error in eq. (8) from the inferred noise. Right: Generations for different strategies of using CFG values for denoising and inversion.
Number of steps for DDIM inversion. Figure ablates the number of steps required for DDIM inversion. We use 10 as it provides a good balance between generation quality and speed.

---

Prompt- "ppotograhp of'a baby raccoon holding a hamburger"
30 steps
20 steps
10 steps
5 steps
3 steps
4h 7min
2h 54min
1h 59min
Ih 44min
1h 22min
83 # 6.1
83 + 5.8
82 + 6.3
76 5.1
71 7.6
Figure 11:
Ablation study of the number of inversion steps. For each configuration we report an average run time and CLIP IQA "quality" computed on 43 different prompts.
7 Conclusion, Limitations, and Future Work
Helping explain the discrepancy between high-quality image generation with diffusion models and the blurry, over-saturated 3D generation of SDS, our derivation exposes how the strategies are reparameterizations of one another up to single term. Our proposed algorithm SDI closes the gap between these methods, matching the performance of the two in 2D and significantly improving 3D results. The ablations show that DDIM inversion adequately approximates the correct noise term, and adding it to SDS significantly improves visual quality. The results of SDI match or surpass state-of-the-ee arr 3D generations, all without separate diffusion models additional generation steps.
Some limitations of our algorithm motivate future work. While we improve the sample quality of each view, 3D consistency between views remains challenging: as result, despite the convexity loss, our algorithm occasionally produces flat or concave "billboards." " A possible resolution might involve supervision with pre-trained depth or normal estimators. A related problem involves content drift from one view to another. Since there was no 3D supervision, there is little to no communication between opposite views, which can lead to inconsistent 3D assets. Stronger view conditioning, multi-view supervision, or video generation models might resolve this problem. Finally, score distillation is capped by the performance of the underlying diffusion model and is hence prone reproduce similar "hallucinations" (e.g., text and limbs anomalies); the algorithm inherits the biases of the 2D diffusion model and can produce skewed distributions. Appendix H.1 demonstrates typical failure cases.
Potential broader impacts. Our work extends existing 2D diffusion models to generation in 3D settings. As such, SDI could marginally improve the ability of bad actors to generate deepfakes or to create 3D assets corresponding to real humans to interact with in virtual environments or games; also inherits any biases present in the 2D model. While this is critical problem in industry, our work does not explicitly focus on this use case and, in our view, represents a negligible change in such risks, highly convincing deepfake tools are already widely available.
Acknowledgments
The authors thank Lingxiao Li and Chenyang Yuan for their thoughtful insights and feedback.
Artem Lukoianov acknowledges the generous support of the Toyota-CSAIL Joint Research Center.
Vincent Sitzmann was supported by the National Science Foundation under Grant No. 221 2211259, by the Singapore DSTA under 0ST000EC220300223 (New Representations for Vision and 3D Self- Supervised Learning for Label-Efficiier tivity (IARPA) via Department of Interior/Interior Business Center (DOL/IBC) under 140D0423C0075, by the Amazon Science Hub, and by IBM.
Vision), by the Intelligence Advanced Research Projects Ac-
The MIT Geometric Data Processing Group acknowledges the generous support of Army Research Office grants W911NF201016 and W91 NNF1 10293, of National Science Foundation grant IIS- 2335492, from the CSAIL Future of Data program. from the MIT-IBM Watson AI Laboratory, from the Wistron Corporation, and from the Toyota-CSAIL Joint Research Center.
10

---

References
[1] Robin Rombach et al.
Image Synthesis with Latent Diffusion Models". In: 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). IEEE, June 2022. DOI: 10. 1109/ cvpr52688 2022. 01042. URL: http: /xx doi. org/ 10. 1109/
cvpr52688 202.200042......
[2] Christoph Schuhmann et "LAION-5B: An open large-scale dataset for training next genera- tion image-text models". In: Advances in Neural Information Processing Systems 35 (2022),
pp. 25278-252944
[3] William Peebles and Saining Xie. "Scalable Diffusion Models with Transformers". In: 2023 IEEE/CVF International Conference Computer Vision (ICCV). IEEE, Oct. 2023, DOI: 10. 1109 icccv1100000. 2023 00387. URL: http: ///x//xxx doi. www.oog.//..o 110//1111110/1c
2023. 00387.
[4] Alexander C. Li et al. "Your Diffusion Model Secretly a Zero-Shot Classifier". In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Oct. 2023. DOI: 10. 1109 /ttp/ccccc10000// 2023. 00210. URL: http: / /x doi org 1109//ccv5101
2023. 00210.
[5]
Ben Poole et al. "Dreamfusion: Text-to-3d using 2d diffusion", In: arXiiv2209. 14988 (2022). Haochen Wang al. "Score Jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation". In: Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern
[6]
Recognition. 2023, pp. 12619-12629
[7] Ben Mildenhall al. "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis" In: Computer Vision ECCV 2020. Springer International Publishing, 2020, pp. 405-421, ISBN: 9783030584528. DOr: 10. 1007 978 3 030 58452 8 24. URL: http://dx .oix org/ 10wg//1..//0 000000009770077 030- 58452-8 8_ 8_24.
[8] Thomas Müller al. "Instant neural graphics primitives with multiresolution hash encoding".
In: ACM Transactions on Graphics 41.4 (July 2022), pp. 1-15. ISSN: 1557-7368. DOI: 10.
1145/3528222 3530127. URL: http: doi. oor.org/// 10. 1 1145/3528222 .3300127.
[9] Bernhard Kerbl al. "3D Gaussian Splatting for Real-Time Radiance Field Rendering". In:
ACM Transactions on Graphics 42.4 (July 2023), 1-14. ISSN: 1557-7368.
DOI: 10. 1145/
3592433. URL: http: http://xxxo...orrr./.... oog. org/10.....oo 1145/3592433.
[10] Sam Earle et al. "DreamCraft: Text-Guided Generation Functional 3D Environments in Minecraft". In: Proceedings of the 19th International Conference the Foundations of Digital Games, FDG 2024. ACM, May 2024. DOI: 10. 1145 /6499264 3649943. URL: http://dx..o doi. wow.i.or//oooooooooooooooooooooo
oor. 10. 11 1145/3649922 3649943.
[11] Zhengyi Wang al. "Prolifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation". (2024).
In: Advances in Neural Information Processing Systems 36
[12] Oren Katzir et al. "Noise-free Score Distillation". In: The Twelfth International Conference on Learning Representations. 2024.
[13] Junzhe Zhu, Peiye Zhuang, and Sanmi Koyejo. "Hifa: High-fidelity text-to-3d generation with advanced diffusion guidance". In: The Twelfth International Conference on Learning Representaatinss. 2023.
[14] Chen-Hsuan Lin et al. "Magic3D: High-Resolution Text-to-3D Content Creation". In: 2023 IEEE/CVF Conference
Computer Vision and Pattern Recognition (CVPR). IEEE, June 2023. DOI: 10. 1109/ cvpr52729. 2023. 00037.
URL: http: / /x doi. org/ 10. 1109/
cvpr52729. 2023..22220 .00037.
[15] Junshu Tang et al. "Make-It-3D. High-Fidelity 3D Creation from A Single Image with Diffu- sion Prior". In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Oct. 2023, DOI: 10. 1109 / iccv51070. 2023. 02086. URL: http: //xx doi org/ 10
1109/iccv51070. 2023. 02086.
[16] Xin Yu et al. "Text-to-3D with Classifier Score Distillation". In: The Twelfth International Conference on Learning Representations. 2024.
[17] Jonathan Ho and Tim Salimans. "Classifier-Fie Diffusion Guidance". In: NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications. 2021.

---

[18] Shanchuan Lin et al. "Common Diffusion Noise Schedules and Sample Steps are Flawed". In: 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE. Jan. 2024. DOT: 10. 1109 2009//and /0/////7777111110 2024. 00532. URT.: http: //txx doi. org/oogg//// 10. 1109/
wacv57701 20222022400 00532.. [19] Ruoshi Liu et al. "Zero-I-to-3: Zero-shot One Image to 3D Object". In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Oct. 2023. DOI: 10. 1109/ iccv51070. 2023. 00853. URL: http: : / dx doi org/ / 10. 1109 iccv51070. 2023. 00853.
[20] Yichun Shi al. "MVDream: Multi-view Diffusion for 3D Generation". In: The Twelfth International Conference on Learning Representations: 2024.
[21] Luke KeriaiK et al. "IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation". In: arXiv:2402. 0.422288822222 (2024).
[22] Andreas Blattmann et al. "Stable video diffusion: Scaling latent video diffusion models to large datasets". In: arXiv:2311.15127 (2023).
[23] Ayush Tewari al. "Diffusion with forward models: Solving stochastic inverse problems without direct supervision". In: Advances in Neural Information Processing Systems 36 (2024).
[24] Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. "Viewset diffusion:(O-) image-conditioned 3d generative models from 2d data". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023, pp. 8863-8873.
[25] Peihao Wang et al. "Steindreamer: Variance reduction for text-to-3d score distillation via stein identity". In: arXiv:2401 2000:2000.00044 (2023).
[26] Geoffrey Roeder, Yuhuai Wu, and David K Duvenaud. "Sticking the landing: Simple, lower-
variance gradient estimators for variational inference". In: Advances in Neural Information Processing Systems 30 (2017).
[27] Boshi Tang et al. "Stable Score Distillation for High-Quality 3D Generation". In: arXiv preprint D.Xiv:33/2009300 (2023).
[28] Yiwen Chen et al. "TT3D: Improved Text-to-3D Generation with Explicit View Synthesis". In: Proceedings the AAAI Conference on Artificial Intelligence 38.2 (Mar. 2024), pp. 1237-1244. ISSN: 2159-5399. 1609/aaai 23812.27886
DOI: 10. 1609/ aaai. v3812. 27886.
URL: http: : //dx//dxxp/ doi org/ 10.
[29] Yixun Liang et al. "LucidDreamer: Towards High-Fidelln Text-to-3D Generation via Interval Score Matching", In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2024, pp. 6517 DOr: 10. 1119/ /000///pr52733.... 2024. 00623.0020000 URL.:
http://dx.. doi. org/ 10. 1109/ cvprs273333333.
.0623..
[30] Jonathan Ho, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models". In: Advances in neural information processing systems 33 (2020), pp. 6840-6851.
[31] Jascha Sohl-Dicksted et al. "Deep unsupervised learning using nonequilibrium thermodynam-
ics". In: International conference on machine learning. PMLR. 2015, pp. 2256-2265. [32] Yang Song et al. "Score- Based Generative Modeling through Stochastic Differential Equa- tions". In: International Conference on Learning Representations. 2021.
[33] Ron Mokady et al. "Null-text Inversion for Editing Real Images using Guided Diffusion Models". In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2023. DO1: 10. 1109/ 2.099/vpr527299... 2023. 00585. URL: http: //dx. doi.
org/10. / cvpr52729. 2023. 00588..
[34] Daiki Miyake et al. "Negative-promp inversion: Fast image inversion for editing with text- guided diffusion models". In: arXiv:2305. 16807 (2023).
[35] Prafulla Dhariwal and Alexander Nichol. "Diffusion models beat GANs on image synthesis". In: Advances in neural information processing systems 34 (2021), pp. 8780-8794.
[36] Mohammadreza Armandpour al. "Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate Janus problem and beyond". In: arXiv:2304. 04968 (2023).
[37] Jiaming Song, Chenlin Meng, and Stefano Ermon. "Denoising Diffusion Implicit Models", In: International Conference on Learning Representaations. 2021.
[38] Jack Hessel et al. "CLIPScore: A Reference-free Evaluation Metric for Image Captioning". In: P Proceedings of the 202 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 2021. DOI: 10. 11653/v1/2222 wttp:/201.11.m...mmmmmmmmmmm1mmm www.le
595. URL: http: /tw. org/10. w1w53//1//2000 mmmlp-mainn.o 595.
12

---

[39]
Nicki Skafte Detlefsen et al. TorchMetrics Measuring Reproducibility in PyTorch. Feb. 2022. DOI: 10. 21105 11100/jost joss 04101. URL: https // gi github. com cwm com/Lightning- AI / torchmetrics.
[40] Alec Radford et al. "Learning transferable visual models from natural language supervision". In: International Conference on Machine Learning. PMLR. 2021, pp. 8748-870
[41] Jiazheng Xu et al. "Imagereward: Learning and evaluating human preferences for text-to-image generation". In: Advances Neural Information Processing Systems 36 (2024).
[42] Jianyi Wang. Kelvin C.K. Chan, and Chen Change Loy. "Exploring CLIP for Assessing the Look and Feel of Images". In: Proceedings of the AAAI Conference Artificial Intelligence
37.2 (June 2023), pp. 2555-2563.
ISSN: 2159-5399, DOI: 10. 1609 aaai v3712. 25353.
URL: http: doi org/10. 1609/ /aa////aa..... v3312..5353.
[43]
Yuan-Chen Guo al. threestudio: A unified framework for 3D content generation. https:
// //github.. wtt.ccooooeeeeeeeeee httm//hhrestudioopoe praeesstttdoppjjjjcet//oc wtrjject//hrreesuddi... 2023.
[44]
Peihao Wang et al. "Taming Mode Collapse in Score Distillation for Text-to-3D) Generation". In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vol. 36. IEEE, June 2024, pp. 9037-9047. DOI: 10. 1109/ / cvpr52733. 2024. 00863. URL: http:
//dx.doi. org/ 10. 1119/ wwt09//vp52233..... 2024. 00863.
[45] Richard I. Burden and Douglas Faires. "2.2 fixed-point iteration". In: Numerical Analysis
(3rd ed.). PWS Publishers (1985), p. 64.
[46] Harold Hill and Alan Johnston. "The Hollow- Face Illusion: Soject-Specifer Knowledge, General Assumptions or Properties of the Stimulus?" In: Perception 36.2 (Feb. 2007), pp. 199-
223, ISSN: 1468-4233, DOI: 10. 1068/p5523. URL: BRL::ttp:/// //dx:://x..r doi. org/oogg// 10. 1068/p5523.
[47] Rui Chen et al. "Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation". In: 2023 IEEE/CVF International Conference Computer Vision (ICCV). IEEE, Oct. 2023. DOI: 10. 1109 / iccv510700 2023 02033. URL: http:
//dx. doi org. 110.1111000111100100000000000 02033.
13

---

Appendix
A Implementation details.
In this section, we provide implementation details for our algorithm. Figure 12 provides side-by-side comparison of SDS and our algorithm.
Algorithm Dreamfusion (SDS)
Algorithm 2 Ours (SDI)
Input: 1/ E RR parametrized 3D shape
Input: & E IR parametrized 3D shape
C sct of cameras around the 3D shape
set of cameras around the 3D shape
text prompt
text prompt
9: R-
X C + R=x=_ differentiable renderer
9: R x C - R -ifferentiaalee renderer
(0)
R + R=**_ trained diffusion model
10
g + R trained diffusion model
Output: 3D shape of procedure DREAMFUSION())
Output: 3D shape vi of
procedure OURS(Y)
for i in range(n, ilers) do
for i in rango(n_ sittti_tttsseeee do
L < Uniform( (0, 1)
t + 1 innesee
+ Uniform( (C)
Uniform(C)
c Normal (0,
A (a($, c))
(a(t)g(t), c) V1-a(t)
Va(l)g(V, v1- a(t)c VoLsns = o() (6 (fr.
CuLSDS = (t)) (4) (x,, v) -
d2 öv
8g
Backpropagate CsDS
Backpropagate V CSDS
SGD update on 1/:
SGD update v
Figure 12: Comparison of the original SDS algorithm and our proposed changes.
Our implementation uses the following choice of kty
(t) 0.3 a())H where EH ~ (0,
(11)
k,y (a) ddim iitim_iiversion2e.. (z,t) Ht))
A.1 Timesteps.
In eq. (10), the added noise inverts eq. (8) to noise level t. but the denoising step happens to slightly lower time - (f1 and f2 in fig. 4, right). Intuitively, 7 controls the effective step size the denoising (or image improvement) process. To accommodate this, we maintain global time variable t that linearly decays for all the views. Then, on each update step we run DDIM inversion up to / + т, where is small constant. In practice, we did not find the algorithm to be sensitive to this constant and sample T (0, 3. where 1 is a typical step size in DDIM.
30
A.2 Geometry regularization.
The Janus problem is common issue reported across multiple score distillation works [5, 11, 12], wherein the model produces frontal views all around the shape due bias for views dominant in the training data. SDS [5] tackles this problem by augmenting the prompts with information about the view direction. Our method's variance reduction, however, enabled the use of lower CFG values 7.5 (instead of 100 for SDS), and as shown in fig. 3, with smaller CFG values, the diffusion model tends to ignore certain parts of the prompts (like "colored" for 7 < 10 in fig. 3). Thus in our algorithm observe the diffusion model to be less attentive to the prompt augmentation yielding stronger Janus problem compared to SDS; we also see the same behavior for other basclines that reduce CFG. To address this issue, we use Perp-Neg [36] and add a small entropy term ~ (N,00.33 a(t) (eq. (11)) the inverted noise reduce mode- seeking behavior as in [44].
Another common issue Score Distillation is the hollow face illusion [46], which a tendency to create holes in the geometry of visually convex shape. To address this problem, we use normal map extracted from the volume as described in [5] and downscaled low frequencies are penalized. Then, calculate the sine between adjacent normals (from left to
x 8 resolution so that only
14

---

right and top to bottom) and penalize it to be negative. We activate this loss for the first 40% of the training steps with weight Cconvexity 0.1 to break the symmetry.
Appendix H.1 demonstrates examples of typical failure cases, including the Janus problem and the hollow face illusion.
A.3 System details.
We implement our algorithm in threestudio [43] on top of SDS [5]. We Stable Diffusion
2.1 [1] as the diffusion model. For volumetric representation we use InstantNGP [8]. Instead of randomly sampling time in SDS, we maintain global parameter that linearly decays from 1 to 0.2 (lower time values do not have significant contribution). Next, for each step we render 512 512 random view and infer k by running DDIM inversion for int int(10t) steps-i.e. use 10 steps of DDIM inversion for f and linearly decrease it for smaller t. We use NVIDIA A6000
GPUs and run each generation for 10k steps with learning rate of 10-2, which takes approximately wall clock hours per shape generation.
A.4 Prompts used in the quantitative evaluation
"A car made out of sushi"
"A delicious croissant"
"A small saguaro cactus planted in clay pot"
"A plate piled high with chocolate chip cookies"
"A 3D model of adorable cottage with thatched roof"
"A marble bust of a mouse"
"A ripe strawberry"
"A rabbit, animated movie character, high detail 3d mode"
"A stack of pancakes covered in maple syrup"
"An ice cream sundae"
"A baby bunny sitting on top of stack of pancakes"
"Baby dragon hatching out of a stone egg" iguana holding balloon" "A blue tulip"
"A cauldron full gold coins" "Bagel filled with cream cheese and lox" "A plush dragon toy"
"A ceramic lion"
"Tower Bridge made out gingerbread and candy"
"A pomeranian an dog"
"A DSLR photo Cthulhu"
"Pumpkin head zombie, skinny, highly detailed, photorealistic" "A shell"
" 'An astronaut is riding horse"
"Robotic bee, high detail"
"A sea turtle"
"A tarantula, highly detailed"
"A DSLR photograph of hamburger"
"A DSLR photo of a soccer ball"
"A DSLR photo of white fluffy cat"
"A DSLR photo of a an old man"
semaisssncee-syle oil painting of a queen" "DSLR photograph of a baby racoon holding a hamburger, 80mm" "Photograph of black leather backpack"
"A DSLR photo of a freshly baked round loaf of sourdough bread"
"A DSLR photo a decorated cupcake with sparkling sugar on top"
"A DSLR photo of a dex-covered peach sitting in soft morning light"
"A photograph a policeman"
"A photograph of ninja"
"A photograph a knight"
An astronaut"
15

---

"A photograph of a firefighter" Viking panda with an axe"
B Comparison with Interval Score Matching
In this section, we extensively compare our algorithm with Interval Score Matching (ISM) [29].
Theoretical assumptions. ISM empirically observes that "pseudo-GT" images used to guide Score Distillation Sampling (SDS) are sensitive to their input and that the single-step generation of pseudo- GT yields over-smoothing. From these observations, starting with SDS guidance, ISM adds DDIM inversion and multi-step generation to empirically improve the stability of the guidance. In contrast, our work starts with 2D diffusion sampling to re-derive acore-distillation guidance and motivate improvements. That is, our work formally connects SDS well-justified techniques in 2D sampling.
DDIM inversion. In ISM, the empirically motivated DDIM inversion is at the basis of the derivation of the final update rule. We suggest a general form of the noise term eq. (8), for which DDIM imversion is just one possible solution. Our theoretical insights are agnostic to particular algorithms of root- finding, which makes possible use more efficient solutions in future research (e.g. train diffusion models as invertible maps to sample noise faster).
Guidance term. The update rules provided in our eq. (10) and ISM's eq.17 have two main differences:
1. Full interval noise. Assuming DDIM inversion finds perfect root of our stationary equation, ISM's update rule can take a similar form to ours eq. (10) (interval noise is equal to x if eq. (8) is satisfied). However, as shown fig. 9, DDIM inversion does not find a perfect root, and thus the two forms are not equivalent. Our theory shows that the full noise term is more accurate. We also show the effect of choosing one term vs. another in fig. 14.
2. Conditional vs. unconditional inversion. Our derivation hints that the roots of eq. (8) are prompt-dependent, motivating our use of conditional DDIM inversion (not used ISM). Our fig. 10 shows how unconditional inversion yields over-saturation.
"A DSLR photo of hamburger"
"Viking axe, fantasy, weapon, blender, 8k, HDR""
"a DSLR photo wearing armor"
"a DSLR photo of bagel filled with cream cheese and lox"
"an icc cream sundae"
"a DSLR photo of freshly baked round loaf of sourdough bread"
"A 3D model of adoruble cottage with a thatched roof"
SCDs (quns)
ISM
SDI (ISM (ISMbaa base)
Figure 13: Comparison of SDI (ours, top) ISM (middle), and SDI in ISM's code base (ours, bottom). To eliminate factors such as 3D representation and initialization, we re-implement our algorithm in the official codebase of ISM. See fig. 14 for the effect of each change made.
Practical Results. To control for different design choices (Gaussian Splattings in ISM vs. NeRF in ours, etc.) we re-implemented our algorithm in the code base of ISM with only the minimal changes discussed in the previous section.
16

---

ISM
full range
+ time annealing
+ conditional
+ SDI guidance (ours)
w/o conditional inversion
w/o time annealing
w/ full range
inversion
Ablation of changes made to the code base of Interval Score Matching (ISM). Prompt "A DSLR photo of a hamburger". We start with the official implementation of ISM and add only four changes to match ours (all the hyper-pane parameters remain tuned for ISM): 1. instead of sampling t from U(0.02, 0.5) we sample from U(O.1, 0.98); introduce linear time annealing: 3. use conditional DDIM inversion with negative guidance; 4. use SDI guidance, i.e. second term in the guidance difference is the full noise added
Figure 14:
Figure 13 provides a qualitative comparison using the prompts and settings in ISM's code. Figure 14 shows the effect of each change made to ISM guidance. Table 2 the quantitative comparison (ISM code base for both).
Table 2:
Quantitative comparisons between SDI (ours) and ISM.
Method
CLIP Score (f)
CLIP IQA (%. f)
Imageleward (1)
Time
VRAM
"quality"
"sharpness"
"real"
ISM. 5k steps
28,60+2.03
0.85=0.02
0.98=0.01 0.9920.00
0.98+0.01
-0.52=00.88
45min
15.4GB
SDI (ours), 5k steps
28.47*1.29 1,29
0.8820.03
0.98+0.01
-0.30:20.2
43min
15.4GB
Summary. We bridge the gap between experimentally-basee score distillation techniques and theoretically-justiiied 2D sampling. Both SDS and ISM can be seen as different approaches to finding roots of eq. (8). This theoretical insight allows us to modify both ISM and SDS, reducing over-saturation for the first and improving general quality of the second.
C Diversity of the generations
Next, explore the diversity of the results generated by our method. Figure 15 depicts generated shapes for different seeds and prompts. Generations with our method exhibit certain degree of diversity in local details, but are mostly the same a coarse level. This is similar the reported results in other works on score distillation [5, 11j. We plan to address this problem in future research.
seed 0
seed
seod 2
seed
seed 4
seed 5
seed 6
seed
Figure 15: Examples of generations with SDI (ours) for different seed values. Prompts are "Pumpkin head zombie, skinny, highly detailed, photorealistic" (top row), "A highly-detailed sandcastle" (middle row), "An ice cream sundae" (bottom row).
17

---

Additionally, we noticed minimal diversity in human generations. For example, fig. 16 demonstrates generated shapes for prompts related different professions.
B
Figure 16: Examples of limited diversity across different human-related prompts: "a photo of firefighter", "a photo of a policeman", "a photo of a soldier."
D Visual comparison of noise patterns
In this section, provide additional examples of the noise samples obtained with different ni. As can be seen from fig. 17, optimization with SGD almost does not change the noise pattern compared to random sampling. Fixed-poin iterations drift in notable but counterproductive direction, and the single-step prediction from the obtained noisy images (depicted on the right side) degrades. DDIM inversion with Yav 7.5 and "iav 7.5 produces similar noise patterns located around the "meaningful" parts of the image; however, "inv = 775 results in noise patterns that are much more compatible with the single-step denoising procedure used in SDS, leading much more accurate details appearing for earlier I.
Random, Resampled
Random, Fixed
Fixed Point Iteration
SGD optimization
DDIM inversion CFG 7.5
DDIM Inversion CFG -7.5
1.0 x())
0.8
0.6
0.4
0.2
0.8
0.6
0.4
0.2
zi (()
Figure 17:
Examples of noisy images (left part) obtained for different choices of kiy On the right part, single step denoised images are presented for the corresponding noisy images.
18

---

E CFG for inversion in 2D
In this section, we provide additional experiments testing the DDIM inversion strategy and study the performance of different combinations of the CFG value on the inversion stage (Niw) and on the forward pass (fiwa). In particular, in fig. 18 we first generate an image with 30 steps of DDIM (left-most image) and
7.5. Then we run DDIM in the opposite direction of with "inv all the way Finally, we use the obtained noise sample as an initialization for a new DDIM inference with 7/wa Ideally, the final image will correspond to the input of the inversion algorithm.
Figure 18 shows that when inverting the DDIM trajectory starting with clean image, the best inversion strategy uses "in = O and then regenerates the image via DDIM with "iwdd = 0. [34] suggests this approach, which inverts the computation almost perfectly. Other approaches introduce bias and only approximate the image on the forward pass.
As we have seen in section 6.2, in 3D, the story is different: Tiav Vod 0 fails to generate any meaningful shape. We explain this behavior by the fact that in our 2D experiment, the original image contains class information; inverting with Vimv 0 encodes this information into the noise and thus does not need additional CFG during reconstruction. In 3D generation, however, the initial NeRF does not contain class information and needs bigger CFG on the forward pass. Additionally, fig. 18 demonstrates the inversion quality of an entire DDIM trajectory, whereas eq. (10) we are interested in inverting single-step denoising only. what might explain the fact that seemingly bad inversion quality of "ffdd = -Yinv 7.5 yields much better results in 3D.
Input y y= 7.5
Vinv 7.5
Tinv 7.5
Vinv 7.5
Vivv
Vivv
lfwd 7.5
"fwwd 7.5
Yiwd 7.5
Ifwdd 7.5
Vwwd 0
30 its
30 its
300 its
30 its
30 its
30 its
Comparison of different DDIM inversion strategies in 2D. Here, compare the quality of inversion, i.e. the forward pass is the entire DDIM trajectory.
Figure 18:
F ODE derivation
Our derivation in section 4 manipulates the time-stepping procedure of DDIM, but similar argument applies to the ODE version of the method. In particular, one can obtain an alternative form of eq. (10) by reparametrizing the ODE eq. (1). To do the change of variables in eq. (1) eq. (5) we need to
differentiate with respect to I and evaluate dz(() Direct differentiation gives us:
de
dru(t) dz(t) (1)
3(t)
do(t) ott) d dt
x(t) dt (ot1))++ +1
(1)
dt dt cg
(12)
Vo(t)++1
+1 :
Solving for dz(() dt and merging with (1), we get:
(1) x(1) dl(())
dro(!) (t)
x(1)
do(l)
d
z(()
Vo(t)² +
dt dt
c
+1't))+1' Vo(()² +1 y
dt
+ o(t)
dt
(Vo(t)2 + 1
dt
= (t) d
I(t)
(1)
(o + 3
dt
dt
d
sa(() + +L())
-((()
(t)
=
dl
Vo(12 +1
dr!
(xo(t)).
(13)
dt
19

---

Or, expressed in other scaling parameters:
dx -a d dt.
(=o(1))+ (()))+((()(()))))))))(
(14)
dt
This expression gives us the DDIM's ODE re-parametrized for the dynamics of (t).). This ODE is velocity field that projects starting image to a conditional distribution learned by the diffusion model. Discretizing this equation leads to eq. (10).
G
Additional Intuition: Out-of-Distribution Correction
Here we provide an intuition why it is so important regress the noise instead randomly sampling it. In particular, we can look into the data that was used to train eD For each 4, the model has seen only clean images with the corresponding amount of noise (()..
Consider a noisy or blurry image &g. Our goal is to use our updated ODE to improve and make look more realistic. For & close 1, the amount of noise is so big. that for any image dn its noisy version ân + (t)e looks indistinguishable from the training data of the denoiser. This explains why successfully diffusion models correct -uttof-diitributtion images.
For small however, the noise levels might be not enough to hide the artifacts of f.. so the image becomes out-of cit-off-ditributton and the predictions of the model become unreliable. We hypothesize, that instead of randomly sampling like Dreamfusion, we can find such a noise sample, that under the same noise level / the artifacts of ên will be hidden and it will be in-distribution of the denoiser. This effectively increases the ranges of where the predictions of the model are reliable and thus improves the generation quality. This intuition is illustrated in fig. 19.
t
0O
5o
Out of distibution
0
In distribution
1
Figure 19: Schematic for distr--distriuution interpretation.
H
Additional Results
Lastly, we discuss failure cases and showcase additional generations using our methods, along with more baseline comparisons.
H.1 Failure Cases
We provide illustrations of typical failure cases of our algorithm in Figures 20 to 23.
"a photograph of a firefighter"
Figure 20: Content Drift: due limited communication between the frames sometimes we can observe that certain properties "drift" around the shape. In this particular example the number on the fireman's hat is different from all the views.
20

---

"an astronaut"
Figure 21: Hollow Face Illusion: shapes might be generated with concave hole in the peometrv. while the rendering annears to be convex.
"A portrait photograph of an orc"
Figure 22: Janus Problem: note how the has two faces that merge on the second to the right
"a Viking panda with an axe"
Figure 23: Diffusion Anomalies: performance of our algorithm is limited by the quality of the underlying diffusion model. In this case the generated shape contains a typical failure case of multiple limbs.
H.2
Additional generations
Figures 24 to 27 provide additional generations produced using our method.
H.3
Additional comparison with baselines
We provide additional qualitative comparisons with other score distillation algorithms in figs. 28 30. We compare with Dreamfusion [5], Magic3D [14), Fantasia3D [47], Stable Score Distillation
(StableSD [27], Classifier Score Distillation (ClassifierSD [16]), Noise-Free Score Distillation
(NFSD [12]), ProlificDreamer or Variational Score Distillation (VSD (11)), Interval Score Matching
(ISM) [29], and HiFA [13].
21

---

"a photograph of knight"
"a ripe strawberry"
"a ceramic lion"
"a photograph of a ninja"
"a DSLR photo of a beautifully decorated cupcake with sparkling sugar on top"
"Tower Bridge made out of gingerbread and candy"
Figure 24: Additional generations from our method,
22

---

"robotic bee, high detail"
"a tank made of chocolate"
"a photograph of a policeman"
"cyberpunk style panda"
"renaissance-style oil painting of a queen"
"a DSLR photo of a dew-covered peach sitting in soft morning light"
Figure 25: Additional generations from our method,
23

---

"pumpkin head zombie, skinny, highly detailed, photorealistic"
"A 3D model of an adorable cottage with a thatched roof"
"a raccoon astronaut holding his helmet"
"a chimpanzee dressed like Henry VIII king of England"
'renaissance-style oil painting of a poor man"
"a tiger dressed as a doctor"
Figure 26: Additional generations from our method,
24

---

"a photograph nurse"
"a DSLR photo of shiny red apple with droplets of water on its surface"
"a lion reading the newspaper"
"photo of luxurious sailing boat"'
"a DSLR photo of a freshly baked round loaf of sourdough bread"
"an ice cream sundae" *
Figure 27: Additional generations from our method,
25

---

Dreamfusion Magic3D
Fantasia3D
StableSD
ClassifterSD
NESD
VSD
ISM
HiFA
Ours
"An ice cream sundae"
Dreamfusion Magic3D
Fantasia3D
StableSD
ClassifierSD
NFSD
VSD
ISM
HiFA
Ours
"A plate piled high with chocolate chip cookies"
Dreamfusion Magic3D
Fantasia3D
StableSD
NFSD
VSD
HIFA
Ours
"A car made out of sushi"
Dreamfusion Magic3D
Fantasia3D ClassifierSD NFSD
ISM
HIFA
Ours
"A ripe strawberry"
Drcamfusion Magic3D
ClassifierSD NFSD
VSD
HIFA
Ours
"A baby bunny sitting on top stack of pancakes"
Dreamfusion
Magic3D
ClassifierSD
NFSD
HiFA
Ours
"Bagel filled with cream cheese and lox" Fantasia3D
Dreamfusion
Magic3D
NFSD
VSD
Ours
"A marble bust a mouse"
Figure 28: Additional comparisons.
26

---

Dreamfusion
M
ClassifierSD
NFSD
VSD
Ours
"A small saguaro cactus planted in a clay pot"
Dreamfusion
Magic3D
StableSD
NFSD
VSD
Ours
"A rabbit, animated movie character, high detail 3d mode"
Dreamfusion
Magic3D
Fantasia3D
ClassifierSD
NFSD
Ours
"A stack of pancakes covered in maple syrup"
Dreamfusion
Fantasia3D
NFSD
VSD
HIFA
Ours
"A delicious croissant"
Drcamfusion
NFSD
ProlificDreamer
HiFA
Ours
"Michelangelo style statue of dog reading news on a cellphone"
Figure 29: Additional comparisons.
27

---

Dreamfusion
Magic3D
NFSD
Ours
"Tower Bridge made out of gingerbread and candy"
"An iguana holding balloon"
"A ceramic lion"
"Baby dragon hatching out of a stone egg"
Figure 30: Additional comparisons.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state the main theoretical and practical contributions in both abstract and introduction. They are theoretically justified section and then experimentally confirmed in section 6.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: Limitations are explicitly discussed in the conclusion and explored in the experiments in appendix H.1.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, c.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to- to-tett system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: While we do not have explicit theorems, all analysis clearly provides any needed assumptions.
Guidelines:
*
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
29

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to
provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: Our method is described fully and relevant experimental details clearly provided. Generation examples are not cherry picked unless explicitly stated. Additionally, we provide the code of our algorithm in supplementary and plan make it public upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
example, if the contribution
While NeurlPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily architecture clearly and fully.
model architecture, the paper should describe the
(c) If the contribution is be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with open-source dataset instructions for how to construct the dataset).
model (e.g., a large language model), then there should either
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users). but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code is included in the supplementary material. No data needed as we generate all results from prompting preexisting diffusion model which we specify in the experiment section.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines fattps://nipt ctt.ppsicccpbppc///cccc guides/Code Cideee//ooeSubbiii ssi imPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
30

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines / guides/ Suts/CCoSSSubbisssseee aassonoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce experimental results for the new proposed method and basclines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We discuss experimental settings and implementation details fully.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to level of detail that is
necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material. Experiment Statistical Significance
Question: Does the paper report eπтor bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the error bars where applicable, i.c., in ablations (section 6.2) and quantitative results.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). should be clear whether the error bar is the standard deviation or the standard error of mean.
is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: We report the wall-clock time and the VRAM used to generations in table 1 as well as the hardware used in appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
31

---

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e-g.. preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttes::///eripppp cttpppbccc/taaccccssuu.eeeellll.
Answer: [Yes]
Justification: While some of our generations involve images of human forms, we used Bing reverse image search (https:/ / /ww. bing com/visual to ensure that they do not correspond to any real individual.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: From the main paper- Our work extends existing 2D diffusion models to 3D settings. As such, our work could marginally improve the ability of bad actors to generate deepfakes or to create 3D assets corresponding to real humans interact with in virtual environments or games. While this is very real problem in the industry at this time, in our view, our work represents only a marginal (if any) increase in such risks, as highly convincing deepfake tools are already widely available. We discuss this in the conclusion of the paper. Guidelines:
The answer NA means that there is no societal impact of the work performed. If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considcrations (c.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied par- ticular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: As a procedure for sampling from preexisting diffusion models, our method, to our knowledge, should need no additional safeguards beyond those already appropriate for the preexisting diffusion models.
Guidelines:
32

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The code for implementing our method is generated by ourselves on top of threestudio [43], any preexisting diffusion models used are properly cited and the relevant licenses followed.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should
be provided. For popular datasets, paperswi snperswithoee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's
creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code we released documented.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their sub-
missions via structured templates. This includes details about training, license, limitations. etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper
include the full text of instructions given participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: No such experiments were conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
33

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No such experiments were conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
34


Reviews:
[{"review_id": "viW3z7PI0q", "reviewer": "Reviewer_r6QS", "summary": "This paper connects score distillation sampling (SDS) to a DDIM sampling process. The proposed method Score Distillation via Inversion (SDI) replaces the original random noise in SDS with DDIM inversion, and show significantly improved quality compared to SDS and other state-of-the-art prior methods.", "questions": "Does the proposed method remedy Janus problem, compared to the baseline SDS method? (If yes, why)\nAlso please check the weakness section.", "limitations": "The authors discussed limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The connection from SDS to DDIM sampling is interesting. The motivation is also clear that it considers the change of the \"sample prediction\" variable to guide the 3D generation process. The method is supported by well derived theory (for 2D) and the DDIM inversion is a clever solution.\n2. The results generally look good. It shows significant improvement over its baseline method SDS, and is competitive to recent state-of-the-art text-to-3D methods. The experiments are thorough.\n3. The proposed method and results could provide insights for future research in the important direction of text-to-3D distillation.", "weaknesses": "1. Overall, the visual results seem to be having a little bit \"gray\" color/style shift. Is this due to the approximation or some bias in the theory? Is there any hypothesis for theoretical / practical reasons?\n2. The ddim inversion and approximation could introduce additional computation cost and training noise.\n3. In \"ISM as a special case\", it seems the main difference of the proposed method and ISM is having the additional text condition. Is there any other key differences in method implementation or theory? More details about the contribution of this work based on ISM could be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank r6QS for the helpful review. We will address their comments in the final version and clarify some points below:\n# Gray color shift\nWe agree that results in the main paper might seem gray or sometimes have a red/green tone. We explain this by the dark background color that is generated in the NeRF. Our algorithm is implemented on top of threestudio’s SDS, where apart from the object-level NeRF, the shapes are also allowed to have a background. Typically the generation converges to darker tones of the background, which biases the diffusion process to reflect illumination in the rendered 3D shapes. For visualization in the original submission, we simply cropped the object from the final renderings. When on a white background, this removes the illumination context and makes the shapes look grayer. \n\n\n\nTo further illustrate this we provide renderings with their original background in Fig.7 of the rebuttal PDF. Additionally, Fig. 1 (bottom row) of the rebuttal PDF shows that when our algorithm is reimplemented in the ISM’s code base, where the background is fixed to be white, our final renderings have brighter, white colors reflecting the global illumination.\n\nIn summary, **the diffusion model bakes in the background illumination into the texture of the shape.** \nOne possible solution that was suggested in prior work consists of randomly changing the background colors to avoid overfitting to a single one. In this case, however, we observed a decrease in the quality of the renderings.\n\n\n# Computational Cost\nWe agree that, unfortunately, DDIM inversion requires additional forward passes with the denoiser, which increases computational cost. However, we would like to clarify a few points:\n\n1. Despite the additional cost required for obtaining the noise sample, our algorithm is typically more stable and requires fewer steps to converge than other state-of-the-art methods, all while having similar quality in the results. The time per generation in Table 1 shows that **our algorithm is still 2-3x times faster than state-of-the-art** since it takes fewer steps.\n\n2. Our ablation in Fig.11 demonstrates that 10 steps are enough for DDIM inversion to obtain good-quality 3D generations. In practice, one NeRF update is so slow that after the 10 additional inferences with the denoiser per optimization step we see only a 2x slow-down. This effect will be more noticeable in future work involving Gaussian Splatting, where 3D shape update is not a bottleneck anymore.\n# Differences with ISM\nFor a detailed comparison of our algorithm and ISM, please refer to the general response above and to the rebuttal PDF attached to the response.\n\n# Janus problem\nThank you for the question. Without additional regularization, our algorithm has a stronger tendency towards the Janus problem compared to SDS. SDS uses prompt augmentation to avoid this problem, adding phrases like “front view” and “back view.” As in Appendix A2, our algorithm’s stronger bias toward the Janus problem can be explained by the property of the diffusion models to ignore parts of the prompt when CFG is not high enough (see Fig.3 in the main paper). The proposed improvements of finding a better noise term allow us to reduce the CFG values from 100 in SDS to the typical 7.5, which leads to a weaker view-dependent prompt augmentation.\n\nTo additionally prevent the Janus problem, we add entropy as in [1], and orthogonalize the augmented prompts as in [2]. In rare cases, our algorithm still might produce multiple faces in one generation. This limitation is reported and illustrated in the appendix E1.\n\n[1] Peihao Wang et al. “Taming Mode Collapse in Score Distillation for Text-to-3D Generation”. In: arXiv:2401.00909 (2023).\n\n[2] Mohammadreza Armandpour et al. “Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate Janus problem and beyond”. In: arXiv:2304.04968 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZwgRLkws02", "reviewer": "Reviewer_4KTy", "summary": "The paper introduces an effective SDS modification that replaces random noise samples in the SDS objective with those obtained with DDIM inversion. The proposed technique enhances text-to-3D generation, outperforming SDS and being comparable to more sophisticated methods.", "questions": "* Please address the questions and concerns raised in \"Weaknesses\".\n\n* In Fig.5, the intermediate $x_t$ do not have noise in the background. DDIM inversion must follow the same marginal distributions as the forward process, but the presented $x_t$ do not. We observed the similar effect for images with monotonic regions if inversion is applied directly to a clean image. We believe such images are likely OOD for the pretrained DM, and injecting slight noise with $\\sigma_{min}$ to an image fixed the problem. Do the authors have any thoughts on this? Would the solution above work for SDI and how does it affect its performance?", "limitations": "The authors addressed the limitations and potential negative social impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well organized and provides valuable intuitions, illustrations, and discussions over reading. The additional discussion in Appendix E is also beneficial.\n\n* The proposed method is simple, reasonable and well motivated. The derivation is clear.\n\n* The experimental results are promising, and the ablation study in Sec 6.2 is highly valuable and interesting.", "weaknesses": "* Inversion with negative guidance is surprising, and its rationale remains somewhat unclear. A more detailed investigation and justification would be useful.\n\n* The idea of using DDIM inversion is very similar to ISM[1]. The primary distinction is guided vs unguided inversion. Although the authors justify the guided inversion with negative guidance in Fig. 10, a thorough discussion and detailed comparisons are essential.\n\n* The quantitative evaluation relies on CLIP-IQA, which does not seem quite suitable for 3D generation evaluation. Conducting a user study would be highly valuable. If it is unavailable, some recent works [2,3] aim to propose automated evaluation metrics. For example, would it be reasonable to evaluate ImageReward[4] following [2]?\n\n* The quantitative results miss NFSD and the qualitative comparisons are presented only for two prompts. Could the authors add NFSD to Tab.1 and provide more visual comparisons with all methods?\n\n* There is no evaluation of diversity under a given prompt. It would be useful to include a few generated results for different seeds.\n\nTo sum up, I am generally optimistic about the submission but have some concerns regarding the evaluation and close connection with [1], as discussed above.\n\n[1] Liang et al. LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching.\n\n[2] He et al. T3Bench: Benchmarking Current Progress in Text-to-3D Generation.\n\n[3] Wu et al. GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation.\n\n[4] Xu et al. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful and constructive review. Below we address the points raised in the review:\n# Inversion with negative CFG\nThank you for pointing this out. We agree that adding this intuition to the main paper will benefit the clarity. Below we support the intuition described in the general response with theoretical and empirical analysis.\n\n## Experimental analysis of negative CFG\nIn the general responce, we base our intuition on the fact that negative CFG attracts generation to the mean of the training dataset. To demonstrate this, we perform a simple experiment of 2D generation with DDIM for different prompts and CFG values of +/-7.5 for each (Fig.3 of the rebuttal PDF). Instead of generating images with opposite prompts, negative CFG directs images to the same class (real estate). This happens because the “real estate” happen to be in the middle of the embedding space, where the process is attracted to. \n## Theoretical justification\nThe effect of attraction to the mean can also be shown more formally. Let’s denote:\n\n$X-\\text{full dataset},X_y-\\text{prompt conditioned dataset}$\n\nFrom the optimal denoiser perspective and notations from [1]:\n\n$\\hat{\\epsilon}\\_\\theta^t(x_t,\\varnothing)=\\sum_{i\\in X}w_i(x_t)x_i$\n\n$\\hat{\\epsilon}\\_\\theta^t(x_t,y)=\\sum_{j\\in X_y}w_j(x_t)x_j$\n\nNow, rewriting the CFG definition:\n\n$\\epsilon_\\theta^t(x_t,y)=(1-\\gamma)\\sum_{i\\in X}w_i(x_t)x_i+\\gamma \\sum_{j\\in X_y}w_j(x_t)x_j=$\n\n$=(1-\\gamma)[\\sum_{i\\in X/X_y}w_i(x_t)x_i+\\sum_{k\\in X_y}w_k(x_t)x_k] + \\sum_{j\\in X_y}w_j(x_t)x_j=$\n\n$=(1-\\gamma)\\sum_{i\\in X/X_y}w_i(x_t)x_i+\\sum_{j\\in X_y}w_j(x_t)x_j$\n\n\nIn the final formula, image generation is attracted to the images in the dataset relevant to the prompt (independently from CFG), and also is repulsed or attracted to the mean of the rest of the dataset, depending on the sign of the CFG. Consequently, **negative CFG values attract the image to the mean of the dataset and repulse it from the mean on DDIM inversion**.\n\nAs mentioned in the general response, our intuition is that *the repulsion from the mean of the dataset in the case of negative CFG serves as a regularization for the noise term and allows to compensate for the bad initialization*.\n# Comparison with ISM\n Please see the general response above for a detailed comparison between our algorithm and ISM.\n# Additional Metrics\nThank you for the useful suggestion. We agree that reporting a metric that reflects human preference will make the paper stronger. We adopted the suggested ImageReward. We are using the official implementation and report the rewards directly. Following the same protocol as in Table 1 of the main paper we average the metric across 50 views for each of the 43 prompts. We report the obtained metrics below and plan to augment Table 1 with it.\n|Method|ImageReward ↑|\n|-|-|\n|SDS, 10k steps|-1.51±0.83|\n|SJC, 10k steps|-1.76±0.51|\n|VSD, 25k steps|-1.17±0.58|\n|ESD, 25k steps|-1.20±0.64|\n|HIFA, 25k steps|**-1.16±0.69**|\n|SDI, 10k steps|-1.18±0.59|\n# Additional comparisons\n## NFSD\nWhile we compare our method with NFSD qualitatively, we omit quantitative comparison. The official implementation of NFSD is not yet available and we were not able to reproduce its results. We will be happy to add the numerical comparison to Table 1 as soon as an official implementation becomes publicly available.\n## Additional qualitative comparisons \nIn response to more qualitative comparisons apart from the two prompts in the main paper, we would like to kindly point out the numerous qualitative comparisons in the appendix. For this, we follow a protocol adopted across the field: for each baseline, we provide renderings reported in the corresponding papers. To the best of our knowledge, we have reported all prompts and images reported in the corresponding baselines. If considered necessary, we will add visual comparisons on all baselines, using open-source implementations.\n# Evaluation of diversity\nWe agree that additional evaluation of diversity will enhance our paper. Please find examples of generations for different seeds and prompts with our method in Fig.4 of the rebuttal PDF. Our generations exhibit a certain degree of diversity but are mostly the same at a coarse level. We did not notice lower diversity compared to SDS[2] or VSD[3].\n\nAdditionally, Fig.6 of the rebuttal PDF illustrates interesting behavior, where different human-related prompts generate similar faces. We are not sure if this is a limitation of the underlying diffusion model or our distillation algorithm and plan to address this in future work.\n\n# Noise in the background of x_t\nWe agree with the intuition about initial images being out-of-distribution. Indeed, it is rare to find perfectly monotonic backgrounds in natural images. It is unclear if the intermediate $x_t$ is unlikely under the marginal distributions or $x_t$ is close to a mode (in the same way as a completely uniform image has the highest probability density in the i.i.d Gaussian). \n\nIn our experiments, however, we did not notice any issues caused by this behavior. Moreover, as we mention in Appendix A.2, we adopt a technique from [4] to fight the Janus problem: we increase the entropy of the process by mixing a small amount of noise during inversion. This helps increase the flexibility of the model to generate non-frontal views. At the same time, it removes the monotonic regions in the noise samples.\n\n[1] Permenter, Frank, and Chenyang Yuan. \"Interpreting and Improving Diffusion Models from an Optimization Perspective.\" arXiv preprint arXiv:2306.04848 (2023).\n[2] Poole, Ben, et al. \"Dreamfusion: Text-to-3d using 2d diffusion.\" arXiv preprint arXiv:2209.14988 (2022).\n\n[3] Wang, Zhengyi, et al. \"Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[4] Peihao Wang et al. “Taming Mode Collapse in Score Distillation for Text-to-3D Generation”. In: arXiv:2401.00909 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ieSRS8AJVU", "reviewer": "Reviewer_ZvgL", "summary": "This paper introduces a novel algorithm called Score Distillation via Inversion (SDI) that enhances the quality of 3D shape generation. By inverting Denoising Diffusion Implicit Models (DDIM) at each step and incorporating the initial noise into the estimated score, SDI addresses the over-smoothing and detail loss issues associated with traditional Score Distillation Sampling (SDS) methods in 3D generation. Experimental results demonstrate that SDI significantly improves the quality of 3D shapes, closely matching the quality of 2D image generation, without the need for additional neural network training or multi-view supervision. The work provides valuable insights into understanding and improving 3D asset generation with diffusion models.", "questions": "Please refer to the weaknesses part.", "limitations": "Please refer to the weaknesses part.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a theoretical analysis that links Score Distillation Sampling(SDS) with Denoising Diffusion Implicit Models(DDIM), offering a deeper understanding of the discrepancies between 2D and 3D generation process, which is different from the experimental enhancement made by most of the previous methods.\n2. The modification to the existing SDS method is straightforward and does not require training additional networks or multi-view supervision. Furthermore, SDI significantly improves the quality of 3D shape generation, which makes it a practical solution for enhancing 3D generation.\n3. This paper provides thorough analysis and present a set of convincing ablation experiments.", "weaknesses": "1.\tAccording to the authors’ theory and hypothesis, the better the noise term is Eq. (8) is estimated, the better the 3D generation quality is improved. However, in the ablations about the choice of k(x), SGD optimization doesn’t show significant improvement. Also, this phenomenon can be observed from Figure.10, in which green curve and purple curve present lowest and comparable MSE with orange curve, while the orange curve get the best result. I would appreciate it if the authors can provide some explanations.\n2.    One of the paper's key perspective is that SDS introduces high variance. I have read another paper[1] dedicated to  minimizing the variance introduced in SDS, and maybe it's also worthwhile to compare with that.\n\n\n\n[1] Tang, B., Wang, J., Wu, Z., & Zhang, L. (2023). Stable score distillation for high-quality 3d generation. ArXiv Preprint ArXiv:2312.09305.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review and recommendations. We provide clarifications below.\n# Numerical error in choices of \\kappa\nThank you for highlighting this effect in the review, we believe a more detailed explanation will indeed benefit the paper. We touch on the intuition around this question in the general response: it is not exactly true that \\kappa with a lower error in eq.8 leads to a better 3D generation. It is true, however, that a more precise solution to eq.8 aligns score distillation more closely with DDIM. Other factors like initialization, view consistency, and DDIM’s generative ability also impact 3D quality. Below we address each of the particular questions in detail.\n\n1. The green configuration in Figure 10 ( $\\gamma\\_\\text{fwd}=1, \\gamma_\\text{inv}=1$) indeed has the lowest error in eq.8. This is consistent with [1, 2], which claim that DDIM inversion accumulates numerical errors in a conditional generation [1] while achieving almost perfect inversion in an unconditional generation [2]. When we use unconditional generation together with unconditional inversion, we succeed at closely matching the generation guidance to a DDIM trajectory. This trajectory, however, is unconditional. Thus the final 3D shape does not correspond to the desired (or any) prompt: exactly what we see in Figure 10.\n\n2. As it was correctly noted, the purple and red configurations in Fig.10 (conditional vs. unconditional inversion) exhibit the same approximation error. In practice, both configurations yield detailed 3D shapes (also Fig.10). However, as described in the general reply (see “Guidance Term”), a combination of unconditional inversion and conditional generation leads to the accumulation of saturation in the final shape. Please also refer to Fig.5 of the rebuttal PDF for the visualization of this effect.\n\n3. Regarding the error of SGD and Random noise in Figure 9, the two lines are mostly within one sigma of each other (please note the translucent areas behind each line signifying standard deviation). The numerical error on the right is computed and averaged across 10 prompts. We did not notice any improvement in 3D generation quality nor in the error induced in eq.8 by performing the stochastic gradient descent on the noise term. The SGD optimization of 10 steps was initialized to a randomly sampled noise image, which might explain the resemblance of the generated shapes to the randomly sampled noise.\n## Comparison with StableSD\nThank you for providing a useful reference. We would like to point out that we already have a qualitative comparison with Stable Score Distillation in the appendix (StableSD in section E3). \n\n\nMathematically, the idea behind this line of work is that the mode-seeking term in score distillation ($\\\\epsilon\\_\\theta^t$) exhibits high variance. Thus the second term ($- \\epsilon$ in SDS or the LoRA in VSD[3]) reduces the variance of the first. A similar idea was also introduced in SteinDreamer [4], where authors numerically analyze variance in SDS and VSD. Both StableSD and SteinDreamer propose to use control variates to minimize the noisiness of the guidance. Please note, that in both of these works, the excessive variance is compensated by subtracting another noise term by finding coefficients that allow for a better correlation.\n \n\nIn our work, however, we explain the second term in SDS not as a control variates, but rather as a projection of the previous step in a re-parametrized DDIM trajectory. Moreover, our variance reduction comes not from using control variates, but from finding a better noise to add to the rendering in the first place. We show that the excessive variance in $\\epsilon\\_\\theta^t(x_t)$ comes from the fact that $x_t$ was obtained using a randomly sampled noise, while our analysis reveals that it should follow a very particular structure (eq.8). By finding a better approximation of the desired noise term we eliminate the root cause of the excessive variance instead of compensating for it later.\n\n\nUnfortunately, the official implementation of Stable Score Distillation is not yet publicly available (nor for SteinDreamer) and we were not able to reproduce their reported results. We will be happy to augment our Table 1 with a quantitative comparison with StableSD as soon as an official implementation gets published.\n\n[1] Ron Mokady et al. “Null-text inversion for editing real images using guided diffusion models”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 6038–6047.\n\n[2] Daiki Miyake et al. “Negative-prompt inversion: Fast image inversion for editing with text guided diffusion models”. In: arXiv:2305.16807 (2023).\n\n[3] Wang, Zhengyi, et al. \"Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[4] Peihao Wang et al. “Steindreamer: Variance reduction for text-to-3d score distillation via stein identity”. In: arXiv:2401.00604 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for the detailed and thoughtful feedback.

We are pleased that they appreciate the theoretical contributions and the novelty of our approach (“*well organized and provides valuable intuitions*” (4KTy), “*insights for future research in the important direction*” (r6QS)). The reviewers found our method to be practical (“*significantly improves the quality of 3D shape generation*” (ZvgL)) and well supported by experiments (“*highly valuable and interesting ablation study*” (4KTy), “*experiments are thorough*” (r6QS), “*thorough analysis and convincing ablation*” (ZvgL)).

In response to requests for further clarifications and comparisons, we were able to execute most of the suggested experiments and plan to add them to the final version of the paper. **Please refer to the rebuttal PDF.** 

Below, we address common questions or questions providing valuable intuition, while the rest of the comments are addressed in individual replies.
# Comparison with ISM
We agree with reviewers r6QS and 4KTy that Interval Score Matching (ISM) is relevant since it uses DDIM inversion to improve the consistency of 3D guidance. Below we highlight the key differences between our work and ISM.
## Theoretical assumptions
ISM empirically observes that ``pseudo-GT’’ images used to guide Score Distillation Sampling (SDS) are sensitive to their input and that the single-step generation of pseudo-GT yields oversmoothing. From these observations, **starting with SDS guidance**, ISM adds DDIM inversion and multi-step generation to **empirically** improve the stability of the guidance.

As highlighted by the reviewer ZvgL, most advances in score distillation—including ISM—were obtained experimentally. In contrast, our work **starts with 2D diffusion sampling** to rederive score-distillation guidance and motivate improvements. That is, our work formally connects SDS to well-justified techniques in 2D sampling.
## DDIM inversion
In ISM the empirically motivated **DDIM inversion is at the basis of the derivation** of the final update rule. We suggest a general form of the noise term (eq.8), for which **DDIM inversion is just one possible solution**. Our theoretical insights are agnostic to particular algorithms of root-finding, which makes it possible to use more efficient solutions in future research (e.g. train diffusion models as invertible maps to sample noise faster).
## Guidance term
The update rules provided in our eq.10 and ISM’s eq.17 have two main differences:

1. **Full vs. interval noise.** Assuming DDIM inversion finds a perfect root of our stationary equation, ISM’s update rule can take a similar form to ours eq.10 (interval noise is equal to $\kappa$ if eq.8 is satisfied). However, as shown in Fig.9, DDIM inversion does not find a perfect root, and thus the two forms are not equivalent. Our theory shows that the **full noise term is more accurate**. We also show the effect of choosing one term vs. another in Fig.2 of the rebuttal PDF.

2. **Conditional vs. unconditional inversion.** Our derivation hints that the roots of eq.8 are prompt-dependent, motivating our use of conditional DDIM inversion (not used in ISM). Our Fig.10 shows how unconditional inversion yields oversaturation. To demonstrate this effect more clearly, Fig.5 of the rebuttal PDF provides a simple 2D experiment.

## Practical Results
To control for different design choices (GaussianSplattings in ISM vs. NeRF in ours, etc.) **we reimplemented our algorithm in the code base of ISM**, with only the minimal changes discussed in the previous section.

Fig.1 of the rebuttal PDF provides a qualitative comparison using the prompts and settings in ISM’s code. Fig.2 shows the effect of each change made to ISM guidance. Below is the quantitative comparison (ISM code base for both).
|Method|CLIP Score ↑|quality ↑|sharpness ↑|real ↑|ImageReward ↑|Time|VRAM|
|-|-|-|-|-|-|-|-|
|ISM, 5k steps|**28.60±2.03**|0.85±0.02|0.98±0.01|0.98±0.01|-0.52±0.48|45min|15.4GB|
|SDI (ours), 5k steps|28.47±1.29|**0.88±0.03**|**0.99±0.00**|0.98±0.01|**-0.30±0.32**|43min|15.4GB|

## Summary
We bridge the gap between experimentally-based score distillation techniques and theoretically-justified 2d sampling. Both SDS and ISM can be seen as different approaches to finding roots of eq.8., This theoretical insight allows us to modify both ISM and SDS, reducing over-saturation for the first and improving general quality of the second. 


# Smaller error in eq.8 = better 3D generation?
In response to reviewer ZvgL, we clarify the intuition behind the relation between 3D generation quality and error in root-finding of the stationary point equation. 
A more accurate solution does not necessarily yield a better 3D generation. It is better to say that **a more precise solution of eq.8 makes the guidance of score distillation closer to that of DDIM**. However, other factors contribute to the quality of 3D shape: initialization, view consistency, denoiser’s generative ability, etc. We address this question in more detail in the individual reply to reviewer ZvgL.
# Why negative CFG?
In response to reviewer 4KTy, we add intuition behind using negative CFG. While the general noise term is formally derived, the choice of negative CFG is mostly intuition-based and supported by experiments.

Contrary to a possible impression, negative CFG is not equal to a prompt of an opposite meaning. In fact, **negative CFG values attract the image to the mean of the dataset in the generation process and repulse it from the mean on DDIM inversion.** To support this claim we provide a theoretical and empirical argument in the individual response to reviewer 4KTy.

We notice that the NeRF initialization in SDS is very different from the i.i.d. Gaussian initialization in DDIM. Our intuition is that *negative CFG acts as regularization of the noise due to its mean-repulsion properties*. It brings the bad initialization of the NeRF closer to the Gaussian noise expected by DDIM.

Meta-review: The paper introduces Score Distillation via Inversion (SDI), a method that enhances 3D shape generation by improving the Score Distillation Sampling (SDS) process. SDI connects SDS with Denoising Diffusion Implicit Models (DDIM), reducing over-smoothing and preserving detail in 3D models. This approach improves 3D generation quality to match that of 2D images without needing additional training or multi-view supervision. SDI connects SDS with DDIM, providing deeper insights into the discrepancies between 2D and 3D generation. The method improves 3D generation quality without the need for additional training or complex setups. After carefully reading the paper, reviews, and rebuttals, the AC recommends to accept the paper.
Decision: Accept (poster)