Rethinking Score Distillation as a Bridge Between Image Distributions
Authors: David McAllister
Songwei Ge
Jia-Bin Huang
David W. Jacobs
Alexei A Efros
Aleksander Holynski
Angjoo Kanazawa
Venue: NeurIPS 2024 poster (2024)
Keywords: Generative Models
Diffusion Models
Primary area: diffusion_based_models

Abstract: Score distillation sampling (SDS) has proven to be an important tool, enabling the use of large-scale diffusion priors for tasks operating in data-poor domains. Unfortunately, SDS has a number of characteristic artifacts that limit its utility in general-purpose applications. In this paper, we make progress toward understanding the behavior of SDS and its variants by viewing them as solving an optimal-cost transport path from some current source distribution to a target distribution. Under this new interpretation, we argue that these methods' characteristic artifacts are caused by (1) linear approximation of the optimal path and (2) poor estimates of the source distribution.
We show that by calibrating the text conditioning of the source distribution, we can produce high-quality generation and translation results with little extra overhead. Our method can be easily applied across many domains, matching or beating the performance of specialized methods. We demonstrate its utility in text-to-2D, text-to-3D, translating paintings to real images, optical illusion generation, and 3D sketch-to-real. We compare our method to existing approaches for score distillation sampling and show that it can produce high-frequency details with realistic colors.

Rethinking Score Distillation as a Bridge Between Image Distributions
David McAllister Songwei Ge Jia-Bin Huang? David W. Jacobs
Alexei A. Efros* Aleksander Holynski* Angjoo Kanazawa' 1 UC Berkeley 2 University of Maryland
htapp:://dss-bbrddee woridgg.ggthub...... github.
Abstract
Score distillation sampling (SDS) has proven to be an important tool, enabling the use of large-scale diffusion priors for tasks operating data-poor domains. Unfortunately, SDS has a number of characteristic artifacts that limit its useful- ness in general-purposs applications. In this paper, we make progress toward understanding the behavior of SDS and its variants by viewing them as solving an optimal-cost transport path from a source distribution to target distribution. Under this new interpretation, these methods seek to transport corrupted images (source) to the natural image distribution (target). We argue that current methods' characteristic artifacts and (2) poor estimates of the source distribution. We show that calibrating the text conditioning of the source distribution can produce high-quality generation and translation results with little extra overhead. Our method can be easily applied across many domains, matching beating the performance of specialized methods. We demonstrate its utility text-to-2D, text-based NeRF optimization, translating paintings real images, optical illusion generation, and 3ID sketch-to-real. We compare our method to existing approaches for score distillation sampling and show that it can produce frequfryy details with realistic colors.
caused by (1) linear approximation of the optimal path
Introduction
Diffusion models have shown tremendous success in modeling complex data distributions like images [51 54 22], videos [59 and robot action policies [33] In domains where data is plentiful, they produce state-of-the- art results. Many data modalities, however, cannot enjoy the same scaling benefits due to their lack of sufficiently large datasets. In these cases, is useful to exploit diffusion models trained domains with rich data sources
prior in optimization framework. Score Distillation Sampling (SDS) 48 699 and its variants 77 20 76) are a widely adopted way to optimize parametric images, i.e., images produced by model like NeRF, with a pre-trained diffusion model. Despite being applicable to wide range applications, SDS is also known to suffer from several significant artifacts, such oversaturation and oversmoothing, As such, several variants have been proposed to alleviate these artifacts 770 76 32 often at the cost of efficiency, diversity, or other artifacts.
In this paper, we investigate the core issues with SDS by casting the class of score distillation optimization problems as Schrödinger Bridge (SB) problem [55] I2 42 which finds the optimal transport between two distributions. Specifically, given some images from the current optimized distribution (e.g., renderings from NeRF), applying the transport maps them to their pair images in a target distribution (e.g., text-conditioned natural image distribution). The density flow formed by these mappings is transport-optimal, as defined in the SB problem. In optimization framework, the difference between paired source and target samples, computed with an SB, can be
"Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

used as a gradient to update the source. Su et al. [65] have shown that this path can be explicitly solved using two pre-trained diffusion models. We show that one can also compose these models as an optimizer to approximate transport paths on the fly.
Under this framework, distribution bridge with the difference of two denoising directions. The denoising scores point to the source and target distributions respectively, with the source representing the current optimized image that updates with each optimization step.
understand SDS and its variants as approximating source-to-tare to-tageet
This framing reveals two sources of errors. First, these methods are a first-order approximation of the diffusion bridge. Specifically, Gaussian noise is sampled to perturb the current optimized image, and single denoising steps, instead of the full PF-ODE simulation, induces error in estimating the desired path. Recent works [4 41 that use multi-step estimation can be explained mitigating this error. Second, estimating the denoising direction to the current source distribution is non-trivial, since the current optimized image may not necessarily look like a real image (e.g., initializing with Gaussian noise or starting from a render of an untextured 3D model). Our analysis reveals that SDS approximates the current distribution with the unconditional image distribution, which not accurate and results in distribution mismatch error. We show that recent can be seen as proposals to improve this distribution mismatch error.
used to estimate the transport. This
SDS variants (70 76 32]
Finally, our analysis motivates simple method that rectifies the distribution mismatch issue without additional computational overhead. Our insight is that the large-scale text-to-image diffusion models learn from billions of caption-image pairs (56), where breadth of image corruptions are present in their training sets. They are also equipped with powerful pre-trained text encoders, which empower the models with zero-shot capacity in generating unseen concepts 53 As such, simply describing the current source distribution with text, even if it is not part of the real image manifold, can approximate the distribution of the current optimized image, leading to improved transport paths. Our simple and efficient solution can be easily applied to any existing application that uses SDS. We show that it consistently improves the visual quality in the desired domain. We comprehensively compare our approach with standard distillation sampling methods over several generation tasks, where our approach matches or outperforms the baselines.
Our contributions are as follows:
We propose to cast the problem of using a pre-trained diffusion model as a prior in an optimization problem as solving the Schrödinger Bridge (SB) problem between two image distributions. Specifically, it can be seen as bridging the distribution of the current optimized image to the target distribution under dual-bridge framework.
We analyze recent SDS-based methods under the lens of our framework and explain the pros and cons of the individual methods.
Our analysis motivates a simple yet effective alternative to SDS by using textual descriptions to specify the current optimized image distribution. It achieves consistently more realistic results than SDS, producing quality comparable with VSD 70 without its computational overhead. We compare various generation tasks to show its wall-clock efficiency and quality generations against state-of-the-art methods.
2
Related Work
Score Distillation Sampling Modalities like 3D, 4D, sketch, and vector graphics (SVGs) lack the large-scale, diverse, and high-quality datasets needed to train a domain-specific diffusion model. In these domains, previous works explore exploiting image or video as a proxy modality [26] 16).
By computing the gradient on a proxy representation with a pretrained model, optimization in the target modality is viable with differentiable mappings, e.g. differentiable rasterization (3 for SVGs or differentiable rendering 44 for 3D objects and scenes. The seminal method, Score Distillation Sampling (SDS) 48 first proposed to apply a pretrained text-to- -oxt-to-mmage diffusion model for text-to-3D generation. However, it requires a high classifier-fre free guidance weight and, therefore, suffers from artifacts such as over-saturation and over-smoothing. Recent works have built upon SDS to adapt it for editing tasks 30
46 29 or more broadly improve over the original SDS NFSD 28 and LMC-SDS ( inspect the individual components of the SDS gradient and propose methods to rectify the high guidance weights. However, the over-
20
formulation (2 70 76 78

---

X0.4
ESBP
Co.
Cp.
Eq.sre
C Gumoond
Volgg
X
X
ESRP
Po.tgt
Current distribution
Target distribution
Current distribution
CSBP
Target distribution
Secondary distribution (eig. unconditional)
(a) Score distillation as bridge between two distributions.
(b) Distribution mismatch error.
Figure I: Optimization with diffusion models as approximation of Schrödinger Bridge Problem (SBP), (a) We propose to formulate optimization with diffusion models as bridging the distribution of the current optimized image vp the target distribution under dual-bridge framework (a). Current methods can be interpreted as approximating the optimal transport SSpP between these distributions via the difference between projections of noised image ze, onto the two distributions. This analysis reveals two sources error: (1) these gradients are linear approximations of the optimal path, as illustrated in (a), and (2) the source distribution used for computing this approximation (e.g.. the unconditional distribution in SDS 48) may not be aligned with the current distribution, illustrated in (b).
formulates the problem as particle- saturation problem mitigated but not fully resolved. VSD based variational inference and proposes to train LoRA 24 on the fly to estimate the score of proxy distribution. We present new framework that allows rethinking all the variants under the same lens. This framework also motivates method that improves the quality of SDS without losing efficiency,
Visual Content Generation with SDS Since SDS was developed for text-to-3D generation, it has
also been adopted generate various other visual content such as SVGs [8 73 sketches [72]
texture 43 6 75), typography 3D bodies [5], dynamic 4D scenes [2 60 37 and illusions (5]
Among these applications, text-to-3D has been the most active research direction. In addition to designing better distillation sampling methods 70
77 28), prior work has also studied the
leveraging multiview data to improve (8). We note that these explorations orthogonal to study and should be able to work jointly with our method. this paper, we look into existing applications like text-based NeRF optimization, painting- to-rel,, and illusion generation. We also propose new AR
underlying 3D neural representations [74] 66, 35|
3D consistency [57 40
39 49
application called 3D sketch-to-real.
3 Method
In this section, we present analytical framework that casts the score distillation sampling (SDS) family of methods as instantiations Schrödinger Bridge problem. We show that many recent SDS based methods can be interpreted step is first-order approximation ODEs [65]. We analyze SDS and its variants under this general framework. Then, present simple solution based on the analysis, which leads to significant quality improvement with little extra computational overhead.
online solver for the problem. That is, each SDS optimization dual diffusion bridge formed by two probability flow (PF)
3.1 Background
Diffusion models define forward "noising" process that degrades data samples gradually from the image distribution to noised samples 2,, and eventually the i.i.d. Gaussian distribution [23 62 This process is indexed by timesteps 4, where = 1 indexes the full Gaussian noise distribution and I == 0 indexes the data distribution. A diffusion model, parameterized by Φ. is then trained to reverse this encoding process, iteratively transforming the noise distribution into the data distribution with the following denoising objective:
Loir(d, (b, x) = (.. [w(() ||e (a.x + oze;y, () - =112]
where wt) is loss weighting function, y conditioning text prompt, and a, and ar are hyperpa- rameters from the predefined noise schedule.

---

Probability Flow ODE. Denoising score matching 664 27 61 shows that the diffusion model denoising prediction can be rewritten as score vector field:
V log = vI d4
(2)
Because of its special connection to marginal probability densities, the resulting ODE is named the probability flow (PF) ODE with the following expression:
dx = ( (1VV> (o lo
where f(x, t) and g(t) are pre-defined schedule parameters. This PF-ODE can be solved determinis- tically [63]. mapping a noise sample to its corresponding data sample through the reverse process and the opposite through the forward process (inversion). This cycle-consistent conversion between image and latent representations is important in establishing dual diffusion implicit bridges.
Dual Diffusion Implicit Bridges. Dual Diffusion Implicit Bridges (DDIBs) [65] compose diffusion inversion and generation process for solving image-to-image translation problems without requiring paired image dataset. Instead, DDIBs use two diffusion models trained on different domains (or, analogously, one model with two different text conditions). DDIB inverts the source image into noise latent via the forward PF-ODE and then decodes the latent in the target domain via the reverse PF-ODE. DDIBs can be interpreted as a concatenation of the Schrödinger Bridges from cource-to-latent and latent-to-target hence the dual bridges in its name. DDIBs enable solving transport between two distributions using a single pre-trained diffusion model. We build on this insight in an optimization context.
3.2 Optimization with Diffusion Model Approximates Dual Schrödinger Bridge
Many generative vision tasks involve optimizing corrupted images to the image manifold. For example, in 3D generation, a 3D representation like NeRF is optimized to render natural images matching a prescribed text prompt. Methods like SDS enable this by using a pre-trained diffusion model prior. We propose formulating such optimization problems as solutions to instantiation of a Schrödinger Bridges Problem (SBP). SBP finds cost-optimal paths between a source image distribution Pue and target image distribution Pig (68 14]. Optimizing parametrized image toward the natural image distribution can be cast as finding the optimal paths between the current optimized image(s) and the natural image distribution. Instead of solving this problem directly, which would require training a generative model from scratch [38] 14 10], we show that pre-trained diffusion models can be exploited by the existing score distillation methods can be viewed as the first-order approximation of this path. This formulation is illustrated in Figure T..
optimizer that approximates the path. Further, the gradient computed
Let X9 E R2 represent a parametric image, i.e., an image produced differentiably by a model with parameter 0, such as NeRF To leverage the pretrained diffusion model, we add noise ~ NO, to obtain latent at timestep l:
Xg, = 0;Xg + O2E
Suppose that the are and the denote the paths obtained by solving the PF ODE as in Eq. from to O. both starting from Xg.t, such that vo E Pae. tho. E Pigt, Vt.sxe = t X0,1- This forms dual diffusion bridge [65] from too,see to too.gtt We approximate this path per-iteration using a pretrained diffusion model. We denote the displacement of this path as:
CBP = V00.gt 3/0,
(5)
Fully simulating this bridge involves solving two PF ODEs, which invokes dozens of neural function
evaluations (NFEs) estimate the gradient of each iteration. Instead, one estimate each half of the bridge with a single-step prediction by computing two denoising directions fo and ca.g We thus obtain a first-order approximation of dual diffusion bridge with the difference vector:
ESBP
Cd.tet Ed,,soo
(6)

---

291
101
XoA
Wozgt
Xg
Verp
FDDS
CSBP
Cysp
Cumentditribucon
Targus disriaation
(a) Variational Score Distillation
(b) Delta Distillation Sampling
Classifier Score Distillation (2nd term)
Figure 2: Comparision of SDS variants under our analysis. illustrate the major gradient components different SDS variants and provide a straightforward comparison with espp.
which is subject to the following sources of errors.
First-order approximation error. Instead of performing full PF-ODE simulations, the single-step noising and prediction are less accurate and induce errors. Recent work ISM [3] can be interpreted as reducing this error with a multi-step simulation to obtain X0,+-
2. Source distribution mismatch. The dual diffusion bridge relies on 6 accurately estimat- ing the distribution of the current sample, Xg. A series of works can be viewed as improving this error 77 28 76
by computing more accurate <..5.
We show that 6..igt fdb.sr is an effective gradient when both the source and target distribution are well expressed. Next, we discuss the popular score distillation methods under this analysis. We argue that their characteristic artifacts can largely be understood due the errors above.
3.3 Analyzing Existing Score Distillation Methods
We analyze SDS and its variants through our framework by inspecting each component in the computed gradient. For notation, Vigt is the text prompt representing the target distribution, and ø denotes the unconditional prompt. For each method, we present its gradient update and discuss its implications.
Score Distillation Sampling
48
CDD Co (xo,; s. (Cp (Xo,,; Ulgs co (xo,1; 2,t)) c,
where the strength of classifier-free guidance. When is small, the functions an averaging term to regress the image to the mean. However, the SDS gradient has been shown to work best with extreme values of classifier-free guidance like 100. We can rewrite the gradient emphasize how the conditional innditionllunnondiiionn delta dominates at high CFG scales.
<SDS = 8 (Fi (x0.15 Bigi, 1) - ss (x0,13 2), +ep (x0,1;0,1,
Dominant when
Experimentally, we produce very similar results at high CFG with or without the non-dominant terms. We argue that SDS should be interpreted through the dominant term, which fits within our analysis. Under this interpretation, the unconditional direction ø (Xa.1 Ø, () approximates the source distribution of Xo poorly, instead representing images of any identity with low contrast and geometric artifacts. Figure [[b) illustrates the effect of poor approximation. The bridge from the unconditional conditional distribution leads the characteristic oversaturation and smoothing SDS results.
Delta Distillation Sampling 20
EDDS = Eg (Xo,e) Sigr t) - E (Xrefiti yarcy t),
where x is a noised version of a reference image in the image editing task. As shown Figure (b). this increases the source distribution mismatch since 60,ste is not calculated based the current optimized image Xe.t-
Noise Free Score Distillation [28]:
ENFSD (Eg (X,,t: 2,t) (t 0.2) es (xo,1 ei Uoegy t)) (Eg (x9, Vlgr cs (xx, 2,t)),
5

---

where the strength of classifier-free guidance is set to 7.5 and Yuog -unreallisti,, blurry, low quality .... NFSD greatly reduces the guidance strength while is observed to perform very similarly to SDS in practice. We can better explain this phenomenon since the prompt Yneg does not accurately describe the source distribution as omits the image's content. In addition, the second component with weight = 7.5 still forms the major part of the gradient, which is the dominant term in SDS.
Classifier Score Distillation 76
CCSD w1 (Cp (Xe,t: Myg. co (Xa,1; 2,t)) + u2 (Co (Xo,e; co (xo,15 Muee t)),
where W1 and Wy hyperparameters, As shown in Figure (c). the second term approximates the bridge from the source distribution to the unconditional distribution, which is not ideal since it does not point the target distribution. It explains the observation made by the authors [76] that this undermines the alignment with the text prompt. Therefore, the authors always anneal wy to during the optimization. However, show this often reintroduces the SDS artifacts in practice.
Variational Score Distillation [70 32
EVSD = Eg (Xe,e; Ø.t) (EG (No, Bhgt t) cΦ (X0,1; R.t)) ELORA (xe,15 yhgt t).
Out of all the discussed methods, VSD attempts minimize the source distribution mismatch error most directly by test-time finetuning a copy of the diffusion model with LoRA on the current set of Xg. Note that in the original paper, the use of LoRA motivated based on particle-based variational framework. Our analysis enables an alternative understanding of VSD. As shown in Figure a),, this approach is well-justified in our dual diffusion bridge framework. However, training LoRA every iteration is computationally expensive, adds complexity, and introduces its own low-rank approximation errors. Given this insight, we propose simple yet efficient approach to mitigating source distribution without LoRA.
3.4 Mitigating Source Distribution Mismatch with Textual Descriptions
Our analysis reveals that the LoRA model in VSD most closely approximates the distribution of the current optimized parametrized image, addressing the distribution mismatch error. Unfortunately. incurs 200 300% runtime overhead on top of SDS, making it impractical, despite its significant performance gains. With this understanding, we propose a simple approach that better expresses the source distribution. Our insight is that pre-trained diffusion models have learned the distribution of natural and corrupted images through a combination of powerful text representation and enormous image-caption datasets. We find that by simply describing image corruptions with text prompt, can improve our estimate of the source distribution.
Specifically, we propose to use the gradient
Enurs w - (Eg (Xe.ri Migt () - (Xo,t5 Msrce ()),
where we get Ysrc by adding descriptions of the current image distribution to that (the base prompt). The remaining question is how to set this description. In generation tasks, we propose a simple two-stage solution.
1. We use CsDS to produce generation with the method's 's characteristic artifacts:
2. We switch to optimization with our gradient, Cours. transport the image parameter toward the natural image distribution.
To describe the artifacts produced by SDS, we append the descriptors ", oversaturated, smooth, pixelated, cartoon, foggy, hazy, blurry, bad structure, noisy, malformed" and drop the descriptors of the high-quality generation. This description Vse does not require hand-crafting based on problem domains-ie it is fixed across all shown examples and use cases. As shown in Appendix Figure A5, we explored searching for other prompts but did not find that variations in these descriptions made a big difference.
In editing tasks, we have an initialization that you describes accurately. In such cases, we omit the first SDS stage and only apply our gradient to optimization. We also append a "domain descriptor." For instance, in painting- rearl,,, this is simply ", painting" to represent the initial distribution.

---

DDIM Sampling
SDS
NFSD
CSD
VSD
Ours
Figure 3: Text-to-image generation results with COCO Captions. We compare different score distillation methods for generating images with COCO captions by optimizing randomly initialized image. DDIM sampling indicates the lower bound that the diffussin model achicve. VSD 70 and our method generate the least color artifacts while ours is more efficient than VSD.
Table 1: Zero-shot FID comparison with different score distillation methods. We report FID scores of text-to-image generation using 5K captions randomly sampled from the COCO dataset. The best score distillation result indicated in bold, while the second best is underlined.
DDIM (lower bound)
SDS [48]
NFSD
CSD 76
VSD 70
Ours
Zero-Shot FID (L)
49.12
86.02
91.70
89.96
59.22
67.89
Zero-Shot CLIP FID (L)
16.56
28.39
29.25
27.07
18.86
20.31
Time per Sample (mins)
0.05
4.48
7.20
6.21
16.02
4.48
While the of such negative prompting has been explored before, such in NFSD, our analysis motivates a principled way to incorporate into score distillation. We find that these simple modifications significantly narrow the quality gap between SDS and resource-intensive methods like VSD. We verify this finding experimentally with qualitative results and quantitative comparisons across applicable tasks.
Experiments
In this section, we test our proposed method on several generation problems where SDS is adopted. We compare against SDS and other task-specific baselines. Note that our goal is state-of-the-art text-to-3D generation method, but to verify our findings, where the proposed score distillation approach based textual description efficiently improves the results by mitigating the source distribution mismatch сттоr. We first perform a thorough experiment in a controlled setting on text-to-image generation. Then, we compare on text-guided NeRF optimization SDS and VSD and evaluate the painting-to-real image translation task against image editing basclines. Please more results in the appendix, including additional qualitative results and comparison, ablation studies and our method's application to optical illusion generation and 3D-sketch-to torccc-tt-raall task.
show another
4.1 Zero-Shot Text-to-Image Generation with Score Distillation
To verify our analysis of existing SDS variants and the proposed method, we perform text-to- to-image generation by optimizing an image size 64 x 64
the Stable Diffusion latent space 770 28
(We explore other base models like MVDream [7]] and SDXL 47 in Appendix Figure A3 The benefit of choosing image generation as the evaluation task that generation quality has the least
confounding variables among other tasks. (e.g., in text-to-3D, many designs like regularizations 77
initialization (35), 3D representations affect the final quality.)
67
74 66 and 2D prior models
39
78 could

---

VSD
SDS
Ours
VSDD
SDS
Ours
A giant rock with detailed, high resolution, high quality, sharp
wooden chair, detailed, high resolution, high quality, sharp
model an adorable cottage with thatched roof.
Various hollow, asymmetrical, textured seashells, collected in sand mildfi, clear glass jar with twine-tied neck
Figure 4: Text-guided NeRF optimization with different score distillation methods. We make fair comparison of SDS and VSD for text-to-3D generation. For each generation, show three uniformly sampled views. SDS results like the cottage and pepper mill still suffer from over-saturation problems, while ours and VSD can produce realistic details, color, and texture.
We use the MS-COCO 33 dataset for the evaluation. Consistent with the prior study Bi we
randomly sample 5K captions from the COCO validation set as conditions for generating images. For each caption, we optimize a randomly initialized the image with the score distillation gradients. We compare our method with several SDS variants including SDS [48), NFSD (28, CSD (76), and VSD 77). For all the methods, we use the same learning rate of 0.01 and optimize for 500 steps where we generally observe convergence. We compute the zero-shot FID [2] and CLIP FID scores 31] between these generated images and the ground truth images. We also report results generated by DDIM with 20 steps a lower bound for renference.
We report the FID scores and the time to optimize one image in Table T Among all the score distillation methods, VSD [70] achieves the lowest FID scores. However, it requires training a LoRA along the optimization process. Instead, ours achieves a comparable FID score with over 3x faster speed. We visualize random examples generated by different score distillation methods in Figure 3 We notice that SDS and NSFD suffer from the over-saturation and werrssoothness issues. CDS has slightly fewer color artifacts. VSD and ours generate the samples that most closely resemble the DDIM sampling.
4.2 Text-guided NeRF Optimization
We now evaluate the text-to-3D generation problem, where we intentionally aim exclude variables that could affect the generation quality other than the score distillation methods. We use the Three- Studio 19 repository optimize a NeRF with settings tuned for ProlificDreamer stage 1 (NeRF optimization) [70]. Note that we do not perform stages 2 and 3, i.e. geometry fine-tuning and texture refinement. Specifically, we initialize the NeRF with the method proposed by Magic3D (35), use the regularization losses on the sparsity and opacity, and optimize for 25K steps. We adopt the native SDS and VSD guidance implementations for comparison. Appendix Figure A4 we evaluate our methods with additional text-to-3I systems, including Fantasia3D 9 Magic3D
35 and CSD 76
We first show visual comparisons of different score distillation methods in Figure 4 We notice that SDS tends to generate fewer details, as shown by the rock and chair examples, and sometimes suffers from over- saturation issues, as in 2D, as demonstrated by the cottage and seashell examples. Instead, both VSD and ours can generate highly photo-realistic 3D ob- jects, while ours does not require training LoRA model and shares a similar computational cost as SDS.
Table 2: Quantitative comparisons of NeRF opti- mization. We measure the average CLIP similarity of rendered views using SDS, VSD and our.
ViT-L/14
ViT-B/16
ViT-B/32
SDS
0.2811
0.3196
0.3139
VSD 70
0.2837
0.3292
0,3166
Ours
0.2848
0.3282
0.3148
8

---

"a DSLR photo of blue pond with water libees"
Input
Plug-and-Play
SDS
SDEdi Strength 0.5
CycloAN
Ours
"a DSLR photo of historic stone church park with gravel and trees"
Input
Plug-and-Play
Strength 0.5
CycleGAN
Ours
Figure 5: Painting-tos Re-l comparison. We compare our gradient in optimization to image restoration and image-condittinal generation baselines. While SDEdit produces convincing textures, it difficult to find a strength value that balances structure and quality. Other baselines fail to reproduce natural image quality, while method produces the best combination of quality and faithfulness.
We also perform quantitative evaluation and user study on the NeRFs optimized based on 31 different text prompts. Note that this number is similar to the choice of existing works the text-to- 3D task + [5] However, different from these works that ignore the confounding 3D variables that contribute to the generation quality, disentangle this by isolating the score distillation method as the only comparison variable. We follow these works to evaluate the generation quality with CLIP [50]. We report the CLIP similarity in Table [ Our method consistently outperforms SDS and achieves comparable results with VSD. addition, pairwise comparisons of rotating 3D renders (i.e., comparisons of result and random choice of VSD or SDS, with the prompt: "For a text-to-3D system, given the prompt [p]. which result would you be happiest with?"), our results were chosen in 75.7% of all responses.
user study consisting of 37 users, shown
4.3 Painting-to-Rao
We examine our method's s ability to serve as general-purpor realism prior. Paintings are "near- manifold" images, meaning they do not possess natural image statistics but live near the image distribution in image space. An effective image prior should guide a painting toward a nearby natural image through optimization.
We initialize latent image by encoding scans of the artwork through Stable Diffusion's encoder. We specify a prompt for each painting to condition the diffusion model and then apply the second opti- mization stage of method (SDS stage omitted). We experimented with automatically generating prompts via pretrained vision language models but found the results inconsistent, so we leave this future work. Since the large image datasets used to train diffusion models contain artwork, we append the domain descriptor painting" to Ysec to optimize away from this distribution.
While SDS is proposed leverage pretrained text-to-image diffusion model as image prior, its artifacts make ineffective in practice. In comparison, our method realistically synthesizes details and relights the image naturally. We observe that SDS methods diverge more easily in 2D experiments than in 3D but that the issue can be mostly resolved with tuning. A future goal to formulate gradient that can be applied idempotently (58). We compare with image reconstruction baselines in Figure and provide a small gallery of painting-to-real results in Figure
Discussion on Solving the Linear Approximation Error
As we have shown that reducing the distribution mismatching error can significantly improve the generation quality of the score distillation optimization, it is natural to ask whether one can also reduce the first-order approximation error, induced by linear bridge estimation, to improve the results further. Several recent studies, including SDI 41 and ISM (34), can be viewed mitigating

---

Figure 6: Painting-to teee-tto-Reee Real results. We show selected Painting- to Real samples with diverse art styles and subjects. Initialization images shown on the left, optimized images shown the right.
Table 3: Reducing first-order approximation error improves generation quality. Using full PF-ODE simulation "FFull-pat"") to replace single-step prediction improves visual quality in all settings.
Approximate the bridge
Single-step
Full-path Uncond. Bridge
Estimate source distribution
Uncond. (SDS) Bridge LoRA (VSD)
LoRA
Zero-Shot COCO FID (L)
86.02
67.89
59.22
63.31
60.07
55.65
this error by replacing the single-step estimation with a multi-step estimation to an intermediate timestep. Under our framework, one can estimate the entire dual bridge by solving both PF-ODE paths. Specifically, via inversion, one can solve the PF-ODE path from do sre to Ie.T, and then walk to the Vo..tr via sampling. In this way, it is possible to obtain the most accurate gradient direction with little approximation error SBP w (foo.gg t/o,.r)) We refer to this approach as "full path". Note that this resolves the linear approximation error, and it independent of handling the source approximation error, which could be addressed via the discussed text description or LoRA.
However, solving the inversion ODE is not trivial (27] We noticed that the inversion the distribution mismatch error and cause the optimization to get stuck at a local optimum at the beginning of the optimization. Instead, the stochasticity of the single-step methods often shows more robustness to the input image. Therefore, we first perform the single-step score distillation optimization to obtain reasonable results and then switch solving the full bridge. We also anneal the timestep endpoint of the bridge throughout the optimization. With this approach, we can now explore addressing both the first and second sources of error. The first source (linear approximation) has "full-path and the second source (source distribution mismatch error) has "Bridge" or "LoRA". We find that using the "full-pathi multi-step (mitigating linear approximation error) always outperforms the single-step methods, achieving lower FID, as shown in Table 3 However, the same trend does not fully transfer to the text-to-3D experiments. We observe that solving the entire bridge typically introduces additional artifacts and makes the optimization less stable. We leave the best way of leveraging this gradient for future research exploration.
exaggerate
Conclusion
We present analysis that formulates the use of a pre-trained diffusion model in optimization framework as seeking an optimal transport between two distributions. Under this lens, we analyze SDS variants with a unified framework. We also develop a simple approach based on textual descriptions that work comparably well to the best-performing approach, VSD, without its significant computational burden. However, neither approach has yet achieve the quality and diversity of images generated by the reverse process. We hope that our analysis enables the development of more sophisticated solution that can one day achieve the same quality and diversity as the reverse process in an optimization framework. Combining our proposed method with multi-step approximations 32 could mitigate the first-order approximation error and further improve the efficiency, which is an interesting future research direction. With the rise of high-quality video diffusion models, we anticipate that the question of how to effectively use such models as prior in various problems will become even more important.
like ISM (34] or schedules like DreamFlow
10

---

Acknowledgment. We thank Matthew Tancik, Jiaming Song, Riley Peterlinz, Ayaan Haque, Ethan Weber, Konpat Preechakul, Amit Kohli and Ben Poole for their helpful feedback and discus- sion. This project is supported in part by Google research scholar award, IARPA DOI/IBC No. 140D0423C0035, and NSF grant No. IIS-2213335. The views and conclusions contained herein are those of the authors and do not represent the official policies or endorsements of these institutions.
References
[1] Thiemo Alldieck, Nikos Kolotouros, and Cristian Sminchisescu. Score distillation sampling with learned manifold corrective, 2024.
[2] Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B. Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. arXiv preprint 1rX1:2311..7984, 2023.
[3] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. eDiff-I: Text-to- image diffusion models with an ensemble of expert denoisers. arXiv preprint arX1v:221..01344 2022.
[4] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
[5] Ryan Burgert, Xiang Li, Abe Leite, Kanchana Ranasinghe, and Michael Ryoo. illusions: Hiding images in plain sight, June 2023.
Diffusion
[6] Tianshi Cao, Karsten Kreis, Sanja Fidler, Nicholas Sharp, and KangXue Yin. Texfusion: Synthesizing 3d textures with text-guided image diffusion models. In ICCV, 2023.
Hsin-Ying [7] Dave Zhenyu Chen, Haoxuan Li, Hsin- Ying Lee, Sergey Tulyakov, and Matthias Nießner. Scenetex: High-quality texture synthesis for indoor scenes via diffusion priors. arXiv preprint arXiv:2311.1726. 2023.
[8] Dave Zhenyu Chen, Yawar Siddiqui, Hsin- Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303. 1396, 2023.
[9] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In ICCV, 2023.
[10] Tianrong Chen, Guan Guan-Horng Liu, and Evangelos A Theodorou. Likelihood training of schrödinger bridge using
sdes theory. In ICLR, 2022.
[11] Yongxin Chen and Tryphon Georgiou. Stochastic bridges of linear systems. IEEE Transactions on Automatic Control, 61(2)5226-331 2016.
[12] Yongxin Chen, Tryphon T. Georgiou, and Michele Pavon. On the relation between optimal transport and schrödinger bridges: A stochastic control viewpoint. Journal of Optimization Theory and Applications, 169:671-691, 2014.
[13] Cheng Chi, Siyuan Feng Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In RSS, 2023.
[14] Valentin De Bortoli, James Thortoo,, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications score-basee generative modeling. In NeurIPS, 2021.
[15] Wenqi Dong, Bangbang Yang, Lin Ma, Xiao Liu, Liyuan Cui, Hujun Bao, Yuewen Ma, and Zhaopeng Cui. Coin3d: Controllable and interactive 3d assets generation with proxy-guided conditioning, arXiv preprint 205.::000000000444.. 2024.

---

[16] Kevin Frans. Lisa Soros, and Olaf Witkowski. Clipdraw: Exploring eextto-drrwing synthe- sis through language-iman encoders. Advances Neural Information Processing Systems, 35:5207-5218, 2022.
[17] Daniel Geng, Inbum Park, and Andrew Owens. Visual anagrams: Generating multi-view optical illusions with diffusion models. CVPR, 2024.
[18] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, pages 10696-10706, 2022.
[19] Yuan-Chen Guo, Ying-Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia-Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang. threestudio: A uni- fied framework for 3d content generation. https://gitthb:: wwtp////reeeessuooooooerrrr proojjjjtt threestudio 2023.
[20] Amir Hertz, Ktir Aberman, and Daniel Cohen-Or. Delta denoising score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2328-2337, 2023.
[21] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurlPS, 2017.
[22] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arxiv:22100023303, 2022.
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[24] Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[25] Shir Nuz, Yael Vinker, Amir Hertz, Daniel Berio, Daniel Cohen-Or, and Ariel Shamir. Word-as- image for semantic typography. SIGGRAPH, 2023.
[26] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. CVPR, pages 867-876, 2022.
[27] Tero Karras, Mika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint 200.:222000000444, 2022.
[28] Oren Katzir, Or Patashnik, Daniel Cohen-Or Or, and Dani Lischinski. Noise-free score distillation. arXiv preprint arXiv:2310.17590, 2023.
[29] Subin Kim, Kyungmin Lee, June Suk Choi, Jongheon Jeong, Kihyuk Sohn, and Jinwoo Shin. Collaborative score distillation for consistent visual editing. In NeurlPS, 2023.
[30] Juil Koo, Chanho Park, and Minhyuk Sung. Posterior distillation sampling. arXiv preprint arXiv:2311.13831, 2023.
[31] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. In ICLR, 2022.
[32] Kyungmin Lec, Kihyuk Sohn, and Jinwoo Shin. Dreamflow: High-quality text-to-3d generation by approximating probability flow. In ICLR, 2024.
[33] Tzu-Mao Li, Michal Lukáč, Michaël Gharbi, and Jonathan Ragan-Kelley. Differentiable vector graphics rasterization for editing and learning. ACM Transactions on Graphics (TOG), 39(5):1-15, 2020.
[34] Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Lucid- dreamer: Towards high-fidelity text-to-3d generation via interval score matching. CVPR, 2023.
12

---

[35] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023.
[36] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[37] Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models, 2024.
[38] Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos Theodorou, Weili Nie, and Anima Anandkumar. 12sb: image-to-image schrödinger bridge. In ICML, 2023.
[39] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1eco-1-to- 3o: Zero-shot one image to 3d object. In CVPR, 2023.
[40] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consisten images from a single-view image. In ICLR, 2024.
[41] Artem Lukoianov, Haitz Sáez de Ocáriz Borde, Kristjan Greenewald, Vitor Campagnolo Guizilini, Timur Bagautdinov, Vincent Sitzmann, and Justin Solomon. Score distillation via reparametrized ddim. arXiv preprint arXiv:2405.15891 2024.
[42] Christian Léonard. A survey of the schrödinger problem and some of its connections with optimal transport, 2013.
[43] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. In CVPR, pages 12663-12673, 2023.
[44] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[45] Lea Muller, Vickie Ye, Georgios Pavlakos, Michael J. Black, and Angjoo Kanazawa. Generative proxemics: A prior for 3D social interaction from images. 2024.
[46] Hyelin Nam, Gihyun Kwon, Geon Yeong Park, and Jong Chul Ye. Contrastive denoising score for text-guided latent diffusion image editing. CVPR, 2022.
[47] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
[48] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.
[49] Guocheng Qian. Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem. Magic One image to high-quality 3d object generation using both 2d and 3d diffusion priors. In ICLR, 2024.
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[51] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:220.006225, 2022.
[52] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, arXiv preprint arXiv:2204.06125, 2022.
13

---

[53] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Πya Sutskever. Zero-shot text-to-image generation. ICML, 2021.
[54] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
[55] E. Schrödinger. Sur la théorie relativiste de l'électron et I'interprétation de la mécanique quantique. Annales de l'institut Henri Poincaré, 2(4):269-310, 1932.
[56] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
[57] Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3d generation. In ICLRR 2024.
[58] Assaf Shocher, Amil V Dravid, Yossi Gandelsman, Inbar Mosseri, Michael Rubinstein, and Alexei A Efros. Idempotent generative network. In ICLR, 2024.
[59] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. In ICLR, 2023.
[60] Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, lurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene generation. arXiv preprint arXiv:2301. //200, 2023.
[61] Jaseha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In /CML, 2015.
[62] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[63] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022.
[64] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[65] Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. In ICLR, 2022.
[66] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation, 2023.
[67] Christina Tsalicogiou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. arXiv preprint arXiv:2304. /2439, 2023.
[68] Gefei Wang, Yuling Jiao, Qian Xu, Yang Wang, and Can Yang. Deep generative learning via schrödinger bridge. In ICML, 2021.
[69] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for generation. In CVPR, pages 12619-12629, June 2023.
[70] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. NewrIPS, 2023.
[71] Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. Gpt-4v(ision) a human-aligned evaluator for text-to-3d generation. In CVPR, 2024.
14

---

[72]
Ximing Xing, Chuang Wang, Haitao Zhou. Jing Zhang, Qian Yu, and Dong Xu. Diffsketcher: Text guided vector sketch synthesis through latent diffusion models. In NeurIPS, 2023.
[73] Ximing Xing, Haitao Zhou, Chuang Wang, Jing Zhang, Dong Xu, and Qian Yu. Svgdreamer: Text guided svg generation with diffusion model. arXiv preprint arXiv:2312.1641 2023.
[74] Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models, 2023.
[75] Kim Youwang. Tac-Hyun Oh, and Gerard Pons-Moll. Paint-it: Text-to-texture synthesis via deep convolutional texture map optimization and physically- rendering. In CVPR, 2024.
[76] Xin Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Song- Soig-Hai Zhang, and Xiaojuan Qi. Text-to-3d with classifier score distillation, 2023.
[77] Junzhe Zhu and Peiye Zhuang. Hifa: High-fidelity text-to-3d generation with advanced diffusion guidance, 2023.
[78] Zi-Xin Zou, Weihao Cheng. Yan-Pei Cao, Shi-Sheng Huang, Ying Shan, and Song- Hai Zhang. Sparse3d: Distilling multiview-consistent diffusion for object reconstruction from sparse views, 2023.
15

---

3D Sketching
Ours with Prompt flower"
SDS Baseline
Figure A1: 3D sketch-to-real. We introduce conditional generation task in 3D where coarse human-drawe mesh is optimized into high-quality mesh. While SDS and our gradient both adhere the prompt and shape conditions, our method produces higher fidelity colors and texture.
Appendix
In this appendix, we discuss the additional experiment details and provide more visual results, including optical illusion sketch, text-based NeRF optimization, and 3D paint-to-real results. We also perform an ablation study of our method.
A Additional Experimental Setup
In this section, we describe our experimental setups in more detail.
Text-to-image generation with score distillation. We the stable sosile-difussoon-2--1 --a-bbas------ model by default for our experiments if not specified. For CSD, we follow the original paper [76] to use wi W2 40 the initialization steps and anneal w2 0 within the first 500 steps. We use 8 = 100 for SDS and 7.5 for NFSD and VSD, which are consistent with the best practice. We use 40 and w 25 for our method. And we optimize with ESDS loss for 500 iterations and then switch to Couss for the rest of 2, 000 iterations. For all the methods, we use a learning rate of 0.01, and train the LoRA in VSD.
we use learning rate of 1e
Text-guided NeRF optimization with score distillation. For our method, we optimize with ESDS loss for 20, 000 iterations and then switch to Cours for the rest of 5, 000 iterations. We use 100 and w for our method. We find that a high s necessary to establish geometry in the first stage of the eext-to-3D setting, but our method is not too sensitive to this hyperparameter in 2D. We use the rest of the learning rates and regularization strengths as the default settings.
B More Visual Results
In this section, we provide extra visual results. Specifically, we show 3D sketch-to-real and optical illusion generation as additional applications of our method. We also report more comparisons and ablation studies of text-based NeRF optimzition.
B.1 Additional Applications
3D Sketch-to-Real
Hcad-mounted displays with hand tracking natural platform for sort of "3D sketching," where 3D primitives trail from your hand like ink from a pen. The resulting coarse mesh is structurally accurate but lacks geometric or texture detail. To this end, we propose new application that transfers these 3D sketches to more realistic versions. We extend our text-to-3D solution to generate these details.
We first an implicit SDF volume to multi-view renders of the mesh, then apply our gradient with the same schedule as in text-based NeRF optimization. We lower the learning rate for geometry parameters prevent divergence from the guiding sketch. Holding other hyperparameters equal, we compare gradient and the SDS gradient in Figure Al
16

---

Base
Rotator
Basc
Rotator
Q
0
90
180
270
0
90
180
270
Base
Rotator
Base
Rotator
Q
0
90"
180
270
0r
90
180
270
SDS 48
Ours
Figure A2: Diffusion illusions. We generate overlaid optic illusions with SDS and method. While SDS suffers from color artifacts, our methods produce more details and proper color.
SDS
Brigge
(a) MVDream
57
(b) SDXL 47
Figure A3: Comparison SDS and ours with MVDream and SDXL We compare SDS with
two-stage process in two new settings (MVDream and SDXL). The two stoestage process produces more natural colors and realistic details.
Illusion Generation. Prior works have shown that diffusion models can be leveraged to generate op- tical illusions []] In these settings, the same image looks semantically different when transformed. To use the diffusion model sampling process, previous study shows that the transformation has to be orthogonal [17]. However, there remain interesting illusions that not formed by orthogonal transformation. One such the rotation overlays. Given a base and rotator image, by composing the base image with the rotator image different angles, rotation overlays four images. As such composition is not defined by an orthogonal matrix, the existing method [5] employs SDS optimize the base and rotator images. Such method suffers from the over-saturation problem, as shown in Figure A2 better visual quality.
images display
We show that our method can generate such optical illusions with
B.2 Additional Qualitative Results
Additional text-to-image results. We explore our proposed method across different base models in cext-to-image experiments, including MVDream [57] and SDXL [47]. Since MVDream denoises
17

---

Fantasia3D SDS
Fantasia3D w/ Brigge
Magic3D w/ SDS
Magic3D w/ Bridge
CSD Bridge
A DSLR photo of an ice cream sundae
A delicious hamburge
A pineapple
Figure A4: Comparison with more text-to-3D baselines. We apply our two-stage optimization as drop-in replacement of SDS in Fantasia3D (9] Magic3D [35] and CSD [76] for texture refinement. We notice that this change greatly improves details and visual quality and reduces SDS artifacts.
Bridge, Ou: Prongt
Bridge,
Bridge, GPT
Bisigs, GPT3
Bridge, Bridge,
Figure A5: Ablation study of negative prompts. We compare SDS results with those from the two-stage optimization (bridge) using our negative prompts and five sets negative prompts generated by GPT (GPT the first set, GPT second, GPT third, ctc.). All negative prompts produce similar results and outperform the SDS baseline.
four camera-conditioned images jointly, we treat the canvas of four images as a single optimization variable for the SDS gradient. Figure [A3] we compare the SDS baseline to the proposed two-stage optimization, in which we generate more natural colors and detail. This is especially noticeable in the background around the crocodile and donkey.
Additional text-guided NeRF optimization results. For text-guided NeRF optimization compari- son against baselines, we show more results in Fig. A7 We test on the prompts used the original paper (70) and additional prompts [7] that we find to be challenging. We notice that SDS often suffers from over-saturation problems. Our method does not require training LoRA while it can still improve SDS by getting rid of the color artifacts and generating more details.
We also perform comparisons with more competitive baselines. We test with Fantasia3D (9] Magic3D 335, and CSD [76] through a drop-in replacement of SDS with our method. Specifi- cally, all three methods optimize a textured DMTet, which initialized from an SDS-optimized NeRF, using SDS or CSD for 5k or 10k iterations. We replace the SDS or CSD stage of these approaches with the two-stage optimization motivated by our framework. Just like our text-to-3D) NeRF experiment, we perform the first stage for 60% of iterations and the second stage for 40% of iterations. Note that we keep all the other hyperparameters the same, which were tuned for the baselines, not our method. This replacement leads to the same optimization time as the original methods. For Fantaisia3D and Magic3D, we use threestudio for fair comparison (Magic3D does not have code available) and the default prompts, which generally believed to work the best with this reimplementation. For CSD, we use the official implementation. As shown in Figure (44 our method improves the visual quality of all the methods by reducing the oversaturated artifacts of SDS and improving the details.
B.3 Ablation Study
Ablation study of the negative prompts. We explore how the choice of negative prompts in our proposed methods affects the optimization. We prompted GPT-4 through ChatGPT single time to generate alternative negative prompts using the following:
Here's a set of "negative prompts" to append to a text-to-image prompt that describe undesirable image characteristics: oversaturated, smooth, pixelated, cartoon, foggy, hazy, blurry, bad structure, noisy, malformed" I want to try variety of them, please bruinstorm many of roughly the same length.
We produce five variants through these methods as the alternative negative prompts:
18

---

(a) SDS + Stage 2 souite prompt (Stage 2 only)
(b) SDS (Stage 1 only)
(c) Ous (Stage + Stage 2)
Figure A6: Ablation study of our method without stage 1. We show directly optimizing with yue from the start can undermine the quality of the geometry and produce unnecessary content.
1. ", washed out, grainy, distorted, flat, smeared, overexposed. undefined, choppy. glitchy, dull"
2. ", low contrast, jumbled, faint, abstract, over-sharpened, muddy, cluttered, vague, jagged, poor detail"
3. ", soft focus, muffled, streaky, patchy, ghosted, murky, unbalanced, skewed, mismatched,
overcrowded"
4. overbright, scrambled, bleary, blocky, misshapen, uneven, fragmented, obscured, chaotic, messy"
5. ", dull tones, compressed, smeary, out of focus, unrefined, lopsided, erratic, irregular, spotty, stark"
We keep other hyperparameters identical and only ablate the negative prompts with the variations. As shown in Figure A5 we do not see obvious differences between our prompts and the variants.
Ablation study stage 2. Instead switching to stage during the optimization process, we ablate with starting without any SDS optimization from the beginning. That is, always use the Yaee with the descriptors " oversaturated, smooth, pixelated, cartoon, foggy, hazy, blurry, bad structure, noisy, malformed". As shown in Figure A6 this makes hard to generate the proper geometry even though the local texture looks reasonable and inclined to produce excessive details that not described by the texts. We suspect that this is because using Voe increases the mismatching error at the beginning of the optimization process when the initialization does not resemble the target prompt all.
C Potential Social Impact
We analyze how to use pre-trained image diffusion as prior in an optimization setup, necessary for domains such as 3D. On the positive side, these models can empower individuals to make 3D content creation more accessibly without requiring specialized skills. Additionally, professional artists and designers could rapidly prototype and visualize their ideas, accelerating the creative process. On the negative side, the ease of generating visual content could facilitate the spread of misinformation. proliferate biases in the training set and enable the usage of generated content for malicious purposes. In addition, there are ethical concerns regarding the potential for job displacement in industries reliant on traditional art-making skills and the copyright issues appeared the training dataset.
19

---

VSD
SDS
Ours
VSD
SDS
Ours
pineapple, detuiled, high resolution, high quality, sharp.
loucan the wood.
A plate piled high with chocolate chip cookies.
llama, detailed, high resolution, high quality, sharp.
A tree of potatocs, detailed, high resolution, high quality, sharp.
An elephant skull.
A solid, smooth, sycanetrical porcelain tcepet, with cobalt blue dragon design, steam rising from the snggesting it's heen filled with hoiling water
large, multi-layerod, symmetrical wedding cake, with smooth Rondant, delicate piping, and lifalike suger flowers full bloom, disployed silver stand.
A model of a house in Tudor style.
A bulldog, detailed, high resolution, high quality, sharp.
walmut, detailed, high resolution, high quality, sharp.
mediuntsezzdd, layered, ridially symmetrical shell, rough the outside, fading from pirk cream, sitting alone sandy beach
Figure A7: Additional comparison of text-guided NeRF optimization. We show more examples compare with different distillation methods, SDS and VSD.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, and we also summarized the contributions of the paper the end of the introduction, which is supported by our analysis in the method section and results in the experiment section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [NA]
Justification: Our theoretical result of score distillation as an optimization that approximates Schrödinger Bridge path is obvious once connect the score distillation formula with the Dual Diffusion Implicit Bridge. We also provide an intuitive error analysis of existing SDS variants.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the details of experiments and will release the code reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
22

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We will release the code with opensource license when the paper is published.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
httppp.cc/pubcccguuuceeee Codeeubmii ssi criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We mention these details the experiment section of the main paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Most of our experiments are too expensive for us run multiple rounds. For example, each run of our text-to-image generation with baseline VSD takes 1.3K GPU hours.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these details in the experiment section as well as Table
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We briefly discuss this in the conclusion section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We use human evaluation for our text-to-3D experiment, and we include the details about how the experimentation was done.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "ht9CkTLsU6", "reviewer": "Reviewer_Cgf2", "summary": "This paper revisits the application of Score Distillation Sampling (SDS) for tasks with limited data availability by proposing a new interpretation based on Schrödinger Bridges for optimal-cost transport between distributions. The paper highlights that existing SDS methods produce artifacts due to linear approximations and poor estimates of source distributions. By aligning the text conditioning more closely with the source distribution's characteristics, the authors demonstrate significant improvements in image generation tasks such as text-to-2D and 3D, and translation between art styles. The proposed method avoids the computational overhead of previous approaches while achieving comparable or superior results in terms of image quality across various domains.", "questions": "See the weakness section for questions. I would like to raise my scores if the above questions are addressed.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Overall I find that the writing is clear, concise, and well-structured, making it easy for readers to follow the arguments and understand the key points. I really like the view of optimal transport between source and target distributions to understand score distillation.\n- This paper provides a comprehensive analysis of existing methods from a unified point of view. They further propose a simple yet effective method in transferring 2D diffusion prior to the 3D scene generation or editing. In contrast to prior state-of-the-art ProlificDreamer, it does not require fine-tuning of diffusion models, which may introduce training inefficiency and instabilities", "weaknesses": "- The description of the proposed method is quite concise. Some technical parts lack enough rationales. For example, estimating the source distribution by negative text prompts lacks the rationales. I believe providing detailed analysis with ablation studies will make the paper more informative to answer the following questions: How do you choose these negative prompts? Does it need to be hand-crafted carefully every time adapting to a new domain/task? Why do you have to propose a two-stage optimization pipeline?\n- How do you choose the value of w? If the value of w is still very large, then the proposed method likely inherits the weaknesses of using a high CFG with standard SDS: less diversity of generations for a given prompt, less realistic generations, and over saturated colors. Besides, if the value of w is still very large, I think the proposed method is more similar to directly using the original negative prompts as guidance.\n- The text-to-3D experimental results are somewhat not convincing since the comparison of both qualitative and quantitative results is inadequate and more competitive baselines should be included, such as Fantasia3D, Magic3D, CSD, NFSD, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. Some technical parts lack enough rationales, such as the negative prompt and two-stage optimization process. How do you choose these negative prompts? \n\nWe propose a two-stage optimization process motivated by our SB framework. To reduce the effect of the distribution mismatch error, we aim to more accurately describe the current distribution across different stages of optimization. While we experimented with more complicated approaches to achieve this, we found prompting to be unreasonably effective and much simpler. The negative prompts were selected based on the characteristic SDS artifacts, e.g., \"oversaturated, smooth, pixelated,..\" (rest in paper). This description does not require hand-crafting based on problem domains—it is kept fixed across all shown examples and use-cases. \n\nWe do not claim that these specific prompts are optimal or even of particular interest. To demonstrate this, we prompted GPT-4 through ChatGPT a single time to generate alternative negative prompts and ablated the results in Figure A2 with otherwise identical hyperparameters.\n\nChatGPT Prompt:\n\n*Here's a set of \"negative prompts\" to append to a text-to-image prompt that describe undesirable image characteristics: \", oversaturated, smooth, pixelated, cartoon, foggy, hazy, blurry, bad structure, noisy, malformed\" I want to try a variety of them, please brainstorm many of roughly the same length*\n\nGenerated Variations:\n\n*1. \", washed out, grainy, distorted, flat, smeared, overexposed, undefined, choppy, glitchy, dull\"*\n\n*2. \", low contrast, jumbled, faint, abstract, over-sharpened, muddy, cluttered, vague, jagged, poor detail\"*\n\n*3. \", soft focus, muffled, streaky, patchy, ghosted, murky, unbalanced, skewed, mismatched, overcrowded\"*\n\n*4. \", overbright, scrambled, bleary, blocky, misshapen, uneven, fragmented, obscured, chaotic, messy\"*\n\n*5. \", dull tones, compressed, smeary, out of focus, unrefined, lopsided, erratic, irregular, spotty, stark\"*\n\n\nIn addition, we show that the same negative descriptors work across different base models, such as MVDream. Since MVDream denoises four camera-conditioned images jointly, we treat the canvas of four images as a single optimization variable for the SDS gradient. In Figure A4-a, we compare the SDS baseline (top) to the proposed two-stage optimization (bottom), in which we generate more natural colors and detail. This is especially noticeable in the grass around the crocodile. Due to the space limit, we will add more results in the revised version.\n\nWe also ablate the proposed two-stage optimization process in the global response. As the source distribution keeps changing along with the optimization process, it is necessary to update the source distribution. We effectively achieve this by first running SDS (stage 1) then updating our source distribution with negative prompts to steer the optimization away from the artifacts (stage 2). In the global response, we show that if we start with such “negative” prompts from the beginning, which do not accurately describe the rendered images at initialization, it causes additional distribution approximation error and fails to generate a plausible object.\n\n> 2. How do you choose the value of w? Is large w value similar with large cfg that causes artifacts?\n\nWe choose $w=40$ to produce a gradient on a similar scale as SDS with CFG scale $=100$. This is because, in text-to-3D, it is crucial to balance many other regularization losses on sparsity, opacity, and so on. Using a similar scale allows us to adopt the same hyperparameters to compare fairly with other SDS variants. In addition, w is different from CFG since SDS also incorporates a term with sampled Gaussian noise, and the CFG term needs to be large enough to make the $(\\epsilon_\\text{tgt}-\\epsilon_\\text{uncond})$ dominant. Instead, $w$ simply scales the gradient, unlike the CFG scale in SDS, which changes the direction of the gradient.\n\nWe also notice that $w$ is relatively robust and gives similar results in a range. Instead, when the CFG scale is small in SDS, the averaging effect of Gaussian noise dominates and creates oversimplified 3D objects. When $w$ or CFG scales are large, with other loss weights intact, the optimization becomes unstable and may diverge. See Figure A5 in the uploaded PDF for this qualitative comparison.\n\n> 3. Missing comparison in text-to-3D with more competitive baselines.\n\nThank you for the suggestion. We ran a comparison with Fantasia3D, Magic3D, and CSD through a drop-in replacement of SDS with our method. We did not compare with NFSD as they did not release the official code and we empirically found that its results resemble SDS results. Specifically, all three methods optimize a textured DMTet, which is initialized from an SDS-optimized NeRF, using SDS or CSD for 5k or 10k iterations. We replace the SDS or CSD stage of these approaches with the two-stage optimization motivated by our framework. Just like our text-to-3D NeRF experiment, we perform the first stage for 60% of iterations and the second stage for 40% of iterations. Note that we keep all the other hyperparameters the same, which were tuned for the baselines, not our method. This replacement leads to the same optimization time as the original methods. For Fantaisia3D and Magic3D, we use threestudio for fair comparison (Magic3D does not have code available) and the default prompts, which are generally believed to work the best with this reimplementation. For CSD, we use the official implementation. As shown in Figure A3, our method improves the visual quality of all the methods by reducing the oversaturated artifacts of SDS and improving the details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8NPczQj8q6", "reviewer": "Reviewer_sBp4", "summary": "This paper proposes interpreting score distillation sampling (SDS), a widely used method for generating 3D, 4D, and vector graphics, through the lens of the Schrödinger Bridge (SB) problem. According to the paper, SDS is a linear approximation of the optimal path moving from the current distribution to the target distribution. The paper identifies two sources of approximation error: first-order approximation error and source distribution mismatch. To address the second error source, the authors suggest a simple method: using a text prompt that describes the source distribution instead of a null prompt. This approach is computationally more efficient than the best variant of SDS, VSD, and the authors demonstrate its effectiveness across various tasks such as text-to-image, text-to-3D, painting-to-real, and illusion generation.", "questions": "- What pre-trained diffusion model did you use in this paper? \n\n- I personally tried text-to-3D with SDS using Diffusion-DPO [A, B], which is a post-trained diffusion model for aesthetic quality, and failed to generate plausible geometry. Can this phenomenon be explained from the perspective of the Schrödinger Bridge problem?\n\n[A] https://huggingface.co/mhdang/dpo-sd1.5-text2image-v1\n\n[B] Wallace et al., Diffusion Model Alignment Using Direct Preference Optimization, CVPR 2024.", "limitations": "The limitations of this paper are stated in section 4, but they need improvement. \n\nAs mentioned in the weaknesses, unlike the method proposed in this paper, VSD can resolve the diversity issue by increasing the number of particles, so line 282 is misleading. \n\nAdditionally, it is necessary to mention the slow generation speed of SDS and the failure to address the Janus problem in text-to-3D, for the benefit of the readers.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The interpretation that SDS finds the optimal path connecting two distributions is novel and aids in understanding the behavior of the widely used SDS.\n\n- The paper addresses the oversaturation problem of conventional SDS and is computationally more efficient than VSD. \n\n- The paper demonstrates effectiveness across a wider range of tasks compared to previous papers (SDS, NFSD, and VSD).\n\n- The paper is well-written and easy to follow.", "weaknesses": "- Up to section 2.3, I enjoyed reading and expected a principled solution. However, the solution in section 2.4 was quite naive and heuristic. A major drawback of this solution is that the pre-trained diffusion model needs to accurately match the proposed descriptions such as \"oversaturated, smooth,…\" with the source distribution. Considering that the text-to-3D experiments are based on Threestudio, my assumption is that the text-to-image model used in this paper is Stable Diffusion 2-base (the exact model is not mentioned in the paper). **It is questionable whether other models (MVDream, SDXL, PixArt, SD3, etc.) can understand these descriptions well.** My guess is that diffusion models trained on such high-quality data will still generate clean images even when descriptions like \"oversaturated, smooth,…\" are appended, and therefore will still suffer from the source distribution mismatch problem.\n\n- The paper does not consider the **Janus problem**, which frequently occurs in text-to-3D. It is questionable whether the proposed methodology would be effective with MVDream [50], a pre-trained diffusion model that addresses this issue. \n\n- **VSD has the strength of** not only ensuring the quality of rendered images but also **achieving sample diversity as the number of particles increases.** Therefore, line 282 is not true, and the proposed method falls behind VSD in terms of sample diversity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. The solution in section 2.4 was quite naive and heuristic. A major drawback is whether other models (MVDream, SDXL, PixArt, SD3, etc.) can understand the descriptions \"oversaturated, smooth,…\" well. \n\nAlthough using negative prompt is a common practice in text-based diffusion models, how to use it with SDS is not well explored. As shown in the global response and Figure 1A, simply using it all the time during the optimization process leads to inferior results. Instead, the two-stage process consistently outperforms the single-stage baselines.\n\nWe show that the prompt generally works well with other models like MVDream and SDXL.\n\nWe perform our 2D generation experiment with MVDream using SDS and our two-stage optimization process. We use the same negative descriptors for the source prompt as proposed in our experiments with Stable Diffusion 2.1. As shown in Figure A4-a, we show that the two-stage optimization (bottom) produces more convincing colors and additional realistic high frequency details compared to an SDS baseline (top) for the same MVDream model.\n\nWe do the same with the SDXL base model, again using the same negative descriptors proposed for Stable Diffusion 2.1. In Figure A4-b, the two-stage optimization process (below) produces less saturation artifacts and more high frequency details than the SDS baseline (above). SDXL is known to perform poorly in the SDS setting and is therefore not commonly used, but we include these results to demonstrate the universality of the proposed optimization.\n\nDespite the fact that these diffusion base models are trained on multi-view or high-quality images whose captions may not contain the proposed negative descriptors, we argue that the powerful pretrained text encoders that embed their prompts represent these artifacts well. For example, the embedding of “oversmoothed” is likely far from the embedding of “detailed” and close to other negative descriptors like “blurry”. Empirically, we find that the same negative descriptors work across base models without needing retuning.\n\n> 2. The paper does not consider the Janus problem.\n\nOur proposed framework is focused on analyzing and improving the diffusion gradient in SDS. MVDream addresses the Janus problem, which requires data priors on objects by training on multi-view data and conditioning generations with camera pose. These problems are orthogonal, and we show that our two-stage optimization process works well with MVDream to address both issues in Figure A4-a.\n\n> 3. VSD has the strength of not only ensuring the quality of rendered images but also achieving sample diversity as the number of particles increases. Therefore, line 282 is not true, and the proposed method falls behind VSD in terms of sample diversity.\n\nThanks, we will make clear the advantage VSD has on sample diversity. In line 282, we mentioned, “...neither approaches have yet to achieve the quality and diversity of images generated by the reverse process.“ We did not intentionally mean that our two-stage optimization achieves better diversity than VSD, but to say, all SDS variants still induce lower diversity than the reverse process. The reason for that is still unclear. We hope that our analysis may inspire future research to better understand this problem.\n\nFor instance, to analyze the diversity issue within our framework, we notice that the ending point of the bridge is deterministically decided by the initial conditions and the ODE processes. When training a LoRA on all the particles, the loss encourages different ODE processes on individual particles. As a result, the LoRA module assigns slightly different directions to each particle and improves diversity. A recent study [1] reinforces this point by introducing a repulsive ensemble method to VSD. In general, this is beyond the scope of our paper. We will add this discussion in the paper.\n\n[1] https://arxiv.org/abs/2406.16683\n\n> Question 1: What pre-trained diffusion model did you use in this paper?\n\nWe use stable-diffusion-v2-1-base for our experiments. We will make this clear in our revised version.\n\n> Question 2: I personally tried text-to-3D with SDS using Diffusion-DPO [A, B], which is a post-trained diffusion model for aesthetic quality, and failed to generate plausible geometry. Can this phenomenon be explained from the perspective of the Schrödinger Bridge problem?\n\nThanks for bringing up this interesting observation. We have also observed that models like SDXL fail to generate reasonable geometry in practice. Similar to Diffusion-DPO, SDXL filters its data using the aesthetic scores. Our hypothesis is that the images with high aesthetic scores overrepresent canonical views of the object. For example, the front view of a dog is often deemed to be more aesthetic than its back view. As a result, this induces an issue that the target distribution of SB heavily biases toward this canonical view. When applying these models to 3D generation, there could be more inconsistency across different views, which makes the optimization less stable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XZzMmCDutC", "reviewer": "Reviewer_XvCb", "summary": "The paper analysis score distillation methods in a single framework and hypotheses about two possible error sources - the single step ODE solver approximation and the mismatch between the assumed and true source image distribution. It then proceeds to tackle the first of them using a custom negative prompt design. The authors show this improves the quality of the generated images in both 2D and 3D tasks as measured by qualitative metrics and in a user study.", "questions": "Q1) The authors use CFG strength s=40 (L483) or even s=100 (L487) to test their method which is a lot more than the range 3-15 considered in the DDS paper [18]. Why the difference?\n\nQ2) The definition of the score function in Eq. 2 differs from [25] and also from what I would expect. It is a bit hard to judge because $\\epsilon_t$ is also not at all defined but I would expect $\\sigma_t$ to play a role, ie. smt like $\\epsilon_t/ \\sigma_T^2$ as in [25, Eq. 3]. Can you please clarify this?", "limitations": "The authors discuss impact of AI art on society and mention that the performance of their method does not match the reverse sampling.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1) The analysis of different methods in a single framework and with illustrations and experiments is quite interesting for a reader and useful for understanding the SotA and its motivation. \n\nS2) The exposition is very clear except for the treatment of the related work discussion.\n\nS3) The proposed negative prompt idea is very effective in the tested scenarios.\n\nS4) Quantitative metrics are complemented by a user study.\n\nS5) The \"3D Sketch-to-Real\" task hidden in the Appendix seems quite original but it would require more examination (and probably a separate paper) to become a substantial contribution on its own.", "weaknesses": "W1) The technical contribution of the paper is limited. Specifically, I see the DDS gradient [18] and the new proposed gradient (eps_ours) as closely related. As far as I can tell, the formulas are the same and the difference is in the choice of the source prompt. Since DDS is presented to work with arbitrary source and target prompt pairs, I feel the new method could be postulated as a specific realization of DDS for one specific source prompt. The authors also omit DDS from all comparisons which supports this viewpoint. The paper still presents a novel analysis of performance in setups that were not shown in the original DDS paper [18] but I am worried the way the method is presented here (\"Ours\") is not optimal from the perspective of originality.\n\nW2) The idea of discouraging artifacts using a negative prompt is likely not a substantial scientific contribution since negative prompts are commonly used by practitioners in image generating interfaces such as Midjourney.\n\nW3) The authors identify two sources of errors - the 1st order approximation and the source distribution match - but they mostly only analyze the latter. The former is speculated to improve performance of ISM but if this hypothesis should be considered seriously, I would expect the multi-step ODE solver to be tested in combination with the various gradient definitions. This could show it in fact helps consistently and how orthogonal it is to the other error type. Currently, not even the basic ISM is included in Table 1 so I find the \"First-order approximation error\" theory plausible but untested.\n\nW4) Other suggestions for the analytical Sec. 2.3: \n\n- The analysis of SDS focuses on cases with s >> 1 but it does not explain the poor performance with s = 1. \n\n- Unlike for SDS, the analysis and Figure for DDS omits the effect of CFG while the original work uses s > 1 ([18]: Eq 3 and Fig. 5). I would expect to see some of the same distribution shifts as in SDS.\n\n\nW5) The related work is deferred to the appendix. The discussion covers the various distillation methods and their applications quite well but due to its location the paper alone is sometimes a bit hard to follow. Therefore, I am personally not very keen of this design choice. Notice that I do not consider this a reason for rejection but I still put it among the negatives since I see it as a minor weakness.\n\nMinor presentation issues and suggestions:\n- [25] is NeurIPS 2022\n- When Fig. 1 is first referenced the meaning of different eps suffixes has not yet been defined in the text.\n\n-----------\n**Justification of recommendation**\n\nOverall this could be a very nice meta-paper on score distillation but to that goal it unfortunately only explores one part of the problem - the score definition - and not the ODE approximation. The negative prompting idea seems effective but not very novel given the common usage of negative prompts in image generation and the similarity of the score gradient to DDS. Therefore I lean towards rejection.\n\n-----------\n**Post rebuttal**\nI increase my rating after the rebuttal because the authors could show that their method is useful also in 3D generation and it also can be combined with the multi-step inversion to demonstrate influence of both limiting factors discussed in the theoretical part. These should be integrated into the paper. I keep my scope just positive since I still see the method as technically very incremental (e.g. the main difference from DDS is just the input image) and I await the discussion of the other reviewers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. The formulas of DDS and the proposed method are the same up to the source prompt.\n\nA small yet fundamental difference is that DDS is a specialized method for editing that computes the source distribution direction based on a reference image instead of the current optimized image. That is, DDS computes the direction to the source distribution as $\\epsilon_\\phi (x_\\text{ref}; y_\\text{src}, t)$ as shown in the equation above line 155, while we compute it as $\\epsilon_\\phi (x_{θ,t}; y_\\text{src}, t)$ as shown in the equation above line 186. This makes DDS incompatible with generation tasks since it requires a reference image. In addition, when there are reference images, it induces additional source distribution error as we explained in line 155.\n\n\n> 2. The idea of using a negative prompt is likely not a substantial scientific\n\nWe emphasize in the global response that our main contribution is to provide a new view of the sources of errors in SDS. To demonstrate that improving the source distribution approximation error can improve generation quality, we propose a two-stage optimization process. Although using negative prompt is a common practice, the proper way to use it with SDS is not well explored. For example, as shown by the two-stage ablation experiment in Figure A1, always using negative prompt in SDS leads to divergence and poor geometry.\n\n> 3. The authors identify two sources of errors - the 1st order approximation and the source distribution match - but they mostly only analyze the latter.\n\nPlease find the qualitative results of ISM in Figure A6. We notice that ISM generally outputs sharper results than SDS. However, it actually still uses single-step estimation to approximate the bridge. Although it uses multi-step inversion to produce a noisy sample instead of adding randomly sampled noise, it computes two epsilon directions using diffusion models to estimate the gradient, similar to SDS. \n\nInstead, here we propose to resolve first order approximation error by solving the entire PF-ODE path to recover the dual bridge and estimate the endpoint of the bridge $\\psi_{0, \\text{tgt}}$ that is coupled with$\\psi_{0, \\text{src}}$. In this way, we obtain the most accurate gradient direction with minimal approximation error $\\psi_{0, \\text{tgt}} - \\psi_{0, \\text{src}}$. We refer to this approach as “full path”. However, solving the inversion ODE is not trivial. We noticed that the inversion can exaggerate the distribution mismatch error and cause the optimization to get stuck at a local optimum at the beginning of the optimization. Instead, the high variance of the single-step methods often demonstrates more robustness to different initializations. Therefore, we first perform the single-step score distillation optimization to obtain reasonable results before moving to solving the full bridge. \n\nWith this approach, we can now explore addressing both the first and second source of error. First source (linear approximation) with “full-path” and second source (source distribution mismatch error) with 2-stage. As shown in the table below, we find that using the “full-path”multi-step (mitigating error source 1) always outperforms the single-step methods, achieving a lower FID. However, the same trend does not fully transfer to the text-to-3D experiments. We observe that it typically introduces additional artifacts and makes the optimization less stable. We leave the best way of leveraging this gradient as a future research exploration.\n\n| Method | Addressing linear approx. error | Addressing dist. mismatch error | FID |\n|---|----|---|---|\n| SDS | No | No  | 79.95 |\n| with two-stage | Yes | No | 69.82 |\n| with full-path PF-ODE | No | Yes | 66.51 |\n| with two-stage & full-path | Yes | Yes  | 62.69 |\n\n\n> 4.a) The analysis of SDS does not explain the poor performance with $s = 1$.\n\nSDS requires $s>>1$ to make $(\\epsilon_\\text{tgt}-\\epsilon_\\text{uncond})$ dominate the gradient direction. The function of the $\\epsilon$ term effectively averages the optimized image by adding different Gaussian noise. As shown in Figure A5 in the uploaded PDF, a small value of $s$ would make the generation over-smoothed and lacking in details.\n\n> 4.b) Unlike for SDS, the analysis for DDS omits the effect of CFG.\n\nWe believe that the effect of changing s is equivalent to changing the learning rate of the optimization, which controls the strength of the editing in DDS without changing the gradient direction. Therefore, we omit it in our gradient analysis.\n\n> 5, 6. The related work is deferred to the appendix. Minor issues.\n\nThank you, We will address these in the revised version.\n\n> Q1. Why use CFG strength $s=40$ or 100 which is more than the range 3-15 in DDS?\n\ns does not affect the direction of the gradient and can be absorbed into the learning rate. In their experiment, learning rate of 0.1 is used while we mostly use a learning rate of 0.01. Equivalently, we can use $w=4$ and learning rate=0.1 which results in a similar scale to the DDS hyperparameters. \n\n> Q2. Can you please clarify the definition of the score function in Eq. 2, compared with [25]?\n\nWe follow the notation in the DDPM paper. Suppose that $\\mathbf{x}_t \\sim \\mathcal{N}(\\sqrt{\\alpha_t}\\mathbf{x}_0; (1-\\alpha_t)\\mathbf{I})$, then the score can be computed as:\n$ p(\\mathbf{x}_t) = \\frac{1}{\\sqrt{2\\pi (1-\\alpha_t)}} \\exp{(-\\frac{1}{2(1-\\alpha_t)}\\cdot (\\mathbf{x}_t - \\sqrt{\\alpha_t}\\mathbf{x}_0)^T(\\mathbf{x}_t - \\sqrt{\\alpha_t}\\mathbf{x}_0))}$\n\n$ \\nabla \\log p(\\mathbf{x}_t) = - \\frac{1}{2(1-\\alpha_t)}\\cdot 2(\\mathbf{x}_t - \\sqrt{\\alpha_t}\\mathbf{x}_0)  $\n\n$ = - \\frac{1}{(1-\\alpha_t)}\\cdot (\\sqrt{\\alpha_t}\\mathbf{x}_0 + \\sqrt{1-\\alpha_t}\\mathbf{\\epsilon}_t - \\sqrt{\\alpha_t}\\mathbf{x}_0) \\text{(Reparametrization trick with } \\mathbf{\\epsilon}_t \\sim \\mathcal{N}(\\mathbf{0}; \\mathbf{I}) \\text{)} $\n\n$ = - \\frac{1}{\\sqrt{1-\\alpha_t}} \\mathbf{\\epsilon}_t $\n\n[25] uses a different noising function, thus a different score function.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their thoughtful feedback. We propose an optimal transport view to understand score distillation, which reviewers “really like” (Cgf2), and find “novel” (sBp4). We provide illustrations and experiments under this single framework, which reviewer XvCb finds “quite interesting … and useful for understanding the state-of-the-art and its motivation.” They also find the method we propose to improve SDS based on this interpretation simple yet effective, (Cgf2, sBp4, XvCb), efficient (Cgf2, sBp4) and explained with clear exposition (Cgf2, sBp4, XvCb).

We want to stress that our primary contribution is the analysis of the sources of error in SDS and its variants (i.e., why it does worse than sampling with reverse process)—forming the hypothesis that accurately expressing the current source distribution is crucial for improving the quality of SDS. We validate this hypothesis with a simple approach that appends negative modifiers to better model the source distribution. While this method is likely useful in itself, it primarily serves as a practical way to empirically support our hypothesis (i.e., since it outperforms baseline methods that less accurately model the source distribution).

Most of the reviewer concerns were centered around the particular design decisions of our experimental optimization approach, or noted similarities with existing methods. In this rebuttal (both here and in the individual reviewer responses) we detail some of the motivations for these design decisions and answer some recurring questions:

Our Schrodinger's Bridge (SB) interpretation presents SDS as transporting the optimization variable from a source distribution toward a target distribution. This interpretation highlights the importance of accurately modeling the source distribution—where inaccuracies may cause the characteristic artifacts of SDS (e.g., saturation, over-smoothing, etc). To validate this, we devise an experimental solution that aims to better model the source distribution at different stages of optimization, and compare it to SDS and its variants. Our experimental optimization procedure has two stages. At early stages of optimization, we use the standard SDS optimization objective, since the source distribution estimated by SDS (i.e., the model’s unconditional prediction) is a reasonably good approximation of the sample distribution at initialization (blob initialization in NeRF optimization or zero initialization for images). At later stages, once the SDS objective has begun to instill many of the characteristic artifacts in the optimized solution—we change to modeling the source distribution with the target scene description, appended with a set of standard negative modifiers that approximately describe the collection of SDS artifacts (“oversaturated, smooth, pixelated, cartoon, foggy, hazy, blurry, bad structure, noisy, malformed,”).  While this descriptor is fixed across all sequences (and therefore does not require per-instance/domain hand-crafting), it much more accurately models the intermediate source distribution and thus we find that it is effective at steering the optimization towards an artifact-free solution. 

SDS by itself is shown to be notably worse in all our comparisons, but one may additionally wonder—why not only use the second stage? As noted above, our objective is to improve the estimate of the source distribution, and at early stages, SDS’s unconditional sample already models the source distribution (random initialization) reasonably well. In fact, the negative prompt modifiers likely model this distribution particularly poorly. We experimented with this variant, and found that optimization either (1) diverges, resulting in an entirely black volume or (2) generates very unreasonable geometry. A three-way comparison is shown in the attached PDF Figure A1, showing that our proposed two-stage approach is clearly better than both (1) only SDS, and (2) only stage-2. 

Our analysis of the error present in SDS includes two potential sources: (1) the first-order approximation error, as well as (2) the source distribution mismatch error described above. SDS incurs the first error by using only a single-step estimate of the Schrodinger Bridge rather than solving it fully through the full-path PF-ODE. To validate that reducing the first source of error can further improve SDS (XvCb), we perform an experiment in which we solve the entire PF-ODE path to recover the dual bridge (instead of using a first-order approximation) and estimate the endpoint of the bridge $\psi_{0, \text{tgt}}$ that is coupled with $\psi_{0, \text{src}}$. We use this endpoint as the target. Although this is slow, it consistently produces better results as shown by the lower COCO-FID scores in the table below. Please see response to XvCb for more details:

| Method                     | Addressing linear approx. error | Addressing dist. mismatch error | FID   |
|----------------------------|---------------------------------|---------------------------------|-------|
| SDS                        | No                              | No                              | 79.95 |
| with two-stage             | Yes                             | No                              | 69.82 |
| with full-path PF-ODE      | No                              | Yes                             | 66.51 |
| with two-stage & full-path | Yes                             | Yes                             | 62.69 |


We have also performed more experiments as suggested, described in the individual responses below. We will add these additional experiments and figures in the revised paper. We hope these additions strengthen the proposed Schrodinger Bridge SDS interpretation, which is our primary contribution.

Meta-review: The paper proposes a new interpretation of generating content using score distillation sampling, that leads to a range of improvements in quality when adjusting existing approaches to this insight.

This submissions received borderline scores. Then, the author response seems to have been effective as two reviewers have increased their score. This indicates authors will be willing and able to produce a much improved final manuscript.

No blocking issues seems to have been brought up during the review and discussion.

An ethics aspects was raised, authors replied, and it seems that this is a simple user study about the fidelity of images, where the users response is not used to train any model or influences the method at all. So no issue here.

The AC did not read the paper or understood in detail, but as scores seem good now, this probably is an accept.
Decision: Accept (poster)