Flow Score Distillation for Diverse Text-to-3D Generation

## Abstract

Abstract Recent advancements in Text-to-3D generation have yielded remarkable progress, particularly through methods that rely on Score Distillation Sampling (SDS). While SDS exhibits the capability to create impressive 3D assets, it is hindered by its inherent maximum-likelihood-seeking essence, resulting in limited diversity in generation outcomes. In this paper, we discover that the Denoise Diffusion Implicit Models (DDIM) generation process ( i.e . PF-ODE) can be succinctly expressed using an analogue of SDS loss. One step further, one can see SDS as a generalized DDIM generation process. Following this insight, we show that the noise sampling strategy in the noise addition stage significantly restricts the diversity of generation results. To address this limitation, we present an innovative noise sampling approach and introduce a novel text-to-3D method called Flow Score Distillation (FSD). Our validation experiments across various text-to-image Diffusion Models demonstrate that FSD substantially enhances generation diversity without compromising quality.

###### Keywords:

## 1 Introduction

In the realm of 3D content creation, a crucial step within the modern game and media industry involves crafting intricate 3D assets. Recently, 3D generation has facilitated the creation of 3D assets with ease. 3D generative models could be trained directly on certain representations (*e.g*. point clouds[, ], voxel[, ] and mesh[]). However, despite the recent efforts of Ojaverse[], 3D data remains relatively scarce, especially when compared to the abundant 2D image data available on the internet. This scarcity constrains the generative capabilities of models trained solely on 3D datasets. Notably, the most prevailing text-to-3D approach is based on Score Distillation Sampling (SDS), proposed by Dreamfusion[] and SJC[]. SDS effectively tackles the scarcity of 3D data by leveraging pretrained 2D text-to-image Diffusion Models, without directly training models on 3D datasets.

Figure: Figure 1: Generation results of FSD and baseline method SDS. FSD uses pretrained text-to-image Diffusion Models to generate realistic 3D models from text prompts. We improve the noise sampling strategy upon SDS and achieve diverse generation results without compromising quality.
Refer to caption: x1.png

Figure: Figure 2: Methods overview of FSD. We propose Flow Score Distillation for text-to-3D generation by lifting a pretrained Diffusion Model. FSD renders an image $\boldsymbol{g}_{\theta}(\boldsymbol{c})$ from the 3D representation and adds noise $\boldsymbol{\epsilon}(\boldsymbol{c})$ to the rendered image. To compute parameter updates according to $L_{\text{FSD}}^{\theta}$, FSD uses a frozen text-to-image Diffusion Model to predict the noise $\boldsymbol{\epsilon}(\boldsymbol{c})$ added on image $\boldsymbol{g}_{\theta}(\boldsymbol{c})$. Similar to SDS[, ], FSD computes $L_{\text{FSD}}^{\theta}$ by an image reconstruction loss between the “clean image” $\hat{\boldsymbol{x}}^{\text{c}}_{t}=\boldsymbol{g}_{\theta}(\boldsymbol{c})$ and “ground-truth image” $\hat{\boldsymbol{x}}_{0}$ predicted by the pretrained Diffusion Model. FSD further adopts timestep annealing schedule and noise sampling strategy. Instead of sampling noise from Gaussian distribution at each step of the optimization like SDS, we generate noise according to the deterministic noise function $\boldsymbol{\epsilon}(\boldsymbol{c})$, which is determined at the beginning of the optimization.
Refer to caption: x2.png

SDS is designed to optimize any representations (*e.g*. Neural Radiance Field[, , ], 3D Gaussian Splatting[], Mesh[, ] or even 2D images) that could render 2D images through probability density distillation[] using the learned score functions from the Diffusion Models. One of the main limitations of current SDS-based methods is that their distillation objectives will maximize the likelihood of the image rendered from the 3D representations, which leads to limited diversity. Additionally, The success of SDS relies heavily on large classifier-free guidance (CFG)[, ]. However, this reliance introduces issues like over-saturation and further restricts the diversity of the generation results. Despite several subsequent efforts[, , , , , , , ] to enhance SDS, the maximum-likelihood-seeking essence of the method remains unchanged. Notably, ProlificDreamer[] introduces Variational Score Distillation (VSD) and uses a fine-tuned Diffusion Model to model distribution on particles, which could alleviate the maximum-likelihood-seeking issues. However, training costs could grow linearly with the particle number of VSD. ESD[] also points out that single-particle VSD is equivalent to SDS, which remains rooted in the essence of maximum likelihood seeking.

In this paper, we present a fresh perspective on SDS by viewing it as a generalized DDIM[] generation process for 3D representations. Specifically, we discovered that the DDIM generation process (*i.e*. PF-ODE []) can be succinctly expressed using an analogue of SDS loss. Surprisingly, by studying the difference between the analogue of SDS loss and the original form of SDS loss, we find that the noise sampling strategy during the noise addition stage appears to be the main cause that drives SDS toward mode-seeking behavior. SDS-based methods typically use random noise sampled from a Gaussian distribution at each optimization step, following the proposal of Dreamfusion[] and SJC[]. However, the variation in sampled noise can lead to varied optimization directions, which may harm the performance of SDS, as observed by ISM[]. As we will show in this work, PF-ODE can be expressed by an analogue of SDS loss that uses the same noise throughout the generation process, which is different from the original proposal of SDS[, ]. Based on this insight, we propose a novel noise sampling strategy to align SDS with the DDIM generation process on 3D representations.

This paper aims to overcome the aforementioned diversity challenge of SDS. We will first reveal an underlying connection between SDS and DDIM on 2D image generation. We will also show that the noise sampling strategy could be the primary factor that leads to the restricted diversity. Based on this insight, we will give our interpretation of SDS. From our novel viewpoint on SDS, we propose a novel approach called Flow Score Distillation (FSD). FSD improves SDS by using a carefully designed noise sampling strategy. We lift our observations on image generation with FSD to 3D by proposing a view-dependent noise function $\boldsymbol{\epsilon}(\boldsymbol{c})$. We conduct validation experiments across various 2D Diffusion Models and demonstrate that FSD can achieve diverse generation outcomes without compromising quality as well as introducing extra training costs. Overall, our contributions can be summarized as follows.

- •
We provide an in-depth analysis of SDS, an effective method in text-to-3D
generation. We present a proposition that reveals the underlying connection between SDS and DDIM. Specifically, the DDIM generation process (*i.e*. PF-ODE) can be succinctly expressed using an analogue of SDS loss. As a result, we can interpret SDS as a generalized DDIM generation process on 3D representations.
- •
Building upon our new insight into SDS, we identify that the diversity degradation stems from the noise sampling strategy. We show that adding the same noise throughout the generative process can enhance both generation quality and diversity in 2D generation with SDS. Hence, we show the noise sampling strategy is the primary factor that restricts the diversity.
- •
We introduce FSD as a cheap but effective solution to tackle the diversity challenges arising from the maximum-likelihood-seeking nature of SDS. We propose a deterministic world-map noise function to generate coarsely aligned noise in 3D space. By applying a reasonable noise sampling strategy, FSD breaks free from the mode-seeking nature of SDS-like methods. Our validation experiments demonstrate that FSD substantially enhances generation diversity without compromising quality.

## 2 Preliminaries and Related Works

In this section, we will provide preliminaries on Diffusion Models, differential equations associated with them, and SDS. Additionally, we will also introduce works related to this paper. Our notation conventions mainly follow Song *et al*. [], Lu *et al*. [] and Poole *et al*. [].

### 2.1 Diffusion Models

Diffusion Models[, , ] are a family of powerful generative models that are trained to gradually transform Gaussian noise to samples from a target distribution $p_{0}$. Their generation ability is further enhanced given datasets comprising billions of image-text pairs[, , ].

Assume the target distribution is $p_{0}$ and condition $y$. Diffusion Models define a forward process $\{x_{t}\}_{t\in[0,T]}$ starting from $\boldsymbol{x}_{0}\sim p_{0}(\cdot|y)$, such that for any $t\in[0,T]$ the distribution of $x_{t}$ conditioned on $x_{0}$ satisfies:

$$ $\boldsymbol{x}_{t}=\alpha_{t}\boldsymbol{x}_{0}+\sigma_{t}\boldsymbol{\epsilon },\quad\boldsymbol{\epsilon}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I}), \quad\boldsymbol{x}_{0}\sim p_{0}(\cdot|y),$ (1) $$

where $\alpha_{t},\sigma_{t}\in\mathbb{R}^{+}$ are functions of $t$, defined by the noise schedule of the Diffusion Model. And the (noisy) distribution at timestep $t$ is noted as $p_{t}$.

In practice, a Diffusion Model is a neural network $\boldsymbol{\epsilon}_{\phi}(\boldsymbol{x}_{t}|y,t)$ parameterized by $\phi$ and is trained by minimizing the following objective[, ]:

$$ $L_{\text{DMs}}^{\phi}=\frac{1}{2}\mathbb{E}_{\boldsymbol{x}_{0}\sim p_{0}( \cdot|y),\boldsymbol{\epsilon},t}\left[w_{t}||\boldsymbol{\epsilon}_{\phi}( \boldsymbol{x}_{t}|y,t)-\boldsymbol{\epsilon}||^{2}_{2}\right],$ (2) $$

where $w_{t}$ is a weighting function. As the training objective implies, a Diffusion Model can be seen as predicting the Gaussian noise added to the clean data $\boldsymbol{x}_{0}\sim p_{0}(\cdot|y)$. Song *et al*. [] proved that:

$$ $\boldsymbol{\epsilon}_{\phi}(\boldsymbol{x}_{t}|y,t)\approx-\sigma_{t}\nabla_{ \boldsymbol{x}}\log p_{t}(\boldsymbol{x}_{t}|y),$ (3) $$

if the Diffusion Model $\boldsymbol{\epsilon}_{\phi}$ is trained to almost optimum. The term $\nabla_{\boldsymbol{x}}\log p_{t}(\boldsymbol{x}_{t}|y)$ in the above equation is also known as the score function.

### 2.2 Diffusion PF-ODE and DDIM

To generate samples from Diffusion Models, there exist different methods among the Diffusion Models family. Denoise Diffusion Implicit Models (DDIM)[] designed a deterministic method for fast sampling from Diffusion Models. Recent works [, , ] have shown that the sampling algorithm of DDIM is a first-order discretization of the Diffusion Probability Flow Ordinary Differential Equation (PF-ODE)[]. Therefore, in this work, we will not make specific distinction between DDIM generation process and PF-ODE.

Theoretically, Diffusion PF-ODE yields the same marginal distribution as the forward process of Diffusion Models (Eq. 1)[]. We can write Diffusion PF-ODE[, ] (see detailed derivation in Appendix) as:

$$ $\displaystyle\frac{\mathrm{d}(\boldsymbol{x}_{t}/\alpha_{t})}{\mathrm{d}t}$ $\displaystyle=\frac{\mathrm{d}(\sigma_{t}/\alpha_{t})}{\mathrm{d}t}\left(- \sigma_{t}\nabla_{\boldsymbol{x}}\log p_{t}(\boldsymbol{x}_{t}|y)\right)$ (4) $\displaystyle=\frac{\mathrm{d}(\sigma_{t}/\alpha_{t})}{\mathrm{d}t}\boldsymbol {\epsilon}_{\phi}(\boldsymbol{x}_{t}|y,t),\quad\boldsymbol{x}_{T}\sim p_{T}( \boldsymbol{x}_{T}|y).$ (5) $$

Notably, one can generate a sample $\boldsymbol{x}_{0}$ in the target distribution $p_{0}(\boldsymbol{x}_{0}|y)$ by following the PF-ODE trajectory from $t=T$ to $t=0$, starting from $\boldsymbol{x}_{T}\sim p_{T}(\boldsymbol{x}_{T}|y)=\mathcal{N}(\boldsymbol{0},
\boldsymbol{I})$.

### 2.3 Score Distillation Sampling

Recently, DreamFusion[] and SJC[] proposed Score Distillation Sampling (SDS) to generate 3D models by optimizing a differentiable 3D representation using priors from text-to-image Diffusion Models. Follow-up works tried to improve upon SDS through various aspects, *e.g*. coarse-to-fine training strategy[, , ], disentangled 2D-3D priors[, , ] and refined formulas[, , , , , , ]. Moreover, due to the lack of comprehensive 3D-aware knowledge, multi-face Janus problem often arises when using SDS []. To mitigate this challenge, one can consider replacing the text-to-image Diffusion Models with Diffusion Models designed for object novel view synthesis[, , , , ] or multi-view Diffusion Models[]. Such an adaptation can significantly alleviate the multi-face Janus problem encountered in 3D generation using SDS.

SDS is first introduced by DreamFusion[] and SJC[] to apply image diffusion priors for 3D generation. SDS can optimize on any representations parameterized by $\theta$, which can render an image $\boldsymbol{g}_{\theta}(\boldsymbol{c})$, given camera parameter $\boldsymbol{c}$. Basically, SDS defines a probability density distillation[] loss, denoted as $L_{\text{SDS}}^{\theta}$, whose gradient writes as follows:

$$ $\displaystyle\nabla_{\theta}L_{\text{SDS}}^{\theta}=$ $\displaystyle\mathbb{E}_{\boldsymbol{c},t}\left[w_{t}\frac{\sigma_{t}}{\alpha_ {t}}\nabla_{\theta}D_{\text{KL}}\left(p_{t}(\boldsymbol{x}_{t}|\boldsymbol{x}_ {0}=\boldsymbol{g}_{\theta}(\boldsymbol{c}))||p_{t}(\boldsymbol{x}_{t}|y) \right)\right]$ (6) $\displaystyle=$ $\displaystyle\mathbb{E}_{\boldsymbol{\epsilon},\boldsymbol{c},t}\left[w_{t} \left(\boldsymbol{\epsilon}_{\phi}(x_{t}|y,t)-\boldsymbol{\epsilon}\right) \frac{\partial\boldsymbol{g}_{\theta}(\boldsymbol{c})}{\partial\theta}\right],$ (7) $$

where ${\epsilon}_{\phi}$ is a text-to-image Diffusion Model and $y$ is the generation condition, *e.g*. text prompts. In order to generate 3D content, SDS needs to go through an optimization process on the 3D representation parameter $\theta$ to generate a single 3D model.

Even though SDS can produce high-fidelity objects, there has been ongoing debate about the underlying theory. Recent works [, ] also show that $L_{\text{SDS}}^{\theta}$ is equivalent to a reconstruction loss:

$$ $L_{\text{SDS}}^{\theta}=\mathbb{E}_{\boldsymbol{\epsilon},\boldsymbol{c},t} \left[\frac{1}{2}w_{t}\frac{\alpha_{t}}{\sigma_{t}}||\hat{\boldsymbol{x}}^{ \text{c}}_{t}-\hat{\boldsymbol{x}}^{\text{gt}}_{t}||_{2}^{2}\right],$ (8) $$

where $\hat{\boldsymbol{x}}^{\text{c}}_{t}=\boldsymbol{g}_{\theta}(\boldsymbol{c})$ and $\hat{\boldsymbol{x}}^{\text{gt}}_{t}=\frac{\boldsymbol{x}_{t}-\sigma_{t}
\boldsymbol{\epsilon}_{\phi}(\boldsymbol{x}_{t}|y,t)}{\alpha_{t}}$ is the one-step “estimated ground-truth image” from Diffusion Models, whose gradient is detached. Some other works[, , ] also tried to explain SDS by analyzing the function of each component of SDS loss. In this paper, we will provide another interpretation of SDS: it can be viewed as a generalized DDIM generation process.

Another simple yet effective technique is to apply timestep annealing trick[, , ], which can improve generation quality significantly. This technique is intuitive because reducing added noise during the latter stages of optimization, enabling models to discern finer details and iteratively improve upon them. Let us denote the time in the SDS optimization process as $\tau$ and the term that was taken expectation in the definition of SDS (Eq. 8) as $L_{\text{sds}}^{\theta}(\boldsymbol{\epsilon},\boldsymbol{c},t)=\frac{1}{2}
\frac{\alpha_{t}}{\sigma_{t}}||\hat{\boldsymbol{x}}^{\text{c}}_{t}-\hat{
\boldsymbol{x}}^{\text{gt}}_{t}||_{2}^{2}$. We use lowercase letters footnote for $L_{\text{sds}}^{\theta}$ to distinguish it from Eq. 8. Finally, the SDS optimization process with timestep annealing can be written as:

$$ $\frac{\mathrm{d}\theta}{\mathrm{d}\tau}=w_{t}\mathbb{E}_{\boldsymbol{\epsilon} ,\boldsymbol{c}}\left[\nabla_{\theta}L_{\text{sds}}^{\theta}(\boldsymbol{ \epsilon},\boldsymbol{c},t=t(\tau))\right],$ (9) $$

where $t(\tau)$ is a monotonically decreasing function of $\tau$.

## 3 Flow Score Distillation for 2D Generation

Figure: SDS[, ]
Refer to caption: extracted/5760085/assets/sds-butterfly-42.png

In this section, we only consider generation on 2D using SDS as SDS loss can also be applied to image representations. In this case, $\theta=\boldsymbol{g}_{\theta}(\boldsymbol{c})$ and $\frac{\partial\boldsymbol{g}_{\theta}(\boldsymbol{c})}{\partial\theta}=
\boldsymbol{I}$. Then $L_{\text{sds}}^{\theta}$ becomes the following form:

$$ $\nabla_{\theta}L_{\text{sds-2d}}^{\theta}(\boldsymbol{\epsilon},t)=\boldsymbol {\epsilon}_{\phi}(x_{t}|y,t)-\boldsymbol{\epsilon}.$ (10) $$

We will reveal a simple but profound connection between SDS and DDIM and give our interpretation of SDS in this section.

### 3.1 Simplified Formulation of Diffusion PF-ODE

We first reveal that PF-ODE (Eq. 5) can be formulated by an analogue of SDS (Eq. 10) in this section. We first define

$$ $\boldsymbol{x}_{t}=\alpha_{t}\hat{\boldsymbol{x}}^{\text{c}}_{t}+\sigma_{t} \tilde{\boldsymbol{\epsilon}},$ (11) $$

where $\tilde{\boldsymbol{\epsilon}}$ is a constant for each ODE trajectory. Notice that when $t=T$, the initial condition of the ODE gives $\tilde{\boldsymbol{\epsilon}}=0\cdot\hat{\boldsymbol{x}}^{\text{c}}_{t}+1\cdot
\tilde{\boldsymbol{\epsilon}}=\boldsymbol{x}_{T}\sim\mathcal{N}(\boldsymbol{0}
,\boldsymbol{I})$. Intuitively, $\tilde{\boldsymbol{\epsilon}}$ can be viewed as the noise added to $\hat{\boldsymbol{x}}^{\text{c}}_{t}$, and $\hat{\boldsymbol{x}}^{\text{c}}_{t}$ as the clean image at timestep $t$. So we will also refer to $\tilde{\boldsymbol{\epsilon}}$ as the initial noise apart from the added noise in this paper. It is noteworthy that the concept of the clean image $\hat{\boldsymbol{x}}^{\text{c}}_{t}=\frac{\boldsymbol{x}_{t}-\sigma_{t}\tilde{
\boldsymbol{\epsilon}}}{\alpha_{t}}$ is different from the aforementioned estimated ground-truth image $\hat{\boldsymbol{x}}^{\text{gt}}_{t}=\frac{\boldsymbol{x}_{t}-\sigma_{t}
\boldsymbol{\epsilon}_{\phi}(\boldsymbol{x}_{t}|y,t)}{\alpha_{t}}$. By applying “change-of-variable” trick and change the variable of the Diffusion PF-ODE from $\boldsymbol{x}_{t}$ to $\hat{\boldsymbol{x}}^{\text{c}}_{t}$, we have:

###### Proposition 1 (An equivalent form of Diffusion PF-ODE)

Diffusion PF-ODE (Eq. 5) can be equivalently formulated by an analogue of SDS loss (Eq. 10):

$$ $\displaystyle\frac{\mathrm{d}\hat{\boldsymbol{x}}^{\text{c}}_{t}}{\mathrm{d}t}$ $\displaystyle=\frac{\mathrm{d}(\sigma_{t}/\alpha_{t})}{\mathrm{d}t}\left[ \boldsymbol{\epsilon}_{\phi}(\boldsymbol{x}_{t}|t,y)-\tilde{\boldsymbol{ \epsilon}}\right]$ (12) $\displaystyle=w_{t}^{\prime}\nabla_{\theta}L_{\text{sds-2d}}^{\theta}(\tilde{ \boldsymbol{\epsilon}},t),$ (13) $$

where $\boldsymbol{x}_{t}=\alpha_{t}\hat{\boldsymbol{x}}^{\text{c}}_{t}+\sigma_{t}
\tilde{\boldsymbol{\epsilon}}$, $\theta=\hat{\boldsymbol{x}}^{\text{c}}_{t}$ and $w_{t}^{\prime}=\frac{\mathrm{d}(\sigma_{t}/\alpha_{t})}{\mathrm{d}t}$ is a weighting scalar.

Please refer to Appendix for detailed derivation of this proposition. Remarkably, in the context of image generation, we observe that the evolution direction of PF-ODE aligns precisely with the gradient of the SDS loss (Eq. 10).

###### Proposition 1 (An equivalent form of Diffusion PF-ODE)

### 3.2 Flow Score Distillation on 2D

Even though we found the evolution direction of Diffusion PF-ODE (Eq. 12) is very samilar to the SDS loss, there exist some notable differences compared to the original definition of SDS loss (Eq. 6).
Specifically,
i) the timestep in a DDIM process is monotonically decreasing, aligning with the timestep annealing technique[, , ]. But SDS uses randomly sampled timestep.
ii) And the change-of-variable trick (Eq. 11) we used during our simplification process implies we should also add the same noise $\tilde{\boldsymbol{\epsilon}}$ throughout the SDS generation process, to align it with DDIM. In contrast, the original SDS uses uncorrelated random noise.

As we will demonstrate in subsequent sections and through our experiments, the second difference between DDIM and SDS significantly influences generation diversity. Therefore, we term our approach that combines timestep annealing and consistent noise sampling strategy throughout the generation process as Flow Score Distillation (FSD) to differ it from SDS. We visualize image generation results using several SDS-like methods, FSD and DDIM in Fig. 3 to demonstrate the differences between SDS-based methods and FSD.

### 3.3 Analysis of the Noise Sampling Strategy

Figure: Figure 4: Visualization of FSD and SDS for image generation. We visualize generation results and the estimated ground-truth images in consecutive steps at halfway of the generation for both FSD and SDS[, ]. We find the estimated ground-truth images of FSD are consistent, while the contents vary greatly in the ground-truth images of SDS. We set CFG=7.5 and adopted the same linear timestep annealing schedule for both FSD and SDS in the experiment for this figure.
Refer to caption: x3.png

Our empirical investigation reveals that the generation results produced by SDS exhibit an undesirable tendency toward over-smoothness and lack of diversity. Remarkably, even with the timestep annealing technique, this issue persists. This tendency originates from the maximum-likelihood-seeking nature implied by its definition (Eq. 6) where it models the current distribution using a Dirichlet function centered at $\boldsymbol{g}_{\theta}(\boldsymbol{c})$. In contrast, not only does FSD yield diverse outcomes, but it can also generate highly detailed samples. This can be attributed to that FSD is more aligned with a DDIM process, which can generate samples from exactly the target distribution $p_{0}$. So we conclude that the noise sampling strategy can affect the generation diversity greatly.

To gain an intuitive understanding of these effects, we also explain the reasons by following the reconstruction loss interpretation [, ]. SDS loss can be seen as the reconstruction error between the rendered images and the one-step estimated ground-truth images predicted by the Diffusion Models. When different noise is added, the generated ground-truth images differ greatly (Demonstrated in Fig. 4), as noted by ISM[]. As a result, SDS can only generate averaged outcomes. In contrast, applying identical noise at each optimization step ensures a more consistent optimization trajectory. So FSD can generate diverse results.

### 3.4 Compare FSD with SDS and DDIM on 2D

We summarize the difference between FSD, SDS, and DDIM when applied to 2D images as follows:

#### 3.4.1 Noise Sampling Strategy

Both FSD and DDIM add fixed noise sampled from $\mathcal{N}(\boldsymbol{0},\boldsymbol{I})$ at the beginning of optimization to the clean image $\hat{\boldsymbol{x}}^{\text{c}}_{t}$. While SDS adds random noise sampled from $\mathcal{N}(\boldsymbol{0},\boldsymbol{I})$ at each step of the optimization to the clean image $\hat{\boldsymbol{x}}^{\text{c}}_{t}$.

#### 3.4.2 Optimizer

Both FSD and SDS use Adam[] (or Adan[]) to update the clean image parameter $\hat{\boldsymbol{x}}^{\text{c}}_{t}$. Moreover, the impact of the weighting parameter $w_{t}$ is negligible as long as it changes slowly over $t$ since Adam optimizer is invariant to the scale of the gradients[]. While DDIM updates on $\boldsymbol{x}_{t}$ directly according to the first-order discretization of PF-ODE.

#### 3.4.3 Diffusion Timestep Schedule

FSD tries to model the schedule of DDIM by using a monotonically decreasing function $t(\tau)$. While SDS uses time schedule which samples $t$ uniformly. Moreover, the convergence time $\tau_{\text{end}}$ of FSD and SDS is hard to determine theoretically, but DDIM will converge when $t=T$.

## 4 Lifting Flow Score Distillation to 3D

Figure: Figure 5: Impact of initial noise $\tilde{\boldsymbol{\epsilon}}$. Experiments show that the local textures of noise added during FSD optimization are highly correlated with the textures of the final image. We shuffle the patches of initial noise $\tilde{\boldsymbol{\epsilon}}$ used by FSD and observe that the textures of generated images are shuffled in the same way. This property inspired our design of world-map noise function $\boldsymbol{\epsilon}(\boldsymbol{c})$ for 3D generation in this work. In this figure, the parts framed by dotted lines of the same color share the same initial noise $\tilde{\boldsymbol{\epsilon}}$ patches.
Refer to caption: x4.png

As highlighted in Sec. 3.3, we have identified that the noise sampling strategy might contribute to the decline of the diversity of SDS significantly. Building upon this insight, we follow the discussion of FSD in Sec. 3.2 and propose to use deterministic noise generation strategy. We can directly generalize FSD to arbitrary 3D representations $\boldsymbol{g}_{\theta}(\boldsymbol{c})$:

$$ $\displaystyle\nabla_{\theta}L_{\text{FSD}}^{\theta}=$ $\displaystyle\mathbb{E}_{\boldsymbol{c}}\left[\nabla_{\theta}L_{\text{sds}}^{ \theta}(\boldsymbol{\epsilon}=\boldsymbol{\epsilon}(\boldsymbol{c}), \boldsymbol{c},t=t(\tau))\right]$ (14) $\displaystyle=$ $\displaystyle\mathbb{E}_{\boldsymbol{c}}\left[\left(\boldsymbol{\epsilon}_{ \phi}(x_{t}|y,t(\tau))-\boldsymbol{\epsilon}(\boldsymbol{c})\right)\frac{ \partial\boldsymbol{g}_{\theta}(\boldsymbol{c})}{\partial\theta}\right],$ (15) $$

where $\boldsymbol{x}_{t}=\alpha_{t}\boldsymbol{g}_{\theta}(\boldsymbol{c})+\sigma_{t
}\boldsymbol{\epsilon}(\boldsymbol{c})$, $\boldsymbol{\epsilon}(\boldsymbol{c})$ is a deterministic noise function generated at the beginning of the optimization and $t(\tau)$ is a monotonically decreasing timestep schedule function to optimization time $\tau$. We do not specify the form of the deterministic noise function $\boldsymbol{\epsilon}(\boldsymbol{c})$ in FSD. However, we propose some rules for designing $\boldsymbol{\epsilon}(\boldsymbol{c})$ based on the actual generation effect in Appendix, according to our practical experiences. We will also introduce the world-map noise function as $\boldsymbol{\epsilon}(\boldsymbol{c})$ for our experiments of 3D generation with FSD in this paper.

Alternatively, FSD loss can be seen as applying DDIM generation process on 3D representations through Jacobian of a differentiable renderer. This viewpoint shares some similarities to the interpretation of SDS from SJC[], who consider SDS as back-propagating the score[] of Diffusion Models through Jocabian of the renderer. Meanwhile, $\boldsymbol{\epsilon}(\boldsymbol{c})$ is a deterministic noise function that generates correlated noise between views, which aligns with the prior that the nearby views are correlated. The design of FSD ensures the optimization directions are consistent, particularly when similar $\boldsymbol{c}$s are sampled. As the $\boldsymbol{\epsilon}(\boldsymbol{c})$s are similar, the generated ground-truth images should be consistent as well. Notably, recent works[, , ] on video generation also find using designed video noise prior can improve the capabilities of Video Diffusion Models.

### 4.1 Designing ϵ ⁢ ( 𝒄 ) bold-italic-ϵ 𝒄 \boldsymbol{\epsilon}(\boldsymbol{c}) bold_italic_ϵ ( bold_italic_c ) .

#### 4.1.1 Failure of a Vanilla Design of ϵ ⁢ ( 𝒄 ) bold-italic-ϵ 𝒄 \boldsymbol{\epsilon}(\boldsymbol{c}) bold_italic_ϵ ( bold_italic_c )

A vanilla design of $\boldsymbol{\epsilon}(\boldsymbol{c})$ can be $\boldsymbol{\epsilon}(\boldsymbol{c})=\boldsymbol{\epsilon}$, which is a constant function. However, according to our experiments on text-to-3D generation, such a design can lead to poor geometry of the generated samples. Typically, holes on the surfaces are observed (Fig. 6). We attribute this effect to the uneven convergence speed of FSD in 3D space caused by the constant noise function. Our experiments on 2D show that the local textures of noise added during FSD optimization are highly correlated with the local textures of the final image (Demonstrated in Fig. 5). And in text-to-3D generation with FSD, the generated ground-truth images have more consistent textures at the center point than other points in 3D space, due to the sampling strategy of camera view $\boldsymbol{c}$. As a result, the convergence speed at the center point is much higher than at other points, leading to holes on the surfaces. Flaws are also observed in Video Diffusion Models that adopt fixed noise prior, due to similar reasons, which is known as the textures sticking problem[].

#### 4.1.2 World-map Noise Function ϵ ⁢ ( 𝒄 ) bold-italic-ϵ 𝒄 \boldsymbol{\epsilon}(\boldsymbol{c}) bold_italic_ϵ ( bold_italic_c ) .

Figure: Figure 6: Compare world-map noise function $\boldsymbol{\epsilon}(\boldsymbol{c})$ with a vanilla design of constant function $\boldsymbol{\epsilon}$. We visualize the rendered images and depth maps for the two noise methods. The vanilla design of noise function $\boldsymbol{\epsilon}$ can easily lead to holes on the surfaces (framed in red), even when the RGB images seem plausible. In contrast, no obvious flaws are observed in the results of FSD with the world-map noise function.
Refer to caption: x5.png

To avoid relating specific noise textures to specific points in 3D space throughout the generation process but still augment the consistency of added noise between camera views, we propose world-map noise function $\boldsymbol{\epsilon}(\boldsymbol{c})$ in this paper (visualized in Fig. 2), which aligns noise textures coarsely in 3D space while avoiding converging too fast at a specific point in 3D space (See discussion in Appendix).

Specifically, we first compute a foreground mask $\boldsymbol{M}(\boldsymbol{r})=[\alpha(\boldsymbol{r})>\alpha_{0}]$ for each ray $\boldsymbol{r}$, where $0\leq\alpha(r)$, $\alpha_{0}\leq 1$ is the opacity of pixel $\boldsymbol{r}$. We add the same noise for the background and query a noise patch from the noise world-map $\boldsymbol{\epsilon}_{p}$ of size $D\times H\times W$ for the foreground. Let us denote the camera parameters sampled on the sphere as $\boldsymbol{c}=(\text{FOV},r_{\text{cam}},\theta_{\text{cam}},\phi_{\text{cam}})$. Both $\boldsymbol{\epsilon}_{b}$ and $\boldsymbol{\epsilon}_{p}$ are sampled from $\mathcal{N}(\boldsymbol{0},\boldsymbol{I})$ at the beginning of the optimization. And $\boldsymbol{\epsilon}$ is re-sampled from $\mathcal{N}(\boldsymbol{0},\boldsymbol{I})$ at each step of the optimization. The world-map noise function $\boldsymbol{\epsilon}(\boldsymbol{c})$ is:

$$ $\boldsymbol{\epsilon}(\boldsymbol{c})=\sqrt{\beta}\cdot\left((1-\boldsymbol{M} )\odot\boldsymbol{\epsilon}_{b}+\boldsymbol{M}\odot\boldsymbol{W}_{(W\frac{ \phi_{\text{cam}}}{2\pi},H\frac{\theta_{\text{cam}}}{\pi})}(\boldsymbol{ \epsilon}_{p})\right)+\sqrt{1-\beta}\cdot\boldsymbol{\epsilon},$ (16) $$

where $\beta$ is a blending factor to balance between random noise and deterministic noise. If not specified, we set $\beta=1$ in this work so that $\boldsymbol{\epsilon}(\boldsymbol{c})$ is a completely deterministic noise function. $\boldsymbol{W}_{(W\frac{\phi_{\text{cam}}}{2\pi},H\frac{\theta_{\text{cam}}}{
\pi})}(\boldsymbol{\epsilon}_{p})$ operation refers to the noise patch of size $D\times H_{\text{hidden}}\times W_{\text{hidden}}$ centered at position $(W\frac{\theta_{\text{cam}}}{2\pi},H\frac{\phi_{\text{cam}}}{\pi})$ on the noise worldmap $\boldsymbol{\epsilon}_{p}$. We also provide a pseudocode of our algorithm in Appendix.

### 4.2 Compare FSD with SDS on 3D

Apart from timestep annealing trick[, , ], FSD is different from SDS in terms of noise sampling strategy as well. In case when the same camera view $\boldsymbol{c}$s are sampled, FSD yields consistent one-step estimated ground-truth images since $\boldsymbol{\epsilon}(\boldsymbol{c})$ is the same. Even when different camera views $\boldsymbol{c}$s are sampled, the ground-truth images are still coarsely aligned. In contrast, one can see SDS as using an uncorrelated noise prior $\boldsymbol{\epsilon}(\boldsymbol{c})$ on $\boldsymbol{c}$, which always yields ground-truth images that are inconsistent and have notable differences. We also discuss the differences between our method with recent works [, ] in Appendix 0.B.

## 5 Experiments

Figure: (a) Stable Diffusion[] as backbone Diffusion Model
Refer to caption: x6.png

### 5.1 Implementation Details

Unless otherwise specified, our experiments are conducted
with the threestudio codebase[]. We mainly conduct text-to-3D experiments using two different categories of pretrained Diffusion Models. Specifically, Stable Diffusion[] that could generate a single image from the text prompt and MVDream[] that could generate multi-view images of an object from the text prompt. We apply several tricks that we found helpful to improve generation quality on both baselines and our method.
We use random seeds from 0 to 3 for each prompt by default to demonstrate the diversity of generated samples. Please refer to Appendix for more implementation details.

### 5.2 Evaluation on Text-to-3D Generation

#### 5.2.1 Stable Diffusion.

We visualize our experiment results of SDS[, ] (baseline) and FSD in Fig. 7(a) using Stable Diffusion[] (Fig. 7(a)). Due to single-image Diffusion Models’ lack of 3D knowledge, both SDS and FSD may have multi-face Janus problem[] and can not generate plausible results for all random seeds. So the generation results are usually cherry-picked in previous works, which may be potentially unfair. Instead of cherry-picking on generated samples, we manually pick some prompts on which SDS-like methods may not suffer from multi-face Janus problem in this experiment.

#### 5.2.2 MVDream.

We visualize our experiment results of SDS [, ] (baseline) and FSD (ours) using MVDream[] in Fig. 7(b). From our practical observations, MVDream usually yields degraded generation results compared with Stable Diffusion[] but seldomly suffers from multi-face Jauns problem. As a result of the degradation, text-to-3D generation results of MVDream are almost identical with different random seeds when using SDS. However, FSD can still generate diverse results with MVDream.

#### 5.2.3 Quantitative Results.

Please refer to Appendix.

Figure: Figure 8: Ablation on blending factor $\beta$. parameter $\beta$ in the world-map noise function (Eq. 16) is designed for balance between FSD ($\beta=1$) and SDS ($\beta=0$). We set $\beta=1$ in the other part of this work and show results of FSD with different $\beta$s in this figure.
Refer to caption: x8.png

### 5.3 Ablation Study

#### 5.3.1 Ablation on blending factor β 𝛽 \beta italic_β .

$\beta$ in Eq. 16 is the blending factor designed for balancing between the deterministic noise of FSD and the random noise of SDS. In the other part of this work, we fix $\beta=1$, which means the noise function $\boldsymbol{\epsilon}(\boldsymbol{c})$ is completely deterministic. We show the impact of beta in Fig. 8.

#### 5.3.2 Ablation on other hyperparameters in ϵ ⁢ ( 𝒄 ) bold-italic-ϵ 𝒄 \boldsymbol{\epsilon}(\boldsymbol{c}) bold_italic_ϵ ( bold_italic_c ) .

We use a parameter $\Theta$ to control the size ($H$ and $W$) of noise world-map in Eq. 16. $H$ and $W$ are determined by $\Theta$ according to $H=H_{\text{hidden}}\frac{\pi}{\Theta}$ and $W=W_{\text{hidden}}\frac{2\pi}{\Theta}$. In practice, we found FSD are prone to the parameter $\Theta$. See details in Appendix.

## 6 Conclusion

#### 6.0.1 Conclusions.

In this work, we systematically study the problem of text-to-3D generation. We first review the theorems of SDS and reveal a simple but profound underlying connection between DDIM and SDS. Following this insight, we propose FSD to tackle the diversity degradation challenge. By using a consistent noise sampling schedule that aligns noise coarsely in 3D space, FSD breaks free from the maximum-likelihood-seeking nature of SDS and could generate diverse results without compromising generation quality.

#### 6.0.2 Limitations and future works

Although FSD could improve the diversity of 3D generation, we found that it is still difficult to generate 3D models as diverse as the images generated through DDIM generation process. Moreover, we only found plausible theorems for FSD-guided 2D generation, and generalized FSD to 3D generation in an intuitive way. As we do not specify the noise function $\boldsymbol{\epsilon}(\boldsymbol{c})$ in the general form of FSD (Eq. 15), we design $\boldsymbol{\epsilon}(\boldsymbol{c})$ manually to coarsely align noise prior in 3D space in this work, where better designs of $\boldsymbol{\epsilon}(\boldsymbol{c})$ may exist. We leave this for future work.

## Appendix

## Appendix 0.A Quantitative Results

We compute several metrics for 3D generation results with SDS[, ] and FSD. The hyperparameters are kept the same as those in Appendix 0.C. When using Stable Diffusion[] as backbone, we use 16 prompts and 4 random seeds for each prompt (Tab. 1). When using MVDream[] as backbone, we also use 16 prompts and 4 random seeds for each prompt (Tab. 2).

**Table 1: Stable Diffusion[] as backbone**
| Method | CLIP ($\uparrow$) | IS ($\uparrow$) | CROSS-FID ($\uparrow$) |
| --- | --- | --- | --- |
| DDIM Images | $33.72\pm 1.83$ | $1.68\pm 0.55$ | - |
| SDS[, ] | $32.57\pm 1.43$ | $1.58\pm 0.47$ | $106.5\pm 58.3$ |
| FSD(ours) | $\mathbf{32.72\pm 1.56}$ | $\mathbf{1.78\pm 0.49}$ | $\mathbf{141.8\pm 57.9}$ |

**Table 2: MVDream[] as backbone**
| Method | CLIP ($\uparrow$) | IS ($\uparrow$) | CROSS-FID ($\uparrow$) |
| --- | --- | --- | --- |
| DDIM Images | $34.64\pm 2.56$ | $2.02\pm 0.47$ | - |
| SDS[, ] | $\mathbf{30.93\pm 3.40}$ | $1.77\pm 0.37$ | $86.6\pm 33.4$ |
| FSD(ours) | $30.12\pm 3.07$ | $\mathbf{2.13\pm 0.35}$ | $\mathbf{174.8\pm 44.5}$ |

#### 0.A.0.1 CLIP score

We compute CLIP score[, ] using ViT-B/32 to measure the semantic similarity between the renderings of the generated 3D object and the input text prompt. We sample 24 views for each prompt and each seed when computing CLIP score.

#### 0.A.0.2 IS score

We compute IS score[] to measure both the image quality and diversity. We first compute the IS scores of sampled views for each prompt and then average the IS scores across prompts.

#### 0.A.0.3 CROSS-FID score

To directly measure the diversity of generation results, it is natural to measure the inception distance between different generated samples. We first sample 24 views for each prompt and each seed. Then we separate the images corresponding to random seeds 0, 1 and 2, 3 into two sets of images. We compute FID[] of the two sets of images and average the FID score across prompts. We term this score as CROSS-FID score since it is different from the standard way of using FID to evaluate GANs[].

## Appendix 0.B Discussions

#### 0.B.0.1 Difference with Consistent3D

Recent work Consistent3D [] also applied fixed noise when conducting SDS-like generation. In Consistent3D, they follow the idea of Consistent Training [] and use the rendered image perturbed with fixed noise to approximate the starting point of the deterministic flow. In our method, for the same camera view, we also add fixed noise to the rendered image, but the noised image is used to simulate a variable in the middle of a PF-ODE trajectory, which is different from Consistent3D. Our FSD loss is also different from the CDS loss in Consistent3D, even when our view-dependent noise function gives the same noise for all camera views, implying an essential difference between our method and Consistent3D.

#### 0.B.0.2 Connection to Signal-ODE

Our reformulated ODE (Eq. 12) is equivalent to the Signal-ODE presented in the concurrent and independent work BOOT [], which aims to distill a fast image generator. When the diffusion model is changed to sample prediction in Eq. 12, our reformulated PF-ODE is the same as the Signal-ODE in BOOT. In BOOT, they let the student image generation model predict the clean variables $\hat{\boldsymbol{x}}^{\text{c}}_{t}$ on the ODE trajectory, while our method uses images rendered from 3D representation $\theta$ to model the clean variables $\hat{\boldsymbol{x}}^{\text{c}}_{t}$ on the ODE trajectory.

## Appendix 0.C Additional Experiment Results

### 0.C.1 Additional Generation Results

Figure: Figure 9: Comparisons between FSD and SDS-like baseline methods. We present results of FSD and compare them to baseline methods, namely ProlificDreamer (VSD)[], SDS[, ], CSD[]. We use random seeds from 0 to 3 for each prompt.
Refer to caption: x9.png

Figure: Figure 10: Experiment results of FSD and SDS[, ] with Stable Diffusion[] as backbone.
Refer to caption: x10.png

Figure: Figure 11: Experiment results of FSD and SDS[, ] with MVDream[] as backbone.
Refer to caption: x11.png

We provide additional 3D experiment results using FSD and other SDS-like methods in Fig. 9, Fig. 10 and Fig. 11.

Specifically, in Fig. 9, we present results of FSD and compare them to baseline methods, namely ProlificDreamer (VSD)[], SDS[, ], CSD[]. In Fig. 10, we provide additional comparisons between SDS and FSD with Stable Diffusion[] as the backbone Diffusion Model. In Fig. 11, we provide additional comparisons between SDS and FSD with MVDream[] as the backbone Diffusion Model. We do not provide experiment results with Diffusion Models designed for novel view synthesis (*e.g*. SyncDreamer[]) since we found FSD does not improve the diversity of the back view of an object significantly given the front view when using Diffusion Models of such category.

### 0.C.2 Additional Ablations

Figure: $\Theta=120^{\circ}$
Refer to caption: extracted/5760085/assets/mvdream-theta/120.png

We use a parameter $\Theta$ to control the noise world-map’s size ($H$ and $W$). $H$ and $W$ are determined by $\Theta$ according to $H=H_{\text{hidden}}\frac{\pi}{\Theta}$ and $W=W_{\text{hidden}}\frac{2\pi}{\Theta}$. We visualize the results corresponding to different $\Theta$s in Fig. 12.

Figure: (a) vanilla constant noise function
Refer to caption: extracted/5760085/assets/vanilla/0.png

We also provide an additional comparison between the vanilla constant noise function and the world-map noise function in Fig. 13. We find the constant noise function may harm the geometry of the generation results.

### 0.C.3 Additional Experiments on Noise Prior

Figure: Figure 14: Impact of initial noise $\tilde{\boldsymbol{\epsilon}}$. We provide additional 2D experiment results generated by FSD show the impact of the initial noise $\tilde{\boldsymbol{\epsilon}}$. The patches framed with the same color in the same row share the same initial noise patches.
Refer to caption: x12.png

We provided additional generation results of FSD on 2D image space with shuffled or flipped initial noise in Fig. 14. The patches framed with the same color in the same row share the same initial noise patches $\tilde{\boldsymbol{\epsilon}}$.

## Appendix 0.D Implementation Details

### 0.D.1 3D Experiments using FSD

The backbone Diffusion Model for Fig. 1 in the main text is MVDream[]. The configuration for this figure is the same as Fig. 7 in the main text.

We use sqrt-annealing proposed by HiFA[] for 3D experiments in this work. We use the same CFG[] scale for SDS[, ] and FSD. We use CFG=1 when applying image loss trick[] since we find the reconstructed images are unrealistic when the CFG scale is large, which is not good guidance. We follow the default setting of threestudio[] code base for other hyperparameters. We keep the iteration number and learning rate of FSD the same as SDS[, ] in 3D experiments. We mainly conduct our experiments using NeRF representation[] since SDS-like methods are not sensitive to the form of 3D representations.

### 0.D.2 2D Experiments using FSD

**Table 3: Implementation details on 2D experiments with FSD.**
| Methods Name | SDS[, ] | NFSD[] | VSD[] | FSD(ours) | DDIM[] |
| --- | --- | --- | --- | --- | --- |
| Iteration Num | 500 | 500 | 500 | 500 | 50 |
| CFG[] | 100 | 7.5 | 7.5 | 7.5 | 7.5 |
| Learning Rate | 2e-2 | 2e-2 | 2e-2 | 3e-3 | - |
| Optimizer | Adam[] | Adam | Adam | Adam | - |
| Timestep Annealing | linear | linear | linear | linear | - |

Here we describe the details of the experiments on 2D images with FSD in the main text. For Fig. 3 in the main text, we show the implementation details in Tab. 3. Below, we provide the loss functions for convenient reference.

Let denote the prediction of Diffusion Models as $\boldsymbol{\epsilon}_{y}^{t}=\boldsymbol{\epsilon}_{\phi}(\boldsymbol{x}_{t}|
y,t)$ and $c$ the classifier-free-guidance (CFG)[] scale. Then

$$ $\nabla_{\theta}L_{\text{SDS}}^{\theta}=\mathbb{E}_{\boldsymbol{\epsilon}, \boldsymbol{c},t}\left[\left(c\cdot(\boldsymbol{\epsilon}^{t}_{y}-\boldsymbol{ \epsilon}^{t}_{\emptyset})+(\boldsymbol{\epsilon}^{t}_{\emptyset}-\boldsymbol{ \epsilon})\right)\frac{\partial\boldsymbol{g}_{\theta}(\boldsymbol{c})}{ \partial\theta}\right]$ (17) $$

is the loss function of SDS[, ], where $\emptyset$ is the empty prompt.

$$ $\nabla_{\theta}L_{\text{NFSD}}^{\theta}=\mathbb{E}_{\boldsymbol{\epsilon}, \boldsymbol{c},t}\left[\left(c\cdot(\boldsymbol{\epsilon}^{t}_{y}-\boldsymbol{ \epsilon}^{t}_{\emptyset})+(\boldsymbol{\epsilon}^{t}_{\emptyset}-\boldsymbol{ \epsilon}^{t}_{y_{\text{neg}}})\right)\frac{\partial\boldsymbol{g}_{\theta}( \boldsymbol{c})}{\partial\theta}\right]$ (18) $$

is the loss function of NFSD[], where $y_{\text{neg}}$ is a text negative prompt.

$$ $\nabla_{\theta}L_{\text{VSD}}^{\theta}=\mathbb{E}_{\boldsymbol{\epsilon}, \boldsymbol{c},t}\left[\left(c\cdot(\boldsymbol{\epsilon}^{t}_{y}-\boldsymbol{ \epsilon}^{t}_{\emptyset})+(\boldsymbol{\epsilon}^{t}_{\emptyset}-\boldsymbol{ \epsilon}^{t}_{\text{lora}})\right)\frac{\partial\boldsymbol{g}_{\theta}( \boldsymbol{c})}{\partial\theta}\right]$ (19) $$

is the loss function of VSD[], where $\boldsymbol{\epsilon}^{t}_{\text{lora}}$ the Diffusion Model fine-tuned by LoRA[].

$$ $\nabla_{\theta}L_{\text{FSD}}^{\theta}=\mathbb{E}_{\boldsymbol{c},t}\left[ \left(c\cdot(\boldsymbol{\epsilon}^{t}_{y}-\boldsymbol{\epsilon}^{t}_{ \emptyset})+(\boldsymbol{\epsilon}^{t}_{\emptyset}-\boldsymbol{\epsilon}( \boldsymbol{c}))\right)\frac{\partial\boldsymbol{g}_{\theta}(\boldsymbol{c})}{ \partial\theta}\right]$ (20) $$

is the loss function of FSD. Additionally, $\boldsymbol{x}_{t}=\alpha_{t}\boldsymbol{g}_{\theta}(\boldsymbol{c})+\sigma_{t
}\boldsymbol{\epsilon}(\boldsymbol{c})$ for FSD.

Our re-implementation of $L_{NFSD}$ is slightly different from the original[] design of NFSD by ignoring the condition when $t<200$, to keep the formulation simple. In this way, we find $\boldsymbol{\epsilon}^{t}_{\text{lora}}$ used in VSD[] may work in a similar way as $\boldsymbol{\epsilon}^{t}_{y_{\text{neg}}}$ in NFSD[]. As a result, single-particle VSD may not be able to generate diverse samples since it is still mode-seeking, aligning with the observation of ESD[]. But we do not conduct further investigations which are out of the scope of our work.

## Appendix 0.E Theory of Flow Score Distillation

We will provide some additional preliminaries and proof of Proposition 1 in the main text in this section.

### 0.E.1 Diffusion PF-ODE

The Diffusion PF-ODE [] can be written in the following form:

$$ $\frac{\mathrm{d}\boldsymbol{x}_{t}}{\mathrm{d}t}=f(t)\boldsymbol{x}_{t}-\frac{ 1}{2}g^{2}(t)\nabla_{\boldsymbol{x}}\log p_{t}(\boldsymbol{x}_{t}|y),\quad \boldsymbol{x}_{T}\sim p_{T}(\boldsymbol{x}_{T}),$ (21) $$

where $f(t)=\frac{\mathrm{d}\log\alpha_{t}}{\mathrm{d}t},\;g^{2}(t)=\frac{\mathrm{d}
\sigma_{t}^{2}}{\mathrm{d}t}-2\frac{\mathrm{d}\log\alpha_{t}}{\mathrm{d}t}
\sigma^{2}_{t}$, according to DPM-solver[]. To get Eq. (5) in the main text, we take the derivative of $\boldsymbol{x}_{t}/\alpha_{t}$:

$$ $\displaystyle\frac{\mathrm{d}(\boldsymbol{x}_{t}/\alpha_{t})}{\mathrm{d}t}=$ $\displaystyle\frac{1}{\alpha_{t}}\frac{\mathrm{d}\boldsymbol{x}_{t}}{\mathrm{d }t}-\frac{\boldsymbol{x}_{t}}{\alpha_{t}^{2}}\frac{\mathrm{d}\alpha_{t}}{ \mathrm{d}t}$ (22) $\displaystyle=$ $\displaystyle\frac{1}{\alpha_{t}}\frac{\mathrm{d}\boldsymbol{x}_{t}}{\mathrm{d }t}-\frac{\boldsymbol{x}_{t}}{\alpha_{t}}f(t)$ $\displaystyle=$ $\displaystyle-\frac{g^{2}(t)}{2\alpha_{t}}\nabla_{\boldsymbol{x}}\log p_{t}( \boldsymbol{x}_{t}|y)$ $\displaystyle=$ $\displaystyle-\frac{\frac{\mathrm{d}\sigma_{t}^{2}}{\mathrm{d}t}-2\frac{ \mathrm{d}\log\alpha_{t}}{\mathrm{d}t}\sigma^{2}_{t}}{2\alpha_{t}}\nabla_{ \boldsymbol{x}}\log p_{t}(\boldsymbol{x}_{t}|y)$ $\displaystyle=$ $\displaystyle-(\frac{1}{\alpha_{t}}\frac{\mathrm{d}\sigma_{t}}{\mathrm{d}t}- \frac{\sigma_{t}}{\alpha_{t}^{2}}\frac{\mathrm{d}\alpha_{t}}{\mathrm{d}t}) \sigma_{t}\nabla_{\boldsymbol{x}}\log p_{t}(\boldsymbol{x}_{t}|y)$ $\displaystyle=$ $\displaystyle\frac{\mathrm{d}(\sigma_{t}/\alpha_{t})}{\mathrm{d}t}\left(- \sigma_{t}\nabla_{\boldsymbol{x}}\log p_{t}(\boldsymbol{x}_{t}|y)\right)$ $\displaystyle=$ $\displaystyle\frac{\mathrm{d}(\sigma_{t}/\alpha_{t})}{\mathrm{d}t}\boldsymbol{ \epsilon}_{\phi}(\boldsymbol{x}_{t}|t,y).$ $$

This equation is the scaled version of Diffusion PF-ODE under the notation of Karras *et al*. [].

### 0.E.2 DDIM Sampling

We derive the DDIM[] sampling algorithm in this section. According to the first order discretization of Eq. 22:

$$ $\frac{\boldsymbol{x}_{t}}{\alpha_{t}}-\frac{\boldsymbol{x}_{s}}{\alpha_{s}}=( \frac{\sigma_{t}}{\alpha_{t}}-\frac{\sigma_{s}}{\alpha_{s}})\boldsymbol{ \epsilon}_{\phi}(\boldsymbol{x}_{s}|s,y),$ (23) $$

the sampling algorithm of DDIM[] can be derived as the following equation:

$$ $\boldsymbol{x}_{t}=\alpha_{t}(\frac{\boldsymbol{x}_{s}-\sigma_{s}\boldsymbol{ \epsilon}_{\phi}(x_{s}|y,s)}{\alpha_{s}})+\sigma_{t}\boldsymbol{\epsilon}_{ \phi}(\boldsymbol{x}_{s}|y,s).$ (24) $$

### 0.E.3 Proof of Proposition 1

We present a detailed derivation of Proposition 1 in the main text in this section. We define

$$ $\boldsymbol{x}_{t}=\alpha_{t}\hat{\boldsymbol{x}}^{\text{c}}_{t}+\sigma_{t} \tilde{\boldsymbol{\epsilon}},$ (25) $$

for the reverse diffusion process and then we can apply change-of-variable on $\boldsymbol{x}_{t}$. Finally

$$ $\displaystyle\frac{\mathrm{d}\hat{\boldsymbol{x}}^{\text{c}}_{t}}{\mathrm{d}t}=$ $\displaystyle\frac{\mathrm{d}\frac{\boldsymbol{x}_{t}-\sigma_{t}\tilde{ \boldsymbol{\epsilon}}}{\alpha_{t}}}{\mathrm{d}t}$ $\displaystyle=$ $\displaystyle\frac{\mathrm{d}(\boldsymbol{x}_{t}/\alpha_{t})}{\mathrm{d}t}- \frac{\mathrm{d}(\sigma_{t}/\alpha_{t})}{\mathrm{d}t}\tilde{\boldsymbol{ \epsilon}}$ $\displaystyle=$ $\displaystyle\frac{\mathrm{d}(\sigma_{t}/\alpha_{t})}{\mathrm{d}t}(\boldsymbol {\epsilon}_{\phi}(\boldsymbol{x}_{t}|t,y)-\tilde{\boldsymbol{\epsilon}}).$ $$

We also derive first-order discretization of FSD for 2D generation from Eq. 23:

$$ $\hat{\boldsymbol{x}}^{\text{c}}_{t}-\hat{\boldsymbol{x}}_{s}^{o}=(\frac{\sigma _{t}}{\alpha_{t}}-\frac{\sigma_{s}}{\alpha_{s}})(\boldsymbol{\epsilon}_{\phi}( \boldsymbol{x}_{s}|s,y)-\tilde{\boldsymbol{\epsilon}}),$ (26) $$

## Appendix 0.F Visualize the Change-of-Variable

Figure: (a) FSD (first-order discretization)
Refer to caption: extracted/5760085/assets/same-fsd.png

Even though we show FSD can generate similar images When using the same initial noise in the main text, there are notable differences between the generated images since FSD uses Adam[] to update parameters while DDIM uses first-order discretization of Diffusion PF-ODE. We show generation results of FSD that use first-order discretization in Fig. 15.

We also visualize the “change-of-variable” trick we used in the derivation of the main theorem. We define our new variable: the clean image $\hat{\boldsymbol{x}}^{\text{c}}_{t}$ according to the following equation:

$$ $\boldsymbol{x}_{t}=\alpha_{t}\hat{\boldsymbol{x}}^{\text{c}}_{t}+\sigma_{t} \tilde{\boldsymbol{\epsilon}}.$ (27) $$

Notably, there is also another similar but different concept: the one-step estimated ground-truth image $\hat{\boldsymbol{x}}^{\text{gt}}_{t}$, defined by:

$$ $\boldsymbol{x}_{t}=\alpha_{t}\hat{\boldsymbol{x}}^{\text{gt}}_{t}+\sigma_{t} \boldsymbol{\epsilon}_{\phi}(\boldsymbol{x}_{t}|y,t).$ (28) $$

Figure: (a) noisy image $\boldsymbol{x}_{t}$
Refer to caption: extracted/5760085/assets/vis-xt.png

Figure: (a) noisy image $\boldsymbol{x}_{t}$
Refer to caption: extracted/5760085/assets/vis-xt-fsd.png

Figure: (a) noisy image $\boldsymbol{x}_{t}$
Refer to caption: extracted/5760085/assets/vis-xt-sds.png

We visualize the trajectory of $\boldsymbol{x}_{t}$, $\hat{\boldsymbol{x}}^{\text{c}}_{t}$ and $\hat{\boldsymbol{x}}^{\text{gt}}_{t}$ in the same DDIM generation process in Fig. 16. As $\hat{\boldsymbol{x}}^{\text{gt}}_{t}-\hat{\boldsymbol{x}}^{\text{c}}_{t}=\frac
{\sigma_{t}}{\alpha_{t}}(\tilde{\boldsymbol{\epsilon}}-\boldsymbol{\epsilon}_{
\phi}(\boldsymbol{x}_{t}|y,t))\propto\nabla_{\theta}L_{\text{FSD}}^{\theta}$ implies, we can see DDIM as a process that tries to align clean image with the one-step estimated ground-truth image generated by the Diffusion Model. We also visualize $\boldsymbol{x}_{t}$, $\hat{\boldsymbol{x}}^{\text{c}}_{t}$ and $\hat{\boldsymbol{x}}^{\text{gt}}_{t}$ of FSD and SDS in Fig. 17 and Fig. 18, respecitively.

## Appendix 0.G Algorithm for Flow Score Distillation

We provide a summarized algorithm for Flow Score Distillation in Algorithm 1.

Figure: Algorithm 1 Flow Score Distillation

## Appendix 0.H Practical Designing Rules for ϵ ⁢ ( 𝒄 ) bold-italic-ϵ 𝒄 \boldsymbol{\epsilon}(\boldsymbol{c}) bold_italic_ϵ ( bold_italic_c )

### 0.H.1 Practical Designing Rules

We do not specify the form of $\boldsymbol{\epsilon}(\boldsymbol{c})$ in the general form of FSD in the main text. However, it is intuitive to align $\boldsymbol{\epsilon}(\boldsymbol{c})$ in 3D space like the noise priors used in Video Diffusion Models[, , ]. We have tried several designs of $\boldsymbol{\epsilon}(\boldsymbol{c})$ and summarized several design rules for designing $\boldsymbol{\epsilon}(\boldsymbol{c})$ as well as the related potential problems if violating the design rules for $\boldsymbol{\epsilon}(\boldsymbol{c})$:

- •
The noise associated with each camera view conditioning on camera view should be an uncorrelated Gaussian noise. *i.e*. $\boldsymbol{\epsilon}(\boldsymbol{c})|\boldsymbol{c}\sim\mathcal{N}(
\boldsymbol{0},\boldsymbol{I})$. If not so, the input image may be out-of-distribution for the pretrained text-to-image Diffusion Models.
- •
The noise associated with different camera views should be aligned in 3D space. Otherwise, FSD may degrade to the original SDS. This is consistent with the observation of recent work[] on Video Generation, which showed that the noise maps corresponding to different video frames are highly correlated.
- •
The noise should not be only aligned in specific points in the 3D space. This may lead to broken geometry since the convergence speed in 3D space is not uniform (see analysis on failure of constant noise function in the main text).

### 0.H.2 World-map Noise Function

We will take a deeper analysis of the property of the world map noise function in the main text. We will discuss how our proposed design rules are followed.

As noted in the main text, the local texture of the added noise is highly correlated with the generated images. In 3D spaces, it is natural to identify points, that correspond to similar points on the noise maps when projected onto the image planes of different camera views $\boldsymbol{c}$, as the points of high convergence speed. Let us denote those points as $\boldsymbol{p}_{+}$. For simplicity, we study the $\boldsymbol{p}_{+}$ that corresponds to the center points in image space.

For object-centric generation, a common camera view sampling strategy is to sample camera positions at $(r_{\text{cam}},\theta_{\text{cam}},\phi_{\text{cam}})$ under spherical coordinate and force the camera to look at the center point. When we apply a constant noise function as $\boldsymbol{\epsilon}(\boldsymbol{c})$, the center point in 3D space is a $\boldsymbol{p}_{+}$ point. As a result, FSD trends to generate geometry with holes. When applying world-map noise function proposed in main text, $\boldsymbol{p}_{+}$ is located at $(r_{+},\theta_{\text{cam}},\phi_{\text{cam}})$. One can compute that

$$ $r^{\theta_{\text{cam}}}_{+}\approx\frac{2\tan\frac{\text{FOV}}{2}}{2\tan\frac{ \text{FOV}}{2}+\Theta}\cdot r_{\text{cam}}$ (29) $$

if consider nearby views with slightly different $\theta_{\text{cam}}$ and

$$ $r^{\phi_{\text{cam}}}_{+}\approx\frac{2\tan\frac{\text{FOV}}{2}}{2\tan\frac{ \text{FOV}}{2}+\Theta\cdot\sin\theta_{\text{cam}}}\cdot r_{\text{cam}}$ (30) $$

if consider views with slightly different $\phi_{\text{cam}}$. In this way,
we align nearby views coarsely. Moreover, for different camera parameters, $r_{+}$ is different, avoiding nonuniform convergence speed in 3D space.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: Q1. Regarding the connection between DDIM and SDS, how different is the analysis in this paper from [NewRef-1]?

Q2. I think Figure 14 is insufficient to claim the limitation of using a fixed noise.
- Can the authors provide more examples with different prompts? 
- Is there any case that the proposed noise sampling also makes the flaws? 
- Can a simple alternative such as using multiple fixed noises resolve the issue, rather than the proposed method?
- Can the statements in Section 4.1.1 be analyzed empirically and validated enough?


Q3. How is the $\mathbf{W}$ operation defined? 

Q4. Why is the background/foreground noise defined separately? Would simply using the world-map noise function for a fixed noise for 3D be insufficient?

Q5. As Figure 4, can the author provide the results of generated images via the world-map noise function? If the assumption on the world-map noise function holds, I think the generated images should show 3D-consistent changes like the results in Figure 4.

Q6. I think the quality of existing methods in this paper seems to be much degraded compared with the results in the original papers (e.g. NFSD and VSD). Can the authors explain the reasons?
soundness: 2
strengths: S1. This paper aims to solve an important problem: the lack of diversity in generated 3D results by score distillation.

S2. The proposed world-map noise function is interesting.

S3. The results show varied 3D objects given the same prompt without performance degradation compared with SDS.
summary: This paper proposes a Flow Score Distillation (FSD) method to combine Score Distillation Sampling (SDS) with the proposed noise sampling strategy. After explaining the connection between DDIM and SDS, this paper suggests using the same noise throughout the optimization process in SDS. For text-to-3D generation, the authors use a world map noise function, which renders noise given specific camera parameters after initializing a 3D-shape of noise. The results show that the proposed technique generates varied 3D objects with different noise initializations.
weaknesses: W1. Lack of novelty and originality. The existing papers have already discussed the connection between DDIM and SDS [NewRef-1, NewRef-2] and using a fixed noise for SDS [NewRef-2, NewRef-3]. In addition, the paper does not include enough rationales of how the proposed noise sampling can resolve the issues in using a fixed noise.


W2. Insufficient experiments. The paper lacks in-depth analysis on the proposed noise sampling technique. In addition, some results of previous methods show much difference from the results in the original papers.


W3. Presentation. I think some notations are confusing or used without definition. For example, $\hat{\mathbf{x_{t}^{c}}}$ in Line 234 is not related to $t$. Section 4.1.2 uses many undefined variables such as $D, H, W, H_\text{hidden}, W_\text{hidden}$. $\mathbf{W}_{(\cdot)}(\epsilon_p)$ is not an operation since it refers to a noise patch. In addition, the operation $\mathbf{W}$ is not defined in the paper, although it is the most important part of the proposed method.



[NewRef-1] Kim et al., “DreamSampler: Unifying Diffusion Sampling and Score Distillation for Image Manipulation”, ECCV 2024.

[NewRef-2] Lukoianov et al., “Score Distillation via Reparametrized DDIM”, NeurIPS 2024.

[NewRef-3] Alldieck et al., Score Distillation Sampling with Learned Manifold Corrective, ECCV 2024.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: NA
soundness: 2
strengths: - This paper formulates SDS as a generalized DDIM (Denoise Diffusion Implicit Models) process and introduces a world-map noise function for 3D generation, the noise mechanism design is simple and seems effective.
- I like the quality of its generated 3D assets, which are sharp and come with fine-grained details, the results quality is consistent across various examples from main paper and supp.
- it also reveals the relationships between initial noise map to final 3D assets, which is insightful
summary: This paper comes up with a simple noise-sampling strategy to address diversity limitations in Score Distillation Sampling (SDS) for text-to-3D generation. It leverages world-map noise function that enables more varied 3D outputs by aligning noise coherently across different camera views and a two-stage generation pipeline.
weaknesses: - While I like the quality of plotted 3D assets examples, the biggest concern I think is on the contribution significance and novelty. All the strategies like leveraging multi-view diffusion models, scheduled noise level annealing, SDS formulation are already extensively explored in previous works starting from MV-Dream, ProlificDreamer, VSD. The innovation on noise map sampling is kind weak
- I think the comparison to baselines are also not fair and informative enough, as the proposed method is using MVDream as a stronger pre-trained 2D diffusion model
- FSD improves over other methods but still requires significant time for generation, raising questions about practical deployment for real-time 3D generation applications. Other recent text-to-image-to-3D can already generate high-quality 3D assets in seconds
- While the world-map noise function is introduced, further clarification is needed on how this function consistently maintains diversity across different views without affecting 3D consistency.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: - In Sec. 3.3, it's better to have some visualization to illustrate the "SDS are over-smoothness and lacks diversity" issue instead of just describing by text. This part of analysis is less convincing to the reader but I roughly get the idea.

- What would be the loss function to optimize for the 2D example in Sec. 3.2? Would that be the same to the objective introduced in Sec. 4?
soundness: 2
strengths: - The paper built a connection between PF-ODE and SDS gradient to show that the SDS gradient is equivalence to some term in diffusion.
- The paper proposes to use a consistent noise (which is only possible with its FSD formulation) to encourage less multi-face problems. 
- The paper shows that the proposed method can improve consistency.
summary: The paper focuses on improving the SDS (score-distillation sampling) method for 3D generation. Original SDS method optimizes the KL divergence between the data distribution and the distribution by the generated model, which lacks diversity. The paper first built a connection between the SDS loss and PF-ODE, and found that a consistent diffusion noise can be used for SDS. By using this method, the paper claimed that the diversity and quality of the 3D generation is improved.
weaknesses: - Inadequate analysis support to the claim of quality and diversity improvement. The paper claims that the proposing method has better quality and diversity comparing with previous method. I appreciate the FID analysis experiment. However, it is not convincing. Firstly, the FID is computed between the rendered images and generated images, where the rendered images are from 16 (prompt) x 4 (seed) = 64 3D objects (from my understanding). The amount of evaluated 3D objects is relatively lower. Secondly, the FID score is not reliable enough and it would be better to have a user study for the comparisons. In my mind, around 50 to 100 prompts for user study and 1K to 2K 3D objects for the metric evaluation would be better. 

- The detailed process of FSD in Sec. 3.2 is not clear. I currently can only roughly construct its process from Table 1. It would be better to move some content in Appendix H (which includes a lot of other designs) to elaborate Sec 3.2. The paper also misses a connection between why using Adam to optimize Eqn 12 (a gradient to t instead of \theta) would lead to the generation goal. 

- The paper would be better to make a clear separation between ground truth distribution, estimated distribution, and the auxiliary variables (usually from the definition). O.w., it would be hard for the reader to re-verify its correctness and leads to mis-concept.
  
   - In Eqn (5), I could accept an interchangeable use of $\epsilon_\phi$ and the gradient of $p_t$, but it would be better to have a clarification in the end of Sec 2.1. 
   
    - It seems that Eqn (11) is a definition over $\hat{x}_t^c$ instead of $x_t$? The gradient of $x_t$  is directly used below in the derivation. From this context, I assume that it is the random variables "generated" from the original diffusion rule (i.e., the random variable with the forward diffusion distribution $p_t$). The current presentation is written in a way that $x_t$ is defined, which is a little bit confusing. 
   
    - Following the above point, the \hat{x}_t^c in Eqn (11) and Eqn (8) are using the same notation but I think that their meaning should be different. In Eqn (8), it is the optimization parameters (an image or 3D objs). In Eqn (11), if it is also the optimization parameters, then we could not do gradient over $x_t$ any more.  Only once the $x_t$ is the distribution of the forward diffusion process at time step $t$, $x_t$ gradient is estimated by $\epsilon_\phi$. However, if $\hat{x}_t^c$ is the optimization parameters, this is not guaranteed.

Overall, I would trust the experimental results that the whole process should work. However, I would expect a revision over the math (especially the exact use of each derivation and each notation) and more proofs of the claimed improvements.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 4
questions: see weaknesses
soundness: 3
strengths: 1. The paper is well-written and easy to understand.
2. The paper build a connection between SDS loss and the DDIM generation process, which is valuable and could inspire further research.
3. The proposed coarse-to-fine pipeline, though primarily an engineering solution, effectively contributes to generating high-quality 3D objects.
summary: This paper aims to enhance SDS for generating diversity. It analyzes the sampling strategy of the SDS loss and establishes a connection between DDIM and the SDS generation process. The paper then proposes sampling deterministic noise and introduces a world-map noise function to align noise in 3D space. Additionally, it presents a coarse-to-fine pipeline to further address the Janus problem.
weaknesses: 1. The concept of applying deterministic (fixed) noise to the SDS loss was introduced in previous work [1], but this prior work is neither cited nor discussed in the paper. My understanding is that the single-noise training in [1] is equivalent to the vanilla design described in Sec 4.1.1.


2. The proposed world-map noise function does not account for depth information -- noise is sampled without considering the shape of the generated object. Moreover, in the context of latent diffusion models, keeping the background noise static (as shown in Eq. 16) does not guarantee static gradients on pixels in the rendered image, since the gradient must pass through a non-linear, non-local VAE encoder. This issue is also noted in Chang et al. (2024) Appendix E. Therefore, it raises concerns about the technical soundness of this approach when applied to latent diffusion models.


[1] Diverse and Stable 2D Diffusion Guided Text to 3D Generation with Noise Recalibration, AAAI24