FLOWDREAMER: EXPLORING HIGH FIDELITY TEXT-TO-3D GENERATION VIA RECTIFIED FLOW

## Abstract

Abstract Recent advances in text-to-3D generation have made significant progress. In particular, with the pretrained diffusion models, existing methods predominantly use Score Distillation Sampling (SDS) to train 3D models such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3D GS).
However, a hurdle is that they often encounter difficulties with over-smoothing textures and over-saturating colors.
The rectified flow model – which utilizes a simple ordinary differential equation (ODE) to represent a straight trajectory – shows promise as an alternative prior to text-to-3D generation.
It learns a time-independent vector field, thereby reducing the ambiguity in 3D model update gradients that are calculated using time-dependent scores in the SDS framework.
In light of this, we first develop a mathematical analysis to seamlessly integrate SDS with rectified flow model, paving the way for our initial framework known as Vector Field Distillation Sampling (VFDS).
However, empirical findings indicate that VFDS still results in over-smoothing outcomes.
Therefore, we analyze the grounding reasons for such a failure from the perspective of ODE trajectories.
On top, we propose a novel framework, named FlowDreamer , which yields high-fidelity results with richer textual details and faster convergence.
The key insight is to leverage the coupling and reversible properties of the rectified flow model to search for the corresponding noise, rather than using randomly sampled noise as in VFDS.
Accordingly, we introduce a novel Unique Couple Matching ( UCM ) loss, which guides the 3D model to optimize along the same trajectory.
Our FlowDreamer is superior in its flexibility to be applied to both NeRF and 3D GS.
Extensive experiments demonstrate the high-fidelity outcomes and accelerated convergence of FlowDreamer.
Moreover, we highlight the intriguing open questions, such as initialization challenges in NeRF and sampling techniques, to benefit the research community.
For more details, please visit our project page at https://vlislab22.github.io/FlowDreamer .

## 1 Introduction

3D generation enjoys broad applications in diverse fields, such as the Metaverse, games, education, architecture design, and films, and has attracted significant research interest recently (Xie et al., ; Wang et al., ; Tang et al., ; Poole et al., ; Chen et al., ; Lin et al., ; Jain et al., ; Tang et al., ; Jiang et al., ). Text-to-3D generation – which generates 3D contents from user-input text – has emerged as one of the promising 3D generation paradigms due to its ease of use (Wang et al., ; Yi et al., ; Wang et al., ; Nichol et al., ; Jun & Nichol, ).

Recently, with the advances of text-to-2D image synthesis techniques based on the diffusion models, text-to-3D generation also undergoes a surge of research interest.
A seminal work, DreamFusion (Poole et al., ) sets a cornerstone by proposing Score Distillation Sampling (SDS) to address this issue by leveraging pretrained text-to-image diffusion model, to train Neural Radiance Fields (NeRF) (Mildenhall et al., ).
It has been rapidly evolved to 3D Gaussian Splatting (3D GS) (Kerbl et al., ; Tang et al., ; Yi et al., ) for faster training and rendering.

Despite the success, existing works  (Lin et al., ; Poole et al., ; Zhu et al., ) unveil that SDS suffers from issues such as over-smoothing textures and over-saturating colors.
For this reason, some attempts, e.g.,Wang et al. (); Liang et al. (); Wu et al. () improve SDS from different perspectives.
For example,
ProlificDreamer (Wang et al., ) introduces variational score distillation (VSD), which models 3D parameters as random variables to distill 3D assets.
However, it requires significantly more time to optimize. Consistent3D (Wu et al., ) designs a consistency distillation sampling method to train the 3D model. Nevertheless, the quality improvements are still limited.
LucidDreamer (Liang et al., ) proposes interval score matching (ISM) loss in the diffusing trajectory, but the loss is formulated based on strong assumptions and drops many terms with the same scale.

Recently, flow matching approaches (Liu et al., ; Lipman et al., ) pave new ways for fast and high-quality generation.
Among them, rectified flow model (Liu et al., ; Esser et al., ; Liu et al., ) uses a simple ordinary differential equation (ODE) to represent a straight trajectory.
It trains a time-independent vector field, indicating that the direction of the vector field remains constant.
Whereas diffusion trajectories result in curved trajectories (Lipman et al., ).
Score (Song et al., ) is time-dependent, meaning that SDS optimizing over uniformly sampled values of $t$ can produce different gradient directions. Owning to its merits, we interestingly find that it
could serve as an alternative prior for text-to-3D generation.

Figure: Figure 2: An example of over-smoothing results.
Refer to caption: x2.png

In light of this, we first develop a mathematical analysis to seamlessly integrate SDS with rectified flow model.
This enables us to build up an initial framework, named as Vector Field Distillation Sampling (VFDS). However, empirical results demonstrate that VFDS still leads to over-smoothing textures (See Figure 2).
To this end, we further analyze the grounding reasons for such a failure from the perspective of ODE trajectories.
This way, we find that, as VFDS randomly samples noise, it leads to multiple ODE trajectories in
nearly the same images, i.e., from camera poses with mild differences (See Figure 5).
This causes inconsistent update directions, leading to over-smoothing textural issues.

Buttressed by the analysis, we propose a novel framework, named FlowDreamer, which yields high-fidelity results with rich textual details and faster convergence.
The key idea of FlowDreamer is to leverage the coupling and reversible properties of rectified flow model.
Importantly, the reversible property is explored to search the corresponding noise from the image while the coupling property ensures that the corresponding noise is unique.
For formality, we define it as a push-backward process to avoid the aforementioned over-smoothing issue caused by multiple ODE trajectories and better make our update directions consistent.
Empirical experiments show that the push-backward process is efficient as three Euler discretization steps are sufficient for it to achieve plausible performance, see Figure 1.Accordingly, we propose a novel Unique Couple Matching (UCM) loss that guides the 3D model to learn the same trajectory.
Our FlowDreamer also enjoys high flexibility as it can be applied to either NeRF or 3D GS generation settings.

We conduct extensive experiments under diverse generation settings, demonstrating high-fidelity results with rich details and faster convergence of FlowDreamer, as shown in Figure 1.
When exploring the application of FlowDreamer to NeRF, we observe some interesting phenomena.
Moreover, we identify some intriguing open questions. First is the initialization problem that emerges when applying FlowDreamer to NeRF. This is because the distribution of the initialized image from NeRF is undefined in the Rectified flow diffusion space. We use a warm-up strategy to mitigate this issue. Secondly, the push-backward process with different numbers of function evaluations (NFE) and sampling methods can generate some interesting results.

In summary, our major contributions are as follows:

- •
We are the first to explore a new direction by leveraging the rectified flow model as an alternative prior for text-to-3D generation. We accordingly build a mathematical analysis to adapt SDS to rectified flow model, paving the way for an initial framework – VFDS.
Empirical results demonstrate that VFDS still leads to over-smoothing. We further analyze the underlying reasons for this issue from the perspective of ODE trajectories.
- •
Based on the analysis, we further propose a text-to-3D framework, FlowDreamer, with a novel UCM loss. The loss
is build opon the push-backward process to search for corresponding noise, rather than using randomly sampled noise in VFDS.
- •
Extensive experiments in both NeRF and 3D GS generation settings demonstrate high-fidelity results with rich details and faster convergence for our FlowDreamer. We also identify some interesting open questions, such as initialization issues for NeRF and sampling techniques in the noise search process.

## 2 Related Works

Text-to-3D generation.
It aims to create 3D assets from user-input text.
DreamFusion (Poole et al., ) proposes Score Distillation Sampling (SDS) that leverages the pretrained diffusion models to train a NeRF.
However, SDS suffers from issues such as over-smoothing textures, low resolution, slow convergence, multi-faced problem (Wang et al., ; Lin et al., ; Poole et al., ), etc.
Magic3D (Lin et al., ) designs a coarse-to-fine two-stage training pipeline and changes the 3D model to DMtet (Shen et al., ) to improve the resolution of the generated 3D results.
Later on, some works (Tang et al., ; Yi et al., ; Liang et al., ; Chen et al., ; Jiang et al., ; Li et al., ; Jiang & Wang, ) take 3D GS as the 3D model for faster training and rendering.
Recently, to overcome the multi-face problem, some works (Shi et al., ; Wang & Shi, ; Tang et al., ) fine-tune the pretrained diffusion models to generate multi-view images.

Design variant of SDS loss.
To overcome the issue of over-smoothing textural issues, some attempts (Wang et al., ; ; Liang et al., ; Wu et al., ; Zhu et al., ; Katzir et al., ; Yu et al., ) focus on designing different SDS losses. For example,
Wang et al. () proposes Score Jacobian Chaining, which applies the chain rule to the estimated score to enhance generation quality.
ProlificDreamer (Wang et al., ) proposes VSD to model 3D parameters as random variables to distill 3D assets.
Although with improved quality, it requires a much longer time to optimize.
Consistent3D (Wu et al., ) designs a consistency distillation sampling method to train the 3D model. Nevertheless, the quality improvements are limited.
LucidDreamer (Liang et al., ) proposes ISM loss in the diffusing trajectory, but the loss drops many terms with the same scale.
These methods build loss towards either DDPM (Ho et al., ) or DDIM (Song et al., ) models. By contrast, we propose a novel UCM loss that is build opon the push-backward process to search for corresponding noise, rather than using randomly sampled noise with rectified flow-based models. Our UCM demonstrates high-fidelity results with rich details and faster convergence under either NeRF or GS generation settings.

Diffusion and Flow based models.
Recent advances in text-to-image generation have witnessed a significant progress.
Diffusion models define a process, which progressively converts a distribution of training data to pure Gaussian noise. By learning the reverse process, one can sample data following the distribution.
DDPM (Ho et al., ) employs a Markov chain to achieve the above process, while DDIM (Song et al., ) proposes to reduce the iteration step by maintaining the marginal distribution hence free from the restriction of Markov property.
The diffusion process can be essentially modeled as stochastic differential equations (SDE) (Song et al., ) or ordinary stochastic equations (ODE) (Lipman et al., ).
On the other hand, flow-base models (Liu et al., ; Lipman et al., ; Liu et al., ) pave new ways for faster and higher-quality generation.

Recently, rectified flow (Liu et al., ) – one of the ODE methods – defines a simple process, optimizing the trajectories in diffusion space to be as straight as possible. We are the first to explore a new direction by leveraging the rectified flow model as an
alternative prior for text-to-3D generation. We propose a novel UCM loss, built upon the push-backward process to search for corresponding noise, Extensive experiments in both NeRF and 3D GS generation settings demonstrate higher-fidelity results with richer details and faster convergence.

Figure: Figure 3: Illustration of our FlowDreamer. Images of random views from different camera poses are sampled and then input to the VAE encoder to obtain the latents. We replace the randomly sampled noise $\epsilon$ in VFDS with $\#_{\phi}[x]$ via the push-backward process. Next, we sample $t$ from $U[0,1]$ and interpolate to obtain $x_{t}$. Finally, the UCM loss with the conditional prompt is applied to update the 3D model.
Refer to caption: x3.png

## 3 FlowDreamer

Overview. An overview of our proposed FlowDreamer is depicted in Figure 3. The key insight is to leverage the coupling and reversible properties of the rectified flow model to search for the corresponding noise, rather than using randomly sampled noise as in our initial framework VFDS (see Sec. 3.1).
Accordingly, we introduce a novel Unique Couple Matching (UCM) loss in Sec. 3.2, which guides the 3D model to optimize along the same trajectory. Finally, our FlowDreamer can be applied to two types of 3D models: 3D Gaussian splatting (Kerbl et al., ) and NeRF (Mildenhall et al., ) settings. Now let’s describe the details (see Sec. 3.3).

### 3.1 VFDS: SDS in the Lens of Rectified flow

Adapting SDS to the rectified flow framework.
We first briefly introduce the rectified flow (Liu et al., ).
Let $\pi_{1}$ and $\pi_{0}$ denote Gaussian distribution $\mathcal{N}(\mathbf{0},\mathbf{I})$ and data distribution, respectively.
$\epsilon$ and $x_{0}$ are respectively sampled from $\pi_{1}$ and $\pi_{0}$.
Rectified flow defines the forward process as(to simplify the representation, below, $x_{0}$ and $x_{t}$ indicate the latent space.):

$$ $x_{t}=t\epsilon+(1-t)x_{0},t\in[0,1]$ (1) $$

Accordingly, the reverse process follows the Ordinary Differential Equation (ODE) to map $\epsilon$ to $x_{0}$.

$$ $dx_{t}=v_{\phi}(x_{t},t)dt,t\in[0,1],$ (2) $$

where the velocity $v_{\phi}$ is estimated by a learnable network $\phi$.
The model is trained as follows:

$$ $\mathcal{L}_{\mathrm{rflow}}(\phi,\mathbf{x})=\mathbb{E}_{x_{0}\sim p_{0}, \epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I}),t\sim U[0,1]}\left[w(t)\|( \epsilon-x_{0})-v_{\phi}(x_{t},t)\|_{2}^{2}\right],$ (3) $$

where $w(t)$ is a time-dependent weighting function, $U[0,1]$ denotes the uniform distribution within $[0,1]$.
Because the rectified flow model is an ODE model,
it has reversible and coupling properties.
The $\epsilon$ from the Gaussian noise distribution is uniquely coupled with the $x$ from the data distribution.
Moreover, the rectified flow is reversible, as shown in Figure 4(a).
Specifically,

1) Reversible property:
The $\epsilon$ from the Gaussian noise distribution can map to $x$, while $x$ from the data distribution can also map to $\epsilon$.

2) Coupling property: The $\epsilon$ is determined, and the $x$ generated by the same rectified flow model is unique. Conversely, the generated $\epsilon$ is unique for a given $x$.

Now, we elucidate how to adapt SDS to the rectified flow to build an initial framework, called Vector Field Distillation Sampling (VFDS).
The loss, denoted as $\mathcal{L}_{\mathrm{VFDS}}$, can be written as:

$$ $\nabla_{\theta}\mathcal{L}_{\mathrm{VFDS}}(\phi,x=g(\mathbf{\theta,c}))= \mathbb{E}_{\epsilon,t}\left[w(t)\left(v_{\phi}(x_{t},t)-\left(\epsilon-x \right)\right)\left(\underbrace{\frac{\partial v_{\phi}(x_{t},t)}{\partial x_{ t}}}_{\text{transformer Jacobian}}\frac{\partial x_{t}}{\partial x}+1\right) \frac{\partial x}{\partial\theta}\right]$ (4) $$

where $\theta$ is the 3D model parameters, $x=g(\mathbf{\theta,c})$ denotes a rendered image from a camera pose $\mathbf{c}$, $\epsilon$ denotes randomly sampled Gaussian noise, $t\sim U[0,1]$.
Following the convention of the SDS, we omit the transformer Jacobian term for effective training.
Therefore, $\left(\frac{\partial v_{\phi}(x_{t},t)}{\partial x_{t}}\frac{\partial x_{t}}{
\partial x}+1\right)$ becomes a constant and can be absorbed by $w(t)$, so we have

$$ $\nabla_{\theta}\mathcal{L}_{\mathrm{VFDS}}(\phi,\mathbf{x}=g(\mathbf{\theta,c} ))\stackrel{{\scriptstyle\Delta}}{{=}}\mathbb{E}_{\epsilon,t}\left[w(t)\left(v _{\phi}(x_{t},t)-\left(\epsilon-x\right)\right)\frac{\partial x}{\partial \mathbf{\theta}}\right]$ (5) $$

Figure: Figure 4: (a): An illustration of the reversible and coupling properties of the rectified flow model. The reversible property indicates that $\epsilon$ can map to $x_{0}$, and $x_{0}$ can map to $\epsilon$ by reversing the direction of $v_{\phi}$. The coupling property indicates that $\epsilon$ and $x_{0}$ can only form a unique coupling. For example, $\epsilon_{2}$ and $x_{0}^{2}$ form a coupling $(\epsilon_{2},x_{0}^{2})$; therefore, $\epsilon_{2}$ and $x_{0}^{1}$ can’t form a coupling $(\epsilon_{2},x_{0}^{1})$ again. (b): An illustration of the trajectories of diffusion and rectified flow. The gradient direction of the diffusion trajectory varies with different $t$, while the rectified flow roughly remains the same for different $t$ under ideal circumstances.
Refer to caption: x4.png

Based on Equation 5, we train a 3D model utilizing a pretrained rectified flow model.
The diffusion model’s trajectory is curved (Lipman et al., ), and the score (Song et al., ; Poole et al., ) direction varies with different $t$. (see Figure 4(b)).
We denote $\epsilon_{\phi}(x_{t},t)$ as the score function, where $\phi$ represents the parameters of the denoise network, and $x_{t}$ follows the diffusion forward process.
In contrast, the trajectory of the rectified flow model (Liu et al., ; ) is straight, and the vector field direction, $v_{\phi}(x_{t},t)$, roughly remains the same for different $t$ under ideal circumstances.
In our VFDS framework – where $t\sim U[0,1]$ is used in every optimization step – the VFDS optimization direction is more consistent, therefore, its convergence speed is faster (please refer to the supplementary material).
However, the over-smoothing issue of SDS still exists.
Therefore, we further analyze the grounding reasons for the over-smoothing issue from the perspective of ODE trajectories.

Elucidating SDS Over-smoothing Issue with VFDS.
Because rectified flow is an ODE (Lipman et al., ) model, it has the coupling property.
Now, we analyze the term $\left(v_{\phi}(x_{t},t)-\left(\epsilon-x\right)\right)$ in Equation 5, where $x_{t}=t\epsilon+(1-t)x$, $\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ is a random sampled noise and $x$ is the generated image from 3D model.
As shown in Figure 5, VFDS randomly samples noise $\epsilon$, which leads to multiple ODE trajectories in the same image $x$.

Figure: Figure 5: Illustration for over-smoothing analysis. An image is coupled with multiple randomly sampled noises, causing the 3D model to learn ODE trajectories.
Refer to caption: extracted/5842814/images/Trajectory.png

When camera poses have only mild differences, the rendered images appear nearly identical.
Different ODE trajectories cause inconsistent update directions, which means that directions of $\left(v_{\phi}(x_{t},t)-\left(\epsilon-x\right)\right)$ are inconsistent.
Figure 5 illustrates a toy example.
$\epsilon_{1},\epsilon_{2},\epsilon_{3}$ are noise randomly sampled from $\mathcal{N}(\mathbf{0},\mathbf{I})$ independently.
According to Equation 1, we can get $x_{t}^{1},x_{t}^{2},x_{t}^{3}$, which looks like a blurred image (a hamburger with noise).
Following the above analysis, the same $x$ (rightmost hamburger in Figure 5) together with different $\epsilon_{1},\epsilon_{2},\epsilon_{3}$ form different trajectories.
During the VFDS training process, the rectified flow model takes $x_{t}^{1},x_{t}^{2},x_{t}^{3}$ as input and outputs the estimation of trajectory gradients, as shown in Figure 5.
Note that the fitting targets are $\epsilon_{1}-x,\epsilon_{2}-x,\epsilon_{3}-x$ respectively.
The 3D model finally learns the multiple trajectories gradient, causing the over-smoothing issue.

### 3.2 Unique Couple Matching Loss

In Sec. 3.1, we have identified the grounding reason for the over-smoothing issue, It is caused by optimizing multiple trajectories during the VFDS training.
On top, we propose a novel Unique Couple Matching (UCM) loss that guides the 3D model to optimize in the same trajectory.
Our key idea is to leverage the coupling and reversible properties of rectified flow model.

We define the process from $x$ to $\epsilon$ as push-backward process, denoted $\#$, which can be written as:

$$ $\displaystyle\#_{\phi}[x]$ $\displaystyle=x+v_{\phi}(x,\delta_{T_{0}})\Delta_{T_{1}}+v_{\phi}(x_{\delta_{T _{1}}},\delta_{T_{1}})\Delta_{T_{2}}+\cdots+v_{\phi}(x_{\delta_{T_{n-1}}}, \delta_{T_{n-1}})\Delta_{T_{n}}$ (6) $\displaystyle x_{\delta_{T_{n-1}}}$ $\displaystyle=x_{\delta_{T_{n-2}}}+v_{\phi}(x_{\delta_{T_{n-2}}},\delta_{T_{n- 2}})\Delta_{T_{n-1}},n\geq 2$ $$

where, $\sum_{i=1}^{n}\Delta_{T_{i}}=1$, $\delta_{T_{0}}=0$.
And $\#_{\phi}[x]$ denotes iteratively calculate $v_{\phi}(x_{\delta_{T_{i}}},t)$ to backtrack to the $\epsilon$ from $x$ in  Equation 6.
Due to the reversible property of rectified flow, we can search for a noise $\epsilon$ from $x$ in the VFDS framework.
Additionally, because of the aforementioned coupling property, the search noise is unique.
By replacing randomly sampled noise $\epsilon$ to $\#_{\phi}[x]$ in Equation 5, our UCM loss is defined as follows:

$$ $\nabla_{\theta}\mathcal{L}_{\mathrm{UCM}}(\theta,\mathbf{x}=g(\mathbf{\theta,c }))\stackrel{{\scriptstyle\Delta}}{{=}}\mathbb{E}_{t}\left[w(t)\left(v_{\phi}( x_{t},t)-\left(\#_{\phi}[x]-x\right)\right)\frac{\partial x}{\partial\mathbf{ \theta}}\right]$ (7) $$

$$ $where,x_{t}=t\#_{\phi}[x]+(1-t)x$ (8) $$

As shown in Figure 3 we can use push-backward instead of randomly sampled noise in VFDS, which guides the 3D model to optimize in the same trajectory.

### 3.3 FlowDreamer for NeRF and 3D GS

Based on the UCM loss, we propose a novel FlowDreamer
framework. It yields high-fidelity results with richer
textual details and faster convergence. FlowDreamer can be applied to two types of 3D models: 3D Gaussian splatting (Kerbl et al., ) and NeRF (Mildenhall et al., ).

Application to 3D Gaussian Splatting.
We use points generated by the text-to-3D generator (Nichol et al., ) as parameter initialization.
Then, we directly train the 3D GS model using our UCM loss.
Our FlowDreamer yields high-fidelity results with richer
textual details (see Figure 1).
It also turns out that our method converges faster than VFDS (please refer to the supplementary material).

Application to NeRF and the initialization issue.
For NeRF, however, a direct application does not perform well.
We call this the initialization issue of NeRF, as we will explain the reasons below.
When searching the noise $\epsilon$ for a given image $x$, i.e., push-backward process, we use the rectified flow model $v_{\phi}$.
It defines a mapping from data distribution $\pi_{0}$ to noise distribution $\pi_{1}$, where $\pi_{0}$ is the distribution of its pre-training datasets.
The effectiveness of the push-backward process depends on that the input distribution for $v_{\phi}$ should be aligned with or at least approximate to $\pi_{0}$ or $\pi_{1}$.
Otherwise, the input lies in an undefined area for $v_{\phi}$ hence the output is unreasonable.

When training NeRF from scratch, it can hardly generate reasonable images based on its randomly initialized parameters. Therefore, the input distribution (denoted as $\pi^{nf}$) is far from $\pi_{0}$ and $\pi_{1}$, causing the rectified flow model $v_{\phi}$ difficult to estimate the gradient of the ODE trajectory.
To solve this issue, we temporarily use the naive VFDS training to warm up as a remedy.
We view this initialization issue as an open question and advocate further investigations.

As for 3D GS models, the issue does not exist.
The example can be found in the supplementary material, when the prompt “A English cottage with stone walls” is provided, the NeRF Initialization is simply a gray image, while the output of the 3D GS model (initialized by Point-E (Nichol et al., )) resembles a cottage.
This indicates the initial distribution of the 3D Gaussian splatting model is more approximate to $\pi_{0}$. The result of push-backward process is also more effective, as they are closer to the gradient from images output by the trained model.

## 4 Experiments

Figure: Figure 6: Qualitative comparison under 3D GS and NeRF generation setting. Our FlowDreamer generates objects with finer details.
Refer to caption: x5.png

### 4.1 3D Generation Settings

3D Gaussian Splatting Generation.
We compare our FlowDreamer with DreamGaussian (Tang et al., ), GaussianDreamer (Yi et al., ) and LucidDreamer (Liang et al., ).
These 3D GS SoTA baselines are based on their official code by employing the Stable Diffusion 2.1 as the prior.
As shown in Figure 6(a), our method generates objects that match well with the input text prompts and exhibit realistic textures.
For example, our generated ‘pumpkin’ is of high fidelity, and only our method generates the ‘spiders’, which matches the text prompt ’plastic’ for the first prompt. The ‘origami pig’ has rich details, such as its eyes and creases, which are relatively realistic for the second prompt.
Although DreamGaussian (Tang et al., ) and GaussianDreamer (Yi et al., ) require comparatively less time, their results are generally subpar.
Our FlowDreamer shows an overall improvement in terms of visual quality and textural details.

NeRF Generation.
We compare our FlowDreamer with DreamFusion (Poole et al., ), ProlificDreamer (Wang et al., ), Consistent3D (Wu et al., ) in NeRF.
Other SoTA baselines (Poole et al., ; Wang et al., ; Wu et al., ) reimplemented by Three-studio (Guo et al., ) codebase and employ Stable Diffusion 2.1 for the prior.
As shown in Figure 6(b), our FlowDreamer achieves results with high fidelity and accurate text alignment.
For example, the ‘saguaro cactus’ and ‘clay pot’ exhibit more detail and greater visual quality for the first prompt. Only FlowDreamer does not render the ‘octopus’ and ‘harp’ as a single object for the second prompt.
Our FlowDreamer takes only more time than DreamFusion Poole et al. (), but the quality of DreamFusion’s results is limited. (For more results, please refer to the supplementary material.)

### 4.2 Quantitative Comparisons

We use CLIP (Radford et al., ) similarity to quantitatively evaluate our method under either NeRF (Mildenhall et al., ) or 3D GS (Kerbl et al., ) settings.

**Table 1: Quantitative comparisons on CLIP (Radford et al., ) similarity with other methods in NeRF generation.**
| Methods | ViT-B-32 | ViT-L-14 | ViT-g-14 |
| --- | --- | --- | --- |
| Dreamfusion (Poole et al., ) | 30.13 | 29.70 | 29.49 |
| Prolificdreamer (Wang et al., ) | 32.62 | 32.55 | 31.60 |
| Consistent3D (Wu et al., ) | 32.34 | 32.56 | 32.01 |
| Ours | 34.96 | 34.19 | 34.58 |

The results of 3D representations with NeRF come from implementation in (Guo et al., ). The results of 3D representation with 3D GS are from their official implementation. The prompts of NeRF results are from DreamFusion, and the prompts of 3D GS are from LucidDreamer Liang et al. () and ChatGPT.

We randomly choose 26 prompts each to compare in 3D GS and NeRF. We randomly select 12 from the rendered images. The rendered images are from azimuth angles from -180 to 180 degrees with a fixed elevation of 15 degrees for both NeRF and 3D GS. We use three CLIP models from OpenCLIP (Ilharco et al., ), ViT-B-32, ViT-L-14, and ViT-g-14, to calculate the CLIP similarity. Our method demonstrates better CLIP similarities both in NeRF and in 3D GS scenarios.

**Table 2: Quantitative comparisons on CLIP (Radford et al., ) similarity with other methods in 3D Gaussian splatting generation.**
| Methods | ViT-B-32 | ViT-L-14 | ViT-g-14 |
| --- | --- | --- | --- |
| DreamGaussian (Tang et al., ) | 22.94 | 23.50 | 20.76 |
| GaussianDreamer (Yi et al., ) | 28.55 | 29.03 | 26.98 |
| LucidDreamer (Liang et al., ) | 28.81 | 29.78 | 28.97 |
| Ours | 30.70 | 30.49 | 30.66 |

As shown in Tab. 1, our CLIP similarity achieved the best results across all three CLIP models, with the largest margin of 2.57 over the second-best result in ViT-g-14 in NeRF results.
And Tab. 2 shows that our CLIP similarity also achieved the best results compared to other methods. In particular, it exceeds the LucidDreamer result by 1.89 in ViT-B-32.

### 4.3 Experimental Insights of our FlowDreamer

Figure: Figure 7: Comparison with other baseline methods using the same rectified flow prior under 3D GS and NeRF generation settings. Our 3D results are with richer details and more realistic colors.
Refer to caption: x6.png

3D generation with rectified flow prior.
To better demonstrate the effectiveness of our method, we replace the SOTA method LucidDreamer’s diffusion prior with the rectified flow prior.
The derivation process is provided in the supplementary material.
We refer to the ISM loss of LucidDreamer with vector field as VF-ISM.
Figure 7 demonstrates that our method can generate results with finer details and more realistic textures compared with VFDS and VF-ISM in both 3D GS and NeRF results.
FlowDreamer achieves convergence in 3D GS and NeRF faster than VFDS or VF-ISM, while demonstrating superior details and more realistic shapes.
For instance, Figure 7(a) in 3D GS indicate that the tank’s tracks and the duck’s feathers appear heavily blurred in VFDS. Furthermore, the tank’s color lacks realism, and the duck’s back is somewhat oversaturated in VF-ISM. Our FlowDreamer not only generates the tank and the duck with richer details but also achieves a more realistic overall appearance.
In addition, the Figure 7(b) NeRF results reveal that the axe shape and details are improved, whereas the corgi in VFDS exhibits excessive smoothing, and the corgi in VF-ISM presents some noise. The shape and details of our corgi remain relatively satisfactory.

Faster convergence.
Our FlowDreamer can be trained to converge faster than the initial framework VFDS.
When reaching 1200 steps, our FlowDreamer can generate fine-grained objects whereas VFDS requires more steps to achieve convergence (please refer to the supplementary material).
When the training step reaches 3000, VFDS still shows much fewer details than FlowDreamer, and it takes less time overall.

Figure: Figure 8: A comparison between our initial framework VFDS (upper) and FlowDreamer (bottom) with different scales of CFG. The results generated by FlowDreamer contain more detailed features. Prompt: “A cake filled with Oreos, highly detailed, photorealistic.”
Refer to caption: x7.png

Impacts of different Classifier-Free Guidance (CFG) scales.
We check the impact of CFG (see Figure 8).
The results indicate that we achieve good performance across various CFG scales, demonstrating strong robustness to different CFG scales compared with VFDS.

Figure: Figure 9: Impacts of different NFEs and sampling methods. “NFE N+1” denotes using $N$ steps of iteration for push-backward, and 1 step for gradient calculation. ‘NFE: 2*3+1’ indicates that the second-order method requires 2 inferences per iteration. The total push-backward takes 3 iterations.
Refer to caption: x8.png

Impacts of different Number of Function Evaluations (NFE).
As NFE increases, the training time also increases, and the generated objects exhibit more details and more complex structures (see Figure 9(a)).
For example, the structure complexity of the front hood of the LEGO car gradually increases.
However, Even with a small NFE, wherein the push-backward process has a small cost, our method can still train a 3D model with good performance.

Impacts of various sampling methods.
We test three sampling methods, namely first-order Eulder, second-order midpoint, and fourth-order Runge-Kutta.
Experimental results in Figure 9(b) show that higher-order solvers do not necessarily yield better performance.
For example, the total push-backward process takes three iterations, and the Euler method actually produces more realistic results while requiring the least amount of time.
For more experimental results, please refer to the supplementary material.

## 5 Conclusion

In this paper, we explored a new direction by using the rectified flow model as an alternative prior to text-to-3D generation.
We developed a mathematical analysis to adapt SDS to rectified flow model, resulting in the initial VFDS framework. However, VFDS still leads to over-smoothing.
We analyzed this issue from the perspective of ODE trajectories and proposed FlowDreamer, a text-to-3D framework with a new UCM loss.
Extensive experiments showed that FlowDreamer achieves high-fidelity results with richer details and faster convergence in both NeRF and 3D GS settings.
We also highlighted open questions, such as initialization issues for NeRF and noise search sampling.
Limitation.
The Jabus problem still exists; simply adding words like ‘front view,’ ‘back view,’ and ‘side view’ to the prompt is insufficient for supervising the generation view.
Although we attempt to mitigate this issue using Perpneg (Armandpour et al., ), it still occasionally occurs. We consider solving the Jabus problem thoroughly as a focus for the future work.

## Appendix

The overview of the Appendix: We provide some illustrations, including the effects of the Transformer Jacobian term, a comparison of the speeds of SDS and VFDS, a comparison of the speeds of VFDS and FlowDreamer, and an example of the initialization difference between NeRF and 3D GS models (see .1). Accordingly, we introduce the details of the experiments in .2. To better compare with other methods, we also conduct a user study to evaluate user preferences in .3. The derivation process of VF-ISM with the Rectified Flow prior is discussed in .4. Additionally, we discuss the steps of NFE and sampling methods in .5. More results include comparisons under 3D GS and NeRF generation settings and additional results from our FlowDreamer (see .6).

### .1 Some illustrations

We provide some illustrations for a better understanding of our paper.
Figure 10 illustrates the effects of the Transformer Jacobian term. It is shown that keeping this term leads to training crashes, making it very difficult to generate meaningful 3D objects. Therefore, we omit this term to achieve an effective gradient for optimization.
Figure 11 shows a comparison of the convergence speeds of SDS and VFDS.
Figure 12 shows a comparison of the convergence speeds of VFDS and FlowDreamer.
Figure 13 illustrates an example of the initialization difference between NeRF and 3D GS models.

Figure: Figure 10: VFDS results of ignoring the Transformer Jacobian. We found that it is difficult to generate meaningful results.
Refer to caption: x9.png

Figure: Figure 11: A comparison of SDS and VFDS with different steps shows that VFDS converges more quickly and achieves relatively good results in just 3000 steps. Prompt: “A lemon cake.”
Refer to caption: x10.png

Figure: Figure 12: A comparison of our VFDS and FlowDreamer with different steps. Our method generates images with higher quality than VFDS at the same training steps, indicating a faster convergence speed. Prompt: “A DSLR photo of a large-capacity handbag suitable for travel.”
Refer to caption: x11.png

Figure: (a) NeRF
Refer to caption: x12.png

### .2 Implementation Details

We adopt Stable Diffusion 3 (SDv3) (Esser et al., ) as our Rectified flow model.
To facilitate a better comparison, we will categorize the methods into two types: one where the 3D model is NeRF, and the other where the 3D model is 3D GS.
For NeRF results comparisons, unlike ProlificDreamer (Wang et al., ) and Consistent3D (Wu et al., ), which use multi-stage rendering with normal, geometric, and texture rendering, our method simply uses Instant-NGP (Müller et al., ) for rendering.
It is optimized with a resolution of 256 for the first 5000 steps with a batch size of 1, and then 512 for the subsequent 3000 steps, also with a batch size of 1.
For the warm-up strategy, we use VFDS to optimize over 1200 steps.
The initial framework VFDS need more steps to optimize, which uses 5000 steps for both 256 and 512 resolutions, with the batch size always set to 1.
For 3D Gaussian splatting comparisons, we utilize the pretrained PointE (Nichol et al., ) to initialize the locations of 3D Gaussians, while other properties of 3D Gaussians adopt random initialization.
Our FlowDreamer uses a batch size of 4 and 3000 iterations, while VFDS uses the same batch size but with 4000 iterations.
The CFG for VFDS is set to 100 on both NeRF and 3D GS, while it is set to 40 for FlowDreamer.

### .3 User study

**Table 3: User study of 3D Gaussian splatting methods.**
| Methods | DreamGaussian (Tang et al., ) | GaussianDreamer (Yi et al., ) | LucidDreamer (Liang et al., ) | Ours |
| --- | --- | --- | --- | --- |
| Scores | 1.18 | 2.03 | 3.09 | 3.70 |

**Table 4: User study of NeRF methods.**
| Methods | DreamFusion (Poole et al., ) | ProlificDreamer (Wang et al., ) | Consistent3D (Wu et al., ) | Ours |
| --- | --- | --- | --- | --- |
| Scores | 1.46 | 2.70 | 2.18 | 3.65 |

To compare our method with other comparison methods based on human perception, we conducted a user study involving 36 participants using generated videos from 26 prompts in NeRF and 3D Gaussian splatting, respectively.
Participants viewed four videos simultaneously and assigned scores of 1, 2, 3, and 4 to each video. In each test, the prompts were shown in the title, and participants were instructed to make their decisions based on the degree of alignment of the video with the text, the detail of the video, the color of the video, and the quality of the video. A higher score indicates that users believe the video is better.
Our FlowDreamer achieves the highest scores among these methods, both in NeRF results and in 3D Gaussian splatting results.

### .4 VF-ISM Derivation

ISM uses DDIM inversion to predict the noisy latent $x_{s}$ as below.

$$ $\displaystyle x_{s}$ $\displaystyle=\sqrt{\bar{\alpha}_{s}}\hat{x}^{s-\delta_{T}}+\sqrt{1-\bar{ \alpha}_{s}}\epsilon_{\phi}(x_{s-\delta_{T}},s-\delta_{T},\emptyset)$ (9) $\displaystyle=\sqrt{\bar{\alpha}_{s}}\left(\hat{x}^{s-\delta_{T}}+\gamma(s) \epsilon_{\phi}(x_{s},s,\emptyset)\right)$ $$

where $x^{s-\delta_{T}}=\frac{1}{\sqrt{\bar{\alpha}_{s}}}x_{s-\delta_{T}}-\gamma(s-
\delta_{T})\epsilon_{\phi}(x_{s-\delta_{T}},s-\delta_{T},\emptyset)$,
$\gamma(t)=\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}$ and $s=t-\delta_{T}$.
The $\hat{x}^{s-\delta_{T}}$ is computed using DDIM inference, while $x$ represents the rendered image from the 3D model, as shown below.

$$ $\displaystyle\hat{x}^{s-\delta_{T}}$ $\displaystyle=x-\gamma(\delta_{T})\left[\epsilon_{\phi}(x_{\delta_{T}},\delta_ {T},\emptyset)-\epsilon_{\phi}(x,0,\emptyset)\right]\ldots$ (10) $\displaystyle\quad-\gamma(s-\delta_{T})\left[\epsilon_{\phi}(x_{s-\delta_{T}}, s-\delta_{T},\emptyset)-\epsilon_{\phi}(x_{s-2\delta_{T}},s-2\delta_{T}, \emptyset)\right]$ $$

Next, ISM computes $x_{t}$ based on $x_{s}$.

$$ $\displaystyle x_{t}$ $\displaystyle=\sqrt{\bar{\alpha}_{t}}\hat{x}^{s}+\sqrt{1-\bar{\alpha}_{t}} \epsilon_{\phi}(x_{s},s,\emptyset)$ (11) $\displaystyle\hat{x}^{s}$ $\displaystyle=x^{s-\delta_{T}}-\gamma(s)\left[\epsilon_{\phi}(x_{s},s, \emptyset)-\epsilon_{\phi}(x_{s-\delta_{T}},s-\delta_{T},\emptyset)\right]$ $$

After that, ISM computes the $\tilde{x}^{t}$ with the denoising process.

$$ $\displaystyle\tilde{x}^{t}=\frac{x_{t}}{\sqrt{\bar{\alpha}_{t}}}-$ $\displaystyle\gamma(t)\epsilon_{\phi}(x_{t},t,y)+\gamma(s)\left[\epsilon_{\phi }(x_{t},t,y)-\epsilon_{\phi}(x_{s},s,y)\right]$ (12) $\displaystyle+...+\gamma(\delta_{T})\left[\epsilon_{\phi}(\tilde{x}_{2\delta_{ T}},2\delta_{T},y)-\epsilon_{\phi}(\tilde{x}_{\delta_{T}},\delta_{T},y)\right]$ $$

ISM computes the $x-\tilde{x}^{t}$ and with the DDIM inversion process.

$$ $\displaystyle x-\tilde{x}^{t}$ $\displaystyle=\gamma(t)\left[\epsilon_{\phi}(\tilde{x}_{t},t,y)-\epsilon_{\phi }(x_{s},s,\emptyset)\right]+\eta_{t}$ (13) $\displaystyle\text{where, }\eta_{t}$ $\displaystyle=+\gamma(s)\left[\epsilon_{\phi}(\tilde{x}_{s},s,y)-\epsilon_{ \phi}(x_{s-\delta_{T}},s-\delta_{T},\emptyset)\right]-\gamma(s)\left[\epsilon_ {\phi}(\tilde{x}_{t},t,y)-\epsilon_{\phi}(x_{s},s,\emptyset)\right]$ $\displaystyle+...$ $\displaystyle+\gamma(\delta_{T})\left[\epsilon_{\phi}(\tilde{x}_{\delta_{T}}, \delta_{T},y)-\epsilon_{\phi}(x,0,\emptyset)\right]-\gamma(\delta_{T})\left[ \epsilon_{\phi}(\tilde{x}_{2\delta_{T}},2\delta_{T},y)-\epsilon_{\phi}(x_{ \delta_{T}},\delta_{T},\emptyset)\right]$ $$

To derive VF-ISM, we adapt ISM to rectified flow.
Concretlty, we first conduct push-backward operation to compute the $x_{s}$ with Euler sample method as below.

$$ $\displaystyle x_{\delta_{T}}$ $\displaystyle=x+v_{\phi}(x,0,\emptyset)dt$ (14) $\displaystyle x_{2\delta_{T}}$ $\displaystyle=x_{\delta_{T}}+v_{\phi}(x_{\delta_{T}},\delta_{T},\emptyset)dt$ $\displaystyle...$ $\displaystyle x_{s}$ $\displaystyle=x_{s-\delta_{T}}+v_{\phi}(x_{s-\delta_{T}},s-\delta_{T}, \emptyset)dt$ $$

where, $dt=\Delta_{T}$.

Then we use the push-backward to compute the $x_{t}$ with Euler method:

$$ $x_{t}=x_{s}+v_{\phi}(x_{s},s,\emptyset)ds$ (15) $$

where, $ds=t-s$.

Due to rectified flow, the $\tilde{x}_{t}^{t}$ can be simplly expressed as below:

$$ $\displaystyle\tilde{x}^{t}$ $\displaystyle=x_{t}-v_{\phi}(x_{t},t,y)ds$ (16) $\displaystyle-v_{\phi}(x_{s},s,y)dt-v_{\phi}(x_{s-\delta_{T}},s-\delta_{T},y)dt$ $\displaystyle-...-v_{\phi}(x_{\delta_{T}},\delta_{T},y)dt$ $$

Then we compute the $x-\tilde{x}^{t}$ with push-backward with rectified flow prior:

$$ $\displaystyle x-\tilde{x}^{t}$ $\displaystyle=\left[v_{\phi}(x_{t},t,y)-v_{\phi}(x_{s},s,\emptyset)\right]ds+ \eta_{t}$ (17) $\displaystyle\text{where,}\eta_{t}$ $\displaystyle=\left[v_{\phi}(x_{s},s,y)-v_{\phi}(x_{s-\delta_{T}},s-\delta_{T} ,\emptyset)\right]dt$ $\displaystyle+...$ $\displaystyle+\left[v_{\phi}(x_{2\delta_{T}},2\delta_{T},y)-v_{\phi}(x_{\delta _{T}},\delta_{T},\emptyset)\right]dt$ $$

Same as ISM, we ignore the $\eta_{t}$ and use $v_{\phi}(x_{t},t,y)-v_{\phi}(x_{s},s,\emptyset)$ as ISM with Rectified flow prior.

### .5 Discussion of the steps of NFE and sample methods

Figure: Figure 14: The results of different NFEs and sample methods.
Refer to caption: x14.png

As the number of iterations for push-backward increases, more training time are required.
As NFE increases, the LEGO car exhibits more complex structures overall.
When viewed vertically, each column represents results of different sampling methods with the same NFE steps.
In terms of results for each column, the Euler sampling method demonstrates strong competitiveness, regardless of number of steps.

### .6 More Results of FlowDreamer

Figure: Figure 15: More results under 3D GS generation setting. Please zoom in for details
Refer to caption: x15.png

Figure: Figure 16: More results under NeRF generation setting. Please zoom in for details
Refer to caption: x16.png

As shown in Figure 15 and Figure 16, our FlowDreamer can generate high-fidelity textures and shapes from pretrained rectified flow models both in 3D GS and NeRF.
Our method can produce realistic objects, such as sweaters and wooden bowls, including fantastical ones that are rare in reality, like pumpkins with glowing runes and baby dragons.

Figure: Figure 17: More qualitative comparison under 3D GS generation setting.
Refer to caption: x17.png

Figure: Figure 18: More qualitative comparison under NeRF generation setting.
Refer to caption: x18.png

Comparison with other methods in text-to-3D generation for NeRF and 3D GS, respectively, shows that our FlowDreamer creates 3D objects that match well with the input text prompts, exhibiting high fidelity and intricate details.
The coupe is realistic in both color and shape, and the generated watch dial appears lifelike in 3D GS results (See Figure 17).
The burrito’s shape and quality are more aligned with the prompts, and the generated tacos are even more realistic in NeRF results (See Figure 18).

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: In addition to W1 & W2 above, the authors can also provide answers for the questions below:

Q1. The authors claimed that FlowDreamer shows faster convergence, but I can't find the detailed analysis in training efficiency except for Figure 7, which shows minor improvements in training speed. Could the authors provide more analysis on convergence speed?
soundness: 2
strengths: S1. This study successfully incorporate a T2I flow model in text-to-3D generation.

S2. Compared to VFDS, the proposed noise sampling techniques shows improvements in generation quality even with a few inversion steps.
summary: This study proposes FlowDreamer to leverage a pretrained text-to-image (T2I) models trained via the rectified flow framework for score distillation sampling. After explaining the training objective of SDS with a flow model, called VFDS, this paper claim that the cause of underperformance results from using multiple noises during optimization. Thus, considering the mapping between a noise and an image, FlowDreamer first use a flow model to conduct the rendered image inversion to noises, and then apply VFDS. FlowDreamer shows better results than existing methods and VFDS for text-to-3D generation
weaknesses: W1. The comparison with existing methods are unfair. Since the existing methods use Stable-Diffusion (SD) 2.1, the performance difference can come from the differences in used T2I models, considering SD 3, which is used for the proposed method, has better T2I performance than SD 2.1.

W2. Unclear contributions. The major contribution lies in the consideration of the characteristics of flow models, different from diffusion models, for text-to-3D generation. However, considering DDIM also conducts a deterministic sampling, and an image-noise pair can be coupled, computing the noise inversion is also effective to diffusion model. For example, DDIM inversion can also be used to resolve the over-smoothing and unrealistic outputs with SDS via diffusion models [NewRef-1].

[NewRef-1] Lukoianov et al., Score Distillation via Reparametrized DDIM, NeurIPS'24.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness
soundness: 3
strengths: 1.  This paper claims the first exploration using rectified flow model for text-to-3D generation.
2.  Their results outperform previous SOTA method (i.e., lucidDreamer)
3.  The push-backward process enhances the details of generated objects.
summary: This paper proposed to adopt the pre-trained rectified flow model for text-to-3D generation. Considering the different network predictions, it modified the SDS to match the formulation of rectified flow model . Moreover, push-backward process is used to estimate noise, which provides more consistent supervision signals, leading to highly detailed objects generation.  Comprehensive experiments are conducted to valid the efficacy of proposed methods.
weaknesses: 1. This paper is based on previous observations. For example, the over-smoothed results are caused by noisy gradient. LucidDreamer solved this by replacing sampled random noise with noisy latents derived from DDIM inverse. This paper also use  similar inverse process, i.e., push-backward process, suitable for rectified flow model. Therefore, this article mainly studies how to adapt previous findings to more advanced rectified flow model, which is not interesting enough.
2.  There are several questions about results. 1) In fig.7, does VF-ISM use the same 3D parameters as flowdreamer? 2) LucidDreamer discards some terms for saving time, which is not optimal. Can you show results using Eq.12 in LucidDreamer with the proposed rectified flow model + push-backward process?  3) In Fig. 8, VFDS with large CFG has good details, does that mean UCM loss is not essential? Please show more cases and cases with larger CFG.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: - Can you elaborate on why you assume the trajectory of the rectified flow model remains straight across different t values? How do you address evidence suggesting that rectified flows are typically curved and adjusted to avoid intersection points?
- Could you provide additional quantitative evidence or large-scale experiments to substantiate the claim that VFDS converges faster and is more stable than SDS?
- How does FlowDreamer differenciates with LucidDreamer's deterministic DDIM inversion?
- Could you include a more comprehensive analysis in either 2D/3D to demonstrate FlowDreamer does improve over-smoothing issue significantly compared to VFDS baseline?
- Given that flow matching models often outperform diffusion models [3], how do you confirm that the improvement in CLIP scores (Tables 1 and 2) is due to your proposed methods rather than merely switching to a flow matching model?
- Could you provide an ablation study or detailed analysis to clarify the role of the warm-up phase in the two-stage training framework? Specifically, how does varying the warm-up duration impact model performance and stability?
- Given that FlowDreamer also stabilizes noise, why is a much higher CFG weight (e.g., ≥40) required to achieve detailed textures in FlowDreamer?
soundness: 2
strengths: - the paper is written in good clarity
- one of the first papers incoporating flow matching to SDS loss and exploit the flow matching coupling/reversible property to improve SDS stability
summary: The paper proposes using a flow matching model as a replacement for the diffusion model within the SDS framework. It offers an analysis of data and noise coupling alongside the reversible properties in flow matching, aiming to mitigate the noisy gradient signals inherent in the SDS framework by back-tracking the corresponding noise for each current rendered view. Empirical results demonstrate an improved CLIP score compared to previous approaches.
weaknesses: - The paper asserts that “the trajectory of the rectified flow model is straight, v_phi(xt, t), and remains approximately constant for different t under ideal conditions.” However, this assumption appears incorrect. Even if trained with a straight velocity, the rectified flow typically displays a curved trajectory, with flow paths re-routed at intersections to prevent crossings. This phenomenon is discussed and illustrated in Fig. 2(b) [1].
- Given this questionable assumption, it also raises doubts about the claimed advantages of VFDS in speed and stability over SDS. Although the paper suggests that VFDS converges more rapidly than the diffusion model and that FlowDreamer converges faster than VFDS, it provides only one qualitative example in the appendix without sufficient large-scale quantitative evidence or convincing benchmark metrics.
- The authors state, “in contrast, we propose a novel UCM loss based on the push-backward process to identify corresponding noise, rather than using randomly sampled noise as in rectified flow-based models.” However, prior work in LucidDreamer demonstrates a similar approach by eliminating randomness in xt through deterministic DDIM inversion, thereby increasing pseudo-GT consistency. This paper should acknowledge the correlation with LucidDreamer and correct its claims. Furthermore, as DDIM is essentially an ODE solver [2], deterministic DDIM inversion closely resembles the noise back-tracking proposed in this work. Both aim to reduce randomness, indicating that the novelty of this approach may be limited.
- Although FlowDreamer is proposed primarily to address over-smoothing issues, the paper provides only three qualitative results in Figures 7 and 8. One could even argue that in Figure 8 the level of detail between VFDS and FlowDreamer appears similar with only different styling, raising questions about the model’s effectiveness in resolving over-smoothing.
- Referring to [3], flow matching models generally perform better than diffusion models. Given the marginal CLIP score improvement in Tables 1 and 2, it remains unclear if the performance gain is due to replacing the diffusion model with the flow matching model.
- The authors highlight a limitation of FlowDreamer in its early stages when the rendered view x falls outside the target data distribution, causing instability in the back-tracked noise epsilon. To address this, a warm-up phase is required, with the initial rendered views needing to be within reasonable limits. However, the paper lacks a detailed ablation study of this two-phase training framework to validate its robustness. For instance, how does varying the warm-up duration impact performance? Is the warm-up phase length dependent on the example’s complexity (e.g., more complex scenes requiring a longer warm-up)?
- ProlificDreamer [4] demonstrates that by using a learnable noise prediction network, the CFG weight can be reduced to a standard range (e.g., 7.5) to yield reasonable results. Given that FlowDreamer is expected to produce more stable noise, it is unclear why a very high CFG value is still required to achieve detailed textures (i.e., ≥40 as shown in Figure 8).

[1] Liu, X., Gong, C. and Liu, Q., 2022. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003.

[2] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C. and Zhu, J., 2022. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35, pp.5775-5787.

[3] Ma, N., Goldstein, M., Albergo, M.S., Boffi, N.M., Vanden-Eijnden, E. and Xie, S., 2024. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.08740.

[4] Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H. and Zhu, J., 2024. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: see weakness. 

I believe that this work demonstrates the potential of distilling a 3D generator using SD3 as the teacher model. However, the current claims put forward in the paper lack strong experimental validation, and I hold reservations about some of the arguments presented by the authors. For these reasons, I recommend rejecting this paper.
soundness: 2
strengths: This paper involves extensive engineering work to distill a high-quality 3D generator using the SD3 model as the foundational framework.
summary: This paper introduces a new framework called FlowDreamer, using a FM based model to update a 3D generator, e.g., Nerf or GS. 

The whole framework mainly contains 2 part: 
1. a score distillation based on FM model (named VFDS in paper) and 
2. a push-back method (named UCM in paper) to align noise and rendered scene. 
 

Finally, extensive experiments demonstrate that the FlowDreamer framework achieves high-fidelity and fast convergence across various 3D generation settings, such as NeRF and 3D GS.
weaknesses: I have significant concerns regarding various aspects of this paper.

1. The paper suffers from a lack of citations, particularly in relation to the references concerning UCM. Similar methods have already been extensively applied in DMD2 [1] and Imagen Flash [2]. The author state UCM is improved from ISM but I rather suggest check the given reference and rethink the novelty of UCM.

2. The authors have not provided adequate experimental evidence to validate the effectiveness of the individual components of their approach. Although they demonstrate strong overall performance metrics, there is no experimental support showing that the use of **straight trajectories contributes to performance improvements**. 

    As FM is simply a parameterized special case of Diffusion. I am not convinced that straight trajectories will have a substantial effect on score distillation. Moreover, in the case of Diffusion, higher-order scores also follow straight trajectories. As such, I would require the authors to present experimental evidence demonstrating the improvements claimed to result from the use of straight trajectories.

3. Furthermore, we observed that the methods used for comparison were based on SD2.1, while the authors employed SD3 as the teacher model, which constitutes an inherently unfair comparison. Notably, this issue was not mentioned in the main manuscript. I am concerned that the observed strong alignment between the text and 3D objects may be attributed to SD3’s MMDIT, rather than the method introduced by the authors.




[1] Yin, Tianwei, et al. "Improved Distribution Matching Distillation for Fast Image Synthesis." arXiv preprint arXiv:2405.14867 (2024).

[2] Kohler, Jonas, et al. "Imagine flash: Accelerating emu diffusion models with backward distillation." arXiv preprint arXiv:2405.05224 (2024).

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: 1. Have I accurately captured the main difference between the proposed algorithm and ISM in weakness 2? I suggest that the authors include a table comparing their methods with ISM for enhanced clarity, given the significant resemblance.

2. Did the authors encounter any challenges when upgrading the base diffusion model to SDv3 for the other baselines using Eq. (2)?
soundness: 2
strengths: 1. The qualititive results are generally good. And the presentation is clear.

2. The authors conducted experiments across various representations, including 3D-GS and NeRF, to validate their algorithm.
summary: This paper presents Vector Field Distillation Sampling (VFDS) to integrate Score Distillation Sampling (SDS) with rectified flow models. The authors further enhance VFDS by substituting randomly sampled noise with pushed-back noise. The authors then experiment with rectified flow model SDv3 to validate their algorithm.
weaknesses: 1. **Lack of Novelty**: The paper presents very limited novelty, as rectified flow is merely a specific case within the broader family of diffusion models. Generalizing SDS-like methods to rectified flow is trivial. Specifically, for the forward diffusion process:

   $$
   \boldsymbol{x}_t = a_t \boldsymbol{x}_0 + b_t \boldsymbol{\epsilon}, \tag{1}
   $$

   Rectified Flow sets $ a_t = 1-t $ and $ b_t = t $. With this schedule, the prediction of rectified flow model SDv3 can be reformulated to epsilon prediction according to the equation below Eq. (12) in SDv3 [1]:

   $$
   \boldsymbol{\epsilon}_\phi = (1-t) \boldsymbol{v} _\phi + \boldsymbol{x}_t. \tag{2}
   $$

   This means that all existing SDS-like methods (which are usually formulated in epsilon prediction) can be effectively adapted to the rectified flow model SDv3 by changing the prediction type using Eq. 2. Consequently, re-deriving VFDS and VF-ISM in the paper is unnecessary, as they are trivially equivalent to SDS and ISM. This re-derivation may not offer substantial or particularly inspiring insights. The authors need to clarify what specific novel contributions or challenges they encountered in applying this approach to text-to-3D generation.

2. **Unfair Comparison**: The comparisons presented in this paper appear to be unfair. As stated in Section 4.1, the baselines utilize the SDv2.1 prior, while this work employs the SDv3 diffusion prior as claimed in Appendix section 2. The improvements reflected in the quantitative results in Tables 1 and 2 likely stem from the upgrade of the base diffusion model. Furthermore, as noted in weakness 1, upgrading other baselines to SDv3 is almost trivial. There should be no hindrance to adopting the same base diffusion model for a fair comparison. The authors should include equitable comparisons in their paper.

3. **Limited Novelty of UCM Loss**: Given the triviality of upgrading SDS-like methods to rectified flow models, the proposed UCM loss bears a strong resemblance to the loss used in ISM, further limiting the contribution of this paper. Both methods employ an inversion process to enhance the noising methods. The primary distinction between UCM and prior work ISM appears to be that ISM reverses to timestep $ t $, while UCM inverts to the maximum timestep and combines the noise with the rendered image at timestep $ t $. The authors should further explain on the difference between ISM and their UCM loss and the potiential improvements of UCM compaired with ISM. The result presented in Fig. 7 demonstrated some effectiveness, but further comparison are necessary to enhance the arguments. An additional fair quantitative comparison is necessary to demonstrate the improvements of UCM, and I would likely adjust my rating based on the outcomes of these experiments.

Meta-review: This work introduces FlowDreamer, a novel approach that leverages pretrained text-to-image (T2I) models within the rectified flow framework for score distillation sampling (SDS). After detailing the training objective of SDS with a flow model, referred to as VFDS, the paper identifies the primary cause of underperformance as the use of multiple noises during optimization. To address this, FlowDreamer introduces a two-step process: first, it employs a flow model to perform rendered image inversion to recover the corresponding noise, and second, it applies VFDS. The proposed method demonstrates improved results compared to existing approaches and VFDS in text-to-3D generation tasks.

However, the majority of reviewers reached a consensus to reject this work. Key concerns include unclear contributions, unfair comparisons, limited novelty, missing ablation studies on straight trajectories, and various other experimental shortcomings. Consequently, the current version does not appear ready for publication. We strongly encourage the authors to address these concerns comprehensively, incorporating the reviewers' feedback to enhance the clarity, rigor, and impact of the work.

Decision: Reject