Consistent Flow Distillation for Text-to-3D Generation
Authors: Runjie Yan
Yinbo Chen
Xiaolong Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Models
Score Distillation
3D Generation
Primary area: generative models
TL;DR: We propose multi-view consistent Gaussian noise and leveraging the gradient of 2D image diffusion ODE or SDE sampling process to guide the 3D generation.

Abstract: Score Distillation Sampling (SDS) has made significant strides in distilling image-generative models for 3D generation. However, its maximum-likelihood-seeking behavior often leads to degraded visual quality and diversity, limiting its effectiveness in 3D applications. In this work, we propose Consistent Flow Distillation (CFD), which addresses these limitations. We begin by leveraging the gradient of the diffusion ODE or SDE sampling process to guide the 3D generation. From the gradient-based sampling perspective, we find that the consistency of 2D image flows across different viewpoints is important for high-quality 3D generation. To achieve this, we introduce multi-view consistent Gaussian noise on the 3D object, which can be rendered from various viewpoints to compute the flow gradient. Our experiments demonstrate that CFD, through consistent flows, significantly outperforms previous methods in text-to-3D generation.

Published as a conference paper at ICLR 2025
CONSISTENT FLOW DISTILLATION FOR TEXT-TO-3D GENERATION
Runjie Yan" UC San Diego
Yinbo Chen UC San Diego
Xiaolong Wang UC San Diego
ABSTRACT
Score Distillation Sampling (SDS) has made significant strides in distilling image- generative models for 3D generation. However, its hit-uumm-iikliioodseeeiiii behavior often leads to degraded visual quality and diversity, limiting its effee- tiveness in 3D applications. In this work, we propose Consistent Flow Distillation (CFD), which addresses these limitations. We begin by leveraging the gradient of the diffusion ODE or SDE sampling process to guide the 3D generation. From the gradient-based sampling perspective, we find that the consistency of 2D im- age flows across different viewpoints is important for high-quality 3D genera- tion. To achieve this, we introduce multi-view consistent Gaussian noise on the 3D object, which can be rendered from various viewpoints to compute the flow gradient. Our experiments demonstrate that CFD, through consistent flows, sig- nificantly outperforms previous methods in text-to-3D generation. Project page: https://run yan. github. io/cdd//
INTRODUCTION
3D content generation has been gaining increasing attention in recent years for its wide range of ap- plications. However, is expensive to create high-quality 3D assets or sean objects in the real world. The scarcity of 3D) data has been a primary challenge in 3D generation. On the other hand, image synthesis has witnessed great progress, particularly with diffusion models trained on large-scale datasets with massive high-quality and diverse images. Leveraging the 2D generative knowledge for 3D generation by model distillation has become a research direction of key importance.
Score Distillation Sampling (Poole al. 2023) (SDS) pioneered the paradigm. It uses a pretrained text-to-image diffusion model to optimize a single 3D representation such that the rendered views seek a maximum likelihood objective. Several subsequent efforts (Zhu al.] 2024 Liang al.,
2023
Katzir et al. 2024; Huang et al. 2024; Tang et al.) 2023 Wang et al., 2023h Armandpour
behavior remains, which has a detrimental effect on the visual quality and diversity. Variational Score Distil- lation (Wang et al. 2024a) (VSD) tackles this issue by treating the 3D representation as a random variable instead of single point as in SDS. However, the random variable is simulated by particles in VSD. Single-partcll VSD is theoretically equivalent SDS (Wang et al.) (023b), assuming the LoRA network in VSD is always trained to optimal. While the optimization-based sampling of VSD times slower with particles.
al. 2023) have been made to improve SDS, while the maximum-likelhhood-eee
In this work, we propose Consistent Flow Distillation (CFD), which distills 3D representations through gradient-based diffusion sampling of consistent 2D image probability flows across different views. We provide theoretical analysis of this process and extend it to wide range of deterministic and stochastic diffusion sampling processes. the distillation process, we identify that a key is to apply consistent flows to the 3D representation. Intuitively, in 2D image generation, the same region is always associated with the same fixed noise for the correct flow sampling. Analogously, in 3D generation, the 2D image flows from different camera views should also use the noise patterns that are consistent on the object surface with correct correspondence. To achieve this, we design a multi- view consistent Gaussian noise based on Noise Transport Equation (Chang et al. (024). which can compute the multi-view consistent noise with negligible cost. During the distillation process, the
"qqual contribution

---

Published as a conference paper at ICLR 2025
"A pirctt go with tisteeeneseer hull that glous fountly in the dankt oceon waheen. flaminating the sheps intricate corvings and sells as siiemly rzsigetes the words"
cute cat covered by snow'
"A futuristic space station"
(a) NeRFs generated by CFD scratch.
steompunk owl with mechanical wings"
"An astronaut is riding horse"
"A polar bear surfing big wave"
(b) 3D textured meshes generated by CFD from scratch.
"A treasure chest full of gold coins and jewels, high resolution, sharp" (c) CFD can generate diverse and high-quality samples from scratch.
"A model toy fighter plane, sharp"
Figure 1: Text-to-3D samples of CFD. CFD generate diverse 3D samples by distilling text-to- image diffusion models. See videos in our project page for additional generation results.
2

---

Published as a conference paper at ICLR 2025
multi-view consistent Gaussian noise is rendered from different views to compute the gradient of 2D image flow. Finally, our method can create high quality and diverse 3D objects by following the diffusion ODE SDE sampling process.
We evaluate our method with different types of pretrained 2D image diffusion models, and compare it with state-of-the-an text-to-3D score distillation methods. Both qualitative and quantitative exper- iments show the effectiveness our approach compared with prior works. Our method generates 3D assets with realistic appearance and shape (Fig. [(a), l(b) and can sample diverse 3D objects for the same text prompt (Fig. [[c) with negligible extra computation cost compared with SDS.
In summary, main contributions are:
An in-depth discussion about using image diffusion PF-ODE SDE to directly guide 3D generation. We present equivalent forms of the ODE and SDE so that their random variables are clean images at any time in the diffusion process, and identified that flow consistency is key this process.
A multi-view consistent Gaussian noise on the 3D object, that keeps pixel i..d. Gaussian property in any single view and has correct correspondence on the object surface between different views.
A method to distill image diffusion models for 3D generation. as simple and efficient as SDS while having significantly better quality and diversity.
2 PRELIMINARIES
2.1 DIFFUSION MODELS AND PROBABILITY FLOW ORDINARY DIFFERENTIAL EQUATION (PF-ODE)
A forward diffusion process (Sohl-Dickstell et al. 2015 Ho et al. 2020 gradually adds noise to a data point By Po(2z), such that the intermediate distribution Pio x; zu) conditioned on initial sample æo at diffusion timestep / is (a,ao, of I), which can be equivalently written as
3t = 0,30 + OrE, E E~ I.,
(1)
where op oo 0 at the beginning, and at 0, in the end, such that pr(æT) is approximately the standard Gaussian (O, o I). A diffusion model 6s is learned to reverse such process, typically with the following denoising training objective (Ho et al. 2020):
LDM = Exo,e,t (wa..../lalllla00::: t) - -c11)
After training, (x(x,,)) R -o, Va, lo (z)), where Va, lo (a:) is termed score function.
A Probability Flow Ordinary Differential Equation (PF-ODE) has the same marginal distribution as the forward diffusion process at any time (Song et al. 2021b). The PF-ODE can be written as:
d(æ;/aa) d(ar/at) (-o, Va, log))
dl di d(or/ar) di
(x,,),, Ør pr((r))
data point Pn can sampled by starting from a Gaussian noise ør ~ and following the PF-ODE trajectory from t T to = 0. typically with discretized timesteps and ODE solver.
2.2 DIFFERENTIABLE 3D REPRESENTATIONS
Differentiable 3D representations are typically parameterized by the learnable parameters and differentiable rendering function go(c) render images corresponding the camera views In many tasks, the gradient is first obtained on the rendered images ge and then backpropagated
through the Jacobian matrix @ga(c) of the renderer the learnable parameters 0.
Common 3D neural representations include Neural Radiance Field (NcRF) (Mildenhall et al. 2021
Müller et al. 2022 Wang et al. 2021 Barron et al. 2021 Xu et al. 2022 3D Gaussian Splatting
3

---

Published as a conference paper at ICLR 2025
(3DGS) (Kerbl al. 2023), and Mesh (Laine et al., 2020: Shen et al. 2021). In this work, perform experiments on various 3D representations and validate that our method is applicable for generation across wide range of 3D representations.
3 CONSISTENT FLOW DISTILLATION
We present Consistent Flow Distillation (CFD), which takes a pretrained and frozen cext-to-mmage diffusion model and distills 3D representation by the gradient from the probability flow of the 2D image diffusion model. We propose to guide 3D generation with 2D clean flow gradients operating jointly 3D object. We identify that key in this process is make the flow guidance consistent across different camera views (see Sec. 3.1). We further propose an SDE, a generalization of the clean flow ODE, that incorporates noise injection during optimization to enhance generation quality (see Sec. 3.2). To achieve the consistent flow, we propose an algorithm to compute multi-view con- sistent Gaussian noise, which provides noise for different views with noise texture exactly aligned on the surface of the 3D object (see Sec. 3.3) Finally, we draw connections between CFD and other
score distillation methods (see Sec.
3.4).
3.1 3D GENERATION WITH 2D CLEAN FLOW GRADIENT
Given pretrained text-to- text-to-image diffusion model co f, y), let y denote the condition (text prompt), the conditional distribution p( can be sampled from the PF-ODE (Song et al. 2021b) trajectory from / = T to / = O, which takes the form
d Et
(5)
a2
Cee
VE
By following the diffusion PF-ODE, pure Gaussian noise transformed an image in the target distribution py) Thus PF-ODE can be interpreted as guiding the refinement of a noisy image to realistic image. Can we use image PF-ODE to directly guide the generation of differentiable 3D representation 0 through the refining process, with 0 as its learnable parameters and go as its differentiable rendering function?
A direct implementation can be substituting the noisy images in Eq. 5 with the rendered images ga(c) at the camera view c by letting E = ge(c). By viewing d( ( as the learning rate of an optimizer and 6 (are, 4 y) as the loss gradient to St, the gradient can be backpropagated through the Jacobian matrix of the renderer (c) to update @ according to
AA -lr (elrrcoloon co(orgo(c)) L2 (, @ga(c)
a8
However, such a direct attempt may not work (see Fig. 5 (a)), since the image x, at diffusion timestep t contains Gaussian noise. It is hard for the images rendered by 3D representation to match the noisy images Øt in an image PF-ODE, particularly around the beginning t = T. where æT is per-pixel independent Gaussian noise. It is generally impossible for continuous 3D represen- tation to be rendered as per-pixel independent Gaussian noise from all camera views simultaneously. As result, the rendered views may be out-of- oe-ditrribution (OOD) as the input to the pretrained image diffusion model, and therefore cannot get meaningful gradient as guidance.
To resolve the OOD issue, we use a coinge-of-variatie (Gu et al. 2023 Yan et al. 2024 to trans- form the original noisy variable ar in PF-ODE (Eq. 5) to new variable that free of Gaussian noise at any time t E [0, T]. For each trajectory (x; (ee(or) of the o; in the original PF-ODE, the new variable at is defined as
32 ō.ē
(7)
where ē is set as the initial noise ē = 2T and is constant for each ODE trajectory ( By Eq. 5 and Eq. 7 the evolution of the new variable aē is derived follows:
or
dă = d( L Ot1
(ao(ar + orE, (,
(8)
VC
4

---

Published as a conference paper at ICLR 2025
Consistent Flow Distillation (CFD) Gradient
Prompt
Clean Flow (Annealing Time)
dgo(cc)
3D Object 0
Rendered Image
go(c2)
Oa(C)
Add Noise E(8,c)
Text to Image Diffusion cp
dgl(c)
Fgl = (,,(), -E) Ag((
Bo(G))
Figure 2: Overview of CFD. The 3D representation 0 is generated with decreasing timesteps. At each timestep t, different views ge(c) are rendered. The 2D image clean flow provides the gradient timestep to the views and backpropagates to The right shows the gradient computation in detail: we add multi-view consistent noise (see Fig. 3 to the rendered image and pass it into the frozen text-to-image diffusion model, gradient is calculated using the model prediction and then backpropagated to 0.
Changing the variable x, the original diffusion PF-ODE to the variable æ makes directly using PF-ODE as 3D guidance possible by providing the following properties (the proof is in Appx. G.3): (i) éi are clean images for E [0, T" (see Appx. Fig. 17). therefore, can be substituted with the rendered clean images (c). (ii) 29 is initialized from zero: EG 0, which can be consistent with the 3D representation initialization (e.g. NeRF, where the entire scene is initialized to a uniform gray). (iii) The endpoint of the new ODE trajectory 2g 26 is sample following the target distribution po (o()) and is completely determined by the constant (thus é can be viewed as the identity of the trajectory). The new variable ãē is therefore termed clean variable. Note that as is different from the "sample prediction" dy =.-oree(w..t..) of diffusion network for æt, which is not directly usable in this framework and we discuss for more details in Appx. We use clean flow denote the ODE (Eq, () of the clean variable 2ç.
Similar to Eq. 6. we use the following gradient to update the 3D representation 0:
(gr(c)
VoLern E, (cclaaigule++ + o, c). 2(0,c)) c)) 00
where / ((T) is predefined monotonically decreasing timestep annealing function of the op- timization time т, and é(0, c) is a multi-view consistent Gaussian noise function, we discuss its design details Sec. 3.3 We let ē(0, c) be a deterministic function of 0 and c, ensuring that the noise remains constant for fixed camera view and geometry, given that ē is constant for a single flow trajectory in clean flow ODE. Since have set of 2D image flows jointly operating on a 3D object, the gradient updates from different camera views in Eq. 9 may interfere with each other. We identify that a key in the 3D sampling process is make the 2D image flows consistent on the 3D object surface. This requires multi-view consistent Gaussian noise function ε(0, c) that is not only view-dependent but also provides the correct local correlation on the object surface. The multi-view consistent Gaussian noise function should apply similar noise pattern to the same region of the object surface, even from different camera views. This corresponds to that the fixed noise pattern is always added to the same region for the clean variable in 2D image clean flow ODE. The overall process of CFD is summarized in Fig. 2
3.2 GUIDING 3D GENERATION WITH DIFFUSION SDE
Despite that PF-ODE and diffusion SDE can recover the same marginal distributions in theory, SDE- based stochastic sampling may result in better generation quality reported in prior works (Song
5

---

Published as a conference paper at ICLR 2025
Rasterize Aggegate
Reference Space Ere!
yR
Warping gr11
Camern tn World Projection
Camera-boe to World Projection
Camera View
W World Space
Camera View
Figure 3: Warping consistent noise for query views. To obtain a query view noise map. for each pixel, its vertices are projected onto the object surface, then wrapped to the coordinates in high-resolution noise map. The values within the region specified by the coordinates on the high- resolution noise map are summed and normalized as the return pixel value in the query view noise.
et al. 2021b a Karras et al. 2022). Motivated by this, we also propose to use image diffusion SDE guide 3D generation.
To achieve this, we propose a reverse-time SDE with a form similar to the clean flow ODE (Eq
dæs
(d( 2t
S,dt) (cslartf + tz t, w E.)
VC
(10)
dé, = E, + /23,dw
with initial condition as. O and ēn No. I), where w. is standard Wiener process in the reverse time from T to 0. It can be further proved that this SDE and its forward-time form are equivalent to the diffusion SDE presented by Song al. (Song et al. (2021b) and EDM (Karras iet al. (022). When we set 3r = O, the SDE becomes deterministic and becomes the clean flow ODE. When 8g * O, new Gaussian noise will be injected into é, during the diffusion process, but & is still of unit variance throughout the whole process from T to O. Furthermore, æ in this SDE still retains the "clean properties" á in the clean flow ODE. Thus, we also use clean flow to refer to this SDE. We provide detailed discussions and proofs about this SDE in Appx. G
The clean flow SDE implies that simple modification on Eq. can make VoLerD(O) correspond to using SDE guidance. As detailed in Appx. G.4.1 only need to inject new Gaussian noise into E(0, e) during optimization by:
E(7 11) -11)))))((()) Vec.
(11)
where y is predefined noise injection rate, is the optimization step, and e NN(, I) is sampled each optimization step.
3.3 MULTI-VIEW CONSISTENT GAUSSIAN NOISE
To get consistent flow, a multi-view consistent Gaussian noise function ē(0, c) is required, which (i) is per-pixel independent Gaussian noise for all camera views (ii) the noise patterns from different views have the correct correspondence according to the 3D object surface. It is non-trivial to satisfy all these properties with common warping and interpolation methods. The query rays from camera views e take continuous coordinates, simply using common interpolation methods such as bilinear may break the per-pixel independent property and result in bad quality (see Fig. (b)).

---

Published as a conference paper at ICLR 2025
Inspired by Integral Noise (Chang et al. 2024), we develop an algorithm that implements the multi- view consistent Gaussian noise with Noise Transport Equation. The Noise Transport Equation was originally proposed for warping noise between two frames in video (Chang et al. 2024). To use it in the 3D task, we generalize the Noise Transport Equation to the warping between two different manifolds and compute the warping from different query camera views to the same reference space Eref. As shown in Fig. 3 given a camera view c, the query pixel p is first projected onto the surface of the object as to-orl ctw. in the world space Eworld, then we map those points from the surface to a reference space Eref through a predefined mapping function T-! (design details are in Appx. D). We define high-resolution Gaussian noise map W on Eref. Finally, we aggregate and return the noise value G(p) for the query pixel p according to
G(p)
Σ W(A), A,E
(12)
is the area covered by p after being warped to Eref A, is a noise cell in Eref, and (A,) is the noise value of unit variance at A, By first projecting query pixels from different camera views the object surface in the world space Eworld. two query pixels P1. P2 from two different camera views that look at the same region on the object will be projected to overlapped regions ctwe, (p1). ctww, (p2) the object. After being warped by the same function T-1, they cover overlapped regions 22p. Sp, and get correct correlation in noise maps G(p1), G(p2).
where P2p =
Our method can be also viewed as deriving a rendering function for the noisy variable æ; in the original form of PF-ODE (Eq, 5 by
x0,0,c) = aggo (c) + o,ē(0,
(13)
As discussed in Integral Noise (Chang et al. 2024), the warping of an image (e follows the transport equation that takes a similar form of Eq. 12 but with a different denominator [stp]. instead of v [S2p] for ē(0, c). thus common 3D representation is incapable of rendering Gaussian Noise ē(0, c), and it is needed to disentangle the noisy variable into the clean part go(c) and noisy part E(0, c). By disentanglement, we can handle the two parts that follow different rendering equations separately and achieve the rendering of the noisy variable for using image PF-ODE (or diffusion SDE) as the guidance for 3D generation.
3.4 COMPARISON WITH OTHER SCORE DISTILLATION METHODS
Comparison with SDS. Both SDS and our CFD share a similar gradient form (Eg(a,, t, é) (gs(c) to update the 3D representation 0 from sampled rendered view. In SDS, t is typically ran- domly sampled from a range [fmin: f. and ? is noise randomly sampled at each step. In contrast to SDS, our CFD uses an annealing timestep t(T) that decreases from fmxx to fmin the deterministic noise ē(0, c) depends from different views have correct correspondence according to the object surface. Notably, SDS with annealing timestep schedule can be viewed as setting y in CFD, where significant stochas- ticity is injected in the optimization. As a comparison, for typical diffusion sampling processes, y 2 0.00024 in DDPM, and 7 0 in DDIM (see Appx. (.4.2). In our CFD, the definition of ? requires that 7 < 1 (Appx. Eq. 37). which implies a difference between CFD and SDS.
both the object surface and the camera view, it is designed to let the noise
Theoretically, when restricted to 2D image generation where = -g),). SDS is equivalent seeking the maximum likelihood point in the noisy distribution P: with a Gaussian distribution Maaz, of I) centered at the image x. When the optimization of SDS loss is near optimal, their generation results are centered around a few modes (Poole et al.) 2023). In contrast, our CFD is sampling from the whole distribution Po and equivalent to diffusion ODE or SDE sampling process with first-order discretization. Thus, CFD can generate more diverse results with better quality.
loss gradient
noising method
SDS
(x1) es Ex( Elora (x:)
e~ C~.0 I)
VSD
e~ C~NNO.,1 I)
ISM
(x,) - - (I,)
DDIM inversion (n)())) (c))
CFD (ours)
Cas (Tx))
غ = غ (0. c)
Table I: Comparison between score distillation gradients.
7

---

Published as a conference paper at ICLR 2025
DecamFusiin (500)
PresiffeDresme (K5)))
LacklDrrsmmr (IS)
(ours)
Figure 4: Visual comparison to baseline methods. We compare rendered images of our method with baselines include DreamFusion (Poole et al. 2023), ProlificDreamer (Wang et al. (024a), HiFA (Zhu et al. 2024), LucidDreamer [Liang et al. 2023). 'he images of baselines are from their official implementations. Prompts: "A 3D model of an adorable cottage with a thatched roof" (top) and "A DSLR photo of an ice cream sundac" (bottom).
Comparison with other score distillation methods. We list the loss and noising of different meth- ods in Tab. ISM (Liang al. 2023) incorporates DDIM inversion noising in their score distil- lation. While this approach can yield finer details than SDS, computing the inversion significantly increases computational costs. We discuss the connection between our method and ISM Appx, H We also list the difference between proposed pipeline and different baseline methods in Appx. F.2
4 EXPERIMENTS
In comparisons to prior methods, we distill Stable Diffusion (Rombach et al. (022) and use the
same codebase threestudio (Guo al. 2023). We compare CFD with various prior state-of-the-ar methods, including SDS (Poole et al.
2023a VSD (Wang al.) 2024a) and 2023 Wang et al.
ISM (Liang et al. 2023). Specifically. VSD incorporates LoRA network training in their score dis-
tillation, ISM incorporates DDIM inversion in their score distillation. Since timestep annealing (Zhul
ct al. 2024] Wang et al. 2024a| Huang et al. (0244) has been shown to help improve generation qual-
2024), also apply timestep annealing to all baseline methods. We use results from the official implementation of other baselines in qualitative
ity (Wang et al. 2024a| Zhu |0224 Huang
comparisons if not specified. In addition, we show results of 2-stage pipeline in Fig. [(a) [(b]]
where we first distill MVDream (Shi et al. (2024), then distill Stable Diffusion, which alleviates
the multi-face face issue (Poole et al. 2023 AArmandoour et al. 2023 Hong al. (023a). We provide implementation details in Appx.
A and details of experiment metrics in Appx.
4.1 COMPARISON WITH BASELINES
We compute 3D-FID following VSD (Wang et al. 2024a) to evaluate the quality and diversity of dif- ferent score distillation methods, and compute 3D- CLIP evaluate prompt alignment for different methods. We provide qualitative comparison in Fig. and quantitative results in Tab. 3 and Appx. We also provide additional comparisons with VSD Appx. Fig. 99 ISM Appx. Fig. [O and SDS in Appx. Fig. 11 As shown in both quantitative and qualitative results, CFD outperforms all baseline methods and has better generation quality (Fig. and and diversity (Appx. Fig. 9 Our method produces rich details and the results are more photorealistic. Addition results and
3D-FID J
3D-CLIP
SDS
88.06
35.07+1..00
ISM
86.00
34.99-0.226
VSD
83.02
35.10:1.200
Tab. 5
CFD (ours)
78.13
35.16.10.23
Table Comparison with baselines on quality, diversity and prompt alignment. We report averaged clip score of different verison of CLIP backbones. We use 10 seeds for each of the different prompts, respec- tively.
Appx. Fig. 9 10
10
comparisons in Appx.
8

---

Published as a conference paper at ICLR 2025
(a) Original PF-ODE
(b) w/ bilinear noise
(c) w/ random noise
(d) w/ consistent noise
Figure 5: Ablation the noise design and the flow space. (a) Directly training 0 with original
PF-ODE using Eq. 6 with noisy variable. (b) Distilling with allinear-interpole noise map. (c)
Distilling with random noise. (d) Distilling with our muew consistent Gaussian noise, which has the best visual quality.
Ranker
Aesthetics
PickScore
Ours vs. SDS
0,54
0.64
Ours vs. VSD
0.60
0.68
Ours vs. ISM
0.56
0.66
Ours vs. FSD
0.54
0.78
Table 3: Automated win rates comparison under reward models. We compare the perfor- mance of our CFD method against baseline models using Aesthetics Scores (Schuhmann) 2022) and PickScores (Kirstain et al. 2023). Our method consistently achieves winning rate higher than 0.5, which demonstrates its effectiveness.
4.2 ABLATION STUDIES
(a) When directly training with original PF- ODE using Eq. 6 with noisy variable, the training fails after several iterations. (b) Simply using bilinear interpolation instead of Noise Transport Equation leads to correlated pixel noise and gener- ates blurry results. (c) When using the random noise in SDS, the results over-smoothed. (d) Our consistent flow distillation with multi-view consistent Gaussian noise generates high-quality re- sults. By using multi-view consistent Gaussian noise, the flow for fixed camera more aligned with a diffusion sampling process, and the quality improves. We also provide additional ablations on our design choices in Appx.
Ablation on the flow space. As shown in Fig. 5
Ablation on noise injection rate ? Noise injection rate 7 Eq. 11 determines the rate which new noise will be injected into the noise function. When 0. no noise will be injected, é will be fixed constant if the geometry and camera view is fixed and CFD corresponds to using ODE guidance. When 7 O, new noise will be injected, and ē(O, c) will gradually change. In this case, CFD corresponds to using SDE guidance. Using SDE-based stochastic samplers may help to improve image generation quality reported in prior works (Song al.) 2021b a Karras et al.) 2022]. In Tab. 4
We also observe that use small nonzero ? helps to improve the performance of CFD. In practice, therefore we set = 0.0001 by default our experiments for CFD. As reference, we calculated a typical equivalent y value of DDPM be y se 0.00024 (see Appx. G.4.2
found that using a y larger than 0.0001 could result in over-smoothed texture,
5 RELATED WORK
Diffusion models Diffusion models (Sohl-Dickstein et al.) 2015 Sharma et al. 2018 Ho et al.) 2020 Song et al. 2021b Changpinyo et al.) 2021] Schuhmann et al. 2022) are generative models that are learned to reverse diffusion process. A diffusion process gradually adds noise data distribution. and the diffusion model is trained to reverse such an iterative process based on the score function. Denoise Diffusion Implicit Models (DDIM) (Song al. 2021a) proposed determinis-

---

Published as a conference paper at ICLR 2025
7
0.0
0.0001
0.001
0.01
1.0
3D-IS (t) 2.24=0.12
2.60:1021
2.47:10.39
2.08
1.77=0.133
Table Ablation noise injection rate 7, We ablate the impact of y on 3D generation diversity and quality. We generate samples with 16 random seeds.
tic sampling method to speed up the sampling. Meanwhile, it is proved that a diffusion process corresponds to Probability Flow Ordinary Differential Equation (PF-ODE) (Song et al. (021b), which yields the same marginal distributions as the forward diffusion process at any timestep. Later
works (Salimans Ho 2022, Karras et al., 2022, Lu al. 2022) demonstrate that DDIM can be viewed as the first-order discretization of the PF-ODE.
Score distillation sampling The score distillation sampling (SDS) paradigm for distilling 2D text- to-image diffusion models for 3D generation is proposed in DreamFusion (Poole et al. 2023) and SJC (Wang et al.) 2023a). During the distillation process, the Iearnable 3D representation with dif- ferentiable rendering is optimized by the gradient to make the rendered view match the given text.
Many recent works follow the SDS paradigm and studied for various aspects, including timestep
annealing (Huang et al.
2024, Wang et al., 2024a Zhu et al.] 2024). coarse-to-fine training (Lin)
jet al.
2023 Wang et al.
2024a| Chen al. 2023), analyzing the components (Katzir et al.
2024
formulation refinement (Zhu et al. 2024 Wang et al. 22024a Liang et al.
2023 Tang et al.
2023
Wang et al. 2023b Yu al.
2024 Armandpour et al. [023 Wu al.,
2024b Yan et al. 2024
geometry-texture disentanglement (Chen et al., 2023; Ma et al. 2023
Wang et al. [2024a). ad-
dressing multi-face Janus problem replacing the lext-to-image diffusion with novel view synthesis
diffusion (Liu et al. 22033 Long et al., 2023 Liu et al. 2024bb Weng et al. 2023
Ye et al. 2023
Wang & Shi 2023) or multi-view diffusion (Shi et al. 2024).
Another prevailing paradigm for 3D generation is to reconstruct the 3D shape given input image. A typical pipeline is first generate sparse-view images and then reconstruct the 3D shapes using reconstruction methods (Wu et al.) 2024a Li et al. 2024) or mod- els (Hong et al. 2023b Liu al. 2024a| Wang et al.) [022bb] Tang al. 2024 By directly training on relatively large scale 3D dataset like Objaverse (Deitke et al.
Reconstruction models
2023 these methods are usually capable of generating plausible shapes with fast speed, but the performance of these models usually limited when facing out of domain input images.
CONCLUSION
In this paper, we proposed Consistent Flow Distillation. We begin by leveraging the gradient of the diffusion ODE or SDE sampling process to guide the 3D generation. From a sampling perspective, identified that using consistent flow to guide the 3D generation is the key to this process. We developed a multi-view consistent Gaussian noise with correct correspondence on the object surface and used it to implement the consistent flow. Our method can generate high-quality 3D represen- tations by distilling 2D image diffusion models and shows improvement in quality and diversity compared with prior score distillation methods.
Limitations and broader impact. Although CFD can generate 3D assets of high fidelity and diversity, similar to prior works SDS, ISM, and VSD, the generation can take one to a few hours, and when distilling eext-to-image diffusion model, due to the properties of the teacher models, the distilled 3D representation sometimes may have multi-face Janus problem and may not be good for complex prompt. Besides, due to 3D representation flexibility and interference from other views, is very hard to guarantee that the sampling process from a rendered view of the 3D object is exactly the same as sampling for 2D images given text in practice. While our 3D consistent noise can reduce the interference and achieve better results, the flow for 3D rendered views may not be exactly the same as 2D flows of the initial noise. Also, like other generative models, it needs to pay attention to avoid generating fake and malicious content.
10

---

Published as a conference paper at ICLR 2025
This work supported, in part, by the Amazon Research Award, the Qual- comm Innovation Fellowship.
Acknowledgements
REFERENCES
Mohammadreza Armandpour, Huangjie Zheng, Ali Sadeghian, Amir Sadeghian, and Mingyuan Zhou, Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond. arXiv preprint arXiv:2304.04966 2023.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 843-852, 2023.
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Bartin-Brualla, and Pratul P. Srinivasan. Mip-nerf: multiscale representation for anti-aliasing neural radiance fields, 2021.
Pascal Chang, Jingwei Tang, Markus Gross, and Vinicius C. Azevedo. How i warped your noise: lemporally-correeatee noise prior for diffusion models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL htttDs:///penrevee...t net. forum?id
pZEInMzqSD
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEF/CVF Inter- national Conference on Computer Vision (ICCV), October 2023.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13142-13153, 2023.
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Joshua M Susskind. Boot: tillation of denoising diffusion models with bootstrapping. In ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling, 2023.
Data-free dis-
Yuan-Chen Guo, Ying-Tiee Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia- Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang. threestudio: A unified framework for 3d content generation. https://githur com wwt.to//eeetttoooooooooooo threestudio, 2023.
Jonathan Ho and Tim Salimans. arXiv:2207.12590 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Susung Hong, Donghoon Ahn, and Seungryong Kim. Debiasing scores and prompts of 2d diffusion for robust text-to-3d generation. arXiv preprint arXiv:2303.15411 2023a.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint 0rXiv:2311.04400 2023b.
Yukun Huang, Jianan Wang, Yukai Shi, Boshi Tang, Xianbiao Qi, and Lei Zhang. Dreamtime: An improved optimization strategy for diffusion-guide 3d generation. In The Twelfth International Conference on Learning Representations, 2024, URLhttps: //tp://///e/eeeeie/ee/e net forum?
id=1 TA=YbAUYYYJTU

---

Published as a conference paper at ICLR 2025
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 14226565-265777 2022.
Oren Katzir, Or Patashnik, Daniel Cohen-Or, and Dani Lischinski. Noise-free score distillation. In The Twelfth International Conference on Learning Representations, 2024. URL https:
aopenreview.nnt//ornnee
IMcml
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663, 2023.
Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendering. ACM Transactions Graphics, 39(6), 2020.
Peng Li, Yuan Liu, Xiaoxiao Long, Feihu Zhang, Cheng Lin, Mengfei Li, Xingqun Qi, Shanghang Zhang, Wenhan Luo, Ping Tan, al. Era3d: High-resolution multiview diffusion using efficient row-wise attention. arXiv preprint arXiv:2405. 1616, 2024.
Yixun Liang Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Lucid- dreamer: Towards high-fidelity text-to-3d generation via interval score matching. arXiv preprint arXiv:2311.11288 2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d con- tent creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 300-309, 2023.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pp. 5404-5411, 2024.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu. and Hao Su. One- 2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov. and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9298-9309, 2023.
Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consisten images from single-view image. In The Twelfth International Conference on Learning Representations, 2024b. URL //ppn//vie.......... http://rruuniidddMMyyyya.aoo....
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using drosin diffusion. arXiv preprint 23..::2300.10000 2023.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li. and Jun Zhu. Dpm-solver: A fast ode solver for diffusson probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5788 2022.
Artem Lukoianov, Haitz Sáez de Ocáriz Borde, Kristjan Greenewald, Vitor Campagnolo Guizilini, Timur Bagautdinov, Vincent Sitzmann, and Justin Solomon. Score distillation via reparametrized ddim. arXiv preprint arXiv:2405. /589/, 2024.
Baorui Ma, Haoge Deng, Junsheng Zhou, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Geo- dream: Disentangling 2d and geometric priors for high high-fidelity and consistent 3d generation. arXiv preprint arXiv:2311.1797. 2023.
12

---

Published as a conference paper at ICLR 2025
David McAllister, Songwei Ge, Jia-Bin Huang, David W Jacobs, Alexei A Efros, Aleksander Holyn- ski, and Angjoo Kanazawa. Rethinking score distillation as a bridge between image distributions. arXiv preprint arXhv:2406.09411 2024.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41 (4):1-155, 2022.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, 2023. URL
https://openreviin wwwp:/nnteeeeettteeeeeee Fo
net
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022. URL
bet/Eorun?1o--TIIIIIaa
Christoph Schuhmann.
Laion-aesthetics.
https://laion.ai/blee
laion aetbon--steeiccc/// 2022. Accessed: 2024-11-22.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 -55294, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 2556-2500 2565, 2018.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. In Advances Neural Information Processing Systems (NeurIPS). 2021.
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3d generation. In The Twelfih International Conference on Learning Representa- tions, 2024.
wwtp:trrneeteeee.ee nnt. forum? patrum?ideee id-FUgrrrpp jq2pbB
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising difffusion implicit models. In Interna- tional Conference on Learning Representations, 2021a. URL htttttt:://ppereeei/// net/
[oormmid==ttgiaare
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference Learning Representations, 2021b. URL /ttpps://eperrevierr wtt.eiiwwwwwweeieeee...........e
ForumP1d=PxTIc122REE
Boshi Tang, Jianan Wang, Zhiyong Wu, and Lei Zhang. Stable score distillation for high-quality 3d generation. arXiv preprint arXiv: 02,0055 2023.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang. Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution content creation. arXiv preprint arXiv:2402.00 05054, 2024.
13

---

Published as a conference paper at ICLR 2025
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Viston and Puttern Recognition, pp. 2228-8238, 2024.
Haochen Wang. Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jaco- bian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12619-12629, 2023a.
Peihao Wang, Dejia Xu, Zhiwen Fan, Dilin Wang, Sreyas Mohan, Forrest landola, Rakesh Ranjan, Yilei Li, Qiang Liu, Zhangyang Wang, et al. Taming mode collapse in score distillation for text- to-3d generation. arXiv preprint arXiv:2401.00909, 2023b.
Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation.
arXiv preprint arXiv:2312.0220. 2023.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 27171-27183, Associates, Inc., 2021. URL bttps:/ wwwp:iinnnnnnnuuuruoe www.nnnnrii... cc/ files/ paper/2021 / Fi 1e e/e41/446 e41e 64 6 httpa//22200001111-a---e-e- pdd
Curran
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024a.
Zhengyi Wang, Yikai Wang. Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint W.Xivv2403.05003 2024b.
Haohan Weng, Tianyu Yang, Jianan Wang, Yu Li, Tong Zhang, CL Chen, and Lei Zhang.
Consistent123: Improve consistency for one image to 3d object synthesis. arXiv preprint 1rX00:2310.08092. 2023.
Kailu Wu, Fangfu Liu, Zhihan Cai, Runjie Yan, Hanyang Wang, Yating Hu, Yueqi Duan, and Kaisheng Ma. Unique3d: High-quality and efficient 3d mesh generation from a single image. arXiv preprint arXiv:2405.20344 2024a.
Zike Wu, Pan Zhou, Xuanyu Yi, Xiaoding Yuan, and Hanwang Zhang. Consistent3d: Towards consistent high-fidelity text-to-3d generation with deterministic sampling prior. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9892-9902, 2024b.
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Uirich Neu- mann. Point-nerf: boint-based neural radiance fields. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 5438-5448, 2022.
Runjie Yan, Kailu Wu, and Kaisheng Ma. Flow score distillation for diverse text-to-3d generation, 2024.
Jianglong Ye, Peng Wang, Kejie Li, Yichun Shi, and Heng Wang. Consistent-l-to-3: Consistent im- age 3d view synthesis via geometry- diffusion models. arXiv preprint arXiv:2310.03000, 2023.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang Freedom: Training-free energy-guided conditional diffusion model. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 231844, 2023.
Xin Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Song-Hai Hai Zhang, and XIAOJUAN QL. Text- to-3d with classifier score distillation. In The Twelfth International Conference on Learning Rep- resentations, 2024.
net tate//foruue wow.mmidduttturttouur
Junzhe Zhu, Peiye Zhuang, and Sanmi Koyejo. HIFA: High-fidelity text-to-3a 3d generation with ad- vanced diffusion guidance. In The Tweifth International Conference on Learning Representations, 2024. URL hhtts::::: hnttts://openreett net. wtw.:neeeooommmmmmmmmmmmmmmm
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A IMPLEMENTATION DETAILS
In this paper, conduct experiments primarily
single NVIDIA-GeFFoce-RRR RT--cce or similar pipelines (including the choice
NVIDIA-L40 GPU. In the quantitative experiments, we similar of 3D representation, training steps, shape initialization, teacher diffusion model, etc.) across meth- ods. We apply timestep annealing for all methods and use the same negative prompts in the quanti- tative experiments. The main differences between methods lie in the loss functions used.
We use CFG (Ho & Salimans 2022 scale of 75 for CFD in quantitative experiments. In practice, We found CFD works the best with CFG scale of 50-75. We apply the same fixed negative prompts (Shi
et al. 2024 Katzir et al. 2024 McAllister et al. 2024) for different text prompts.
For simple prompts, we directly use CFD to distill Stable Diffusion v2.1 (Fig. 12 and 13
For mesh generation, we first use CFD to generate coarse shapes with MVDream (Shi et al. Then we use CFD and follow the geometry and mesh refinement stages in VSD (Wang et al. 2024a) with Stable Diffusion v2.1 to generate the mesh results in Fig. [(b)
2024
For complex prompts, we adopt 2 stage pipeline (Fig. ((a)
I(e)
9
10 and IT]. We first
generate coarse shape by distilling MVDream to avoid multi-face problems. Then we distill Stable Diffusion v2.1 refine the details and colors (stage 2). We randomly replace the rendered image with normal map with 0.2 probability to regularize the geometry in stage 2. The total training time approximately hours on A100 GPU.
B EXPERIMENT DETAILS
3D-FID We compute the FID score between the rendered images for the generated 3D samples and the images generated by the teacher diffusion models following the evaluation setting of VSD (Wang et al. 2024a). For the experiments with 10 prompts in Tab. 2 we sampled 5,000 images for each prompt from Stable Diffusion, creating a real image set with total of 50,000 images. We generated 3D objects using different score distillation methods, with 10 different seeds per prompt for each method. We rendered 60 views for each 3D object, resulting in fake image set of 6,000 images. We use FID implementation from torchmetrics package with feature=2048.
3D-IS We compute the Inception Score (IS) for the front-view images measure the quality and diversity. We set split=2 to compute the standard variance of the metric. Due to limited compute budget, we use 16 random seeds for each parameter setting of y and then the rendered front view to compute IS metric. The IS implementation used in our experiments is from the torchmetrics package.
3D-CLIP We compute the CLIP cosine similarity between the rendered images of the 3D samples and the corresponding text prompt. For one sample, we render 120 views and take the maximum CLIP score. Then we average the CLIP score across different seeds and prompts (and CLIP models). We use CLIP socre implementation from torchmetrics package.
Aesthetic evaluation Following Diffusion-DPO (Wallace et al., 2024). we conduct an automated win rate comparison under reward models in Tab. 3 The performance of our CFD method is evalu- ated against baseline models using Aesthetics Scores (Schuhmann) (022) and PickScores (Kirstain et al. 2023). We calculate the scores rendered images generated from 50 samples, each corre- sponding to a randomly selected prompt.
C ADDITIONAL QUALITATIVE COMPARISON
We present more comparison between baseline methods and CFD in Fig. present additional generation results of CFD in Fig. Fig. Fig.
T0 and Fig. II
We
and Fig. 12
15

---

Published as a conference paper at ICLR 2025
"A steampunk owl with mechanical wings"
An astronaut riding a horse"
"A llama in tuxedo at fancy gala"
A cowboy raccoon with lasso"
Figure 6: Diverse NeRF results of CFD distilling MVDream then Stable Diffusion (Rombach et al. 2022) on complex prompts.
16

---

Published as a conference paper at ICLR 2025
"A manga magical girl with magic wand"
"A 3D anime-styls style dragon girl with shimmering scales, horns, and confident expression"
"A samurai panda with a bamboo sword"
"A painter hedgehog with a palette"
"A knight fox in shining armor"
"A wizard frog with spellbook"
"A 3D model of a medieval house with grass, vines, stone, wood, and medieval decor"
"A 3D model of DSLR camera, photography, box modeling, Maya"
Figure 7: NeRF results of CFD distilling MVDream then Stable Diffiusion (Rombach et al. 2022) complex prompts.
17

---

Published as a conference paper at ICLR 2025
"A squirrel playing the guitar"
pig wearing backpack"
"A weathered brass compass on an old, map. The compass is slightly tarnished, showing signs age, bathed in soft, diffused sunlight."
glass resting
cracked ceramic mug, chipped along and faded from years of use, resting on rustic weeden table, with morning sunlight casting shodows surface."
"An old leather suitcase, its corners froyed and its surface marked with age, wooden floor bathed in seft light."
"A delicate porcelain teocup with a gold nimned edge, resting on on embroidered tablecloth, Soft light glooms off the fine, china, pevesling its intricate flordl design and subtie cracks from age."
vintage travel tags, placed
Figure 8: NeRF results of CFD distilling MVDream then Stable Diffusion (Rombach et al, 2022) on complex prompts. CFD successfully generated multiple objects and most align with long prompts.
D ALGORIHHM
We provide pseudo algorithms for CFD in Algorithm Algorithm multi-view consistent Gaussian noise E(0,c). c).
presents how to compute the
Choices of warping function T-1 and reference space Eref Generally speaking, correct corre- spondence of noise map between different camera views can be achieved with any choice of con- tinuous warping function T-1 and reference space Eref- In this work, we choose Eref to be 2D square space Eref = [-1, 1J2 to utilize existing fast rasterization algorithms, so that Algorithm can be efficiently computed. We design a warping function T-1 to map points in 3D world space Ewortd to 2D reference space Eres- Specifically, compute the warping T-1 first convert the
18

---

Published as a conference paper at ICLR 2025
VSD
CFD (ours)
"A rotary telephone carved out of wood"
"A sliced loaf of fresh bread"
plush dragon toy"
"A tarantula, highly detailed"
Figure 9: Additional comparison with ProlificDreamer (VSD) (Wang et al. 2024a). We use results from the official implementation of the baseline. We show generation results of different methods with different seeds the last row.
19

---

Published as a conference paper at ICLR 2025
ISM
CFD (ours)
wooden car"
DSLR photo Rugged, vintogeinspired hiking boots weathered quality, HD"
"Saber from Fate stay Night, 3D, girl, anime"
"Zombie JOKER, head, HDR, photorealistic, 8K"
Figure 10: Additional comparison with LucidDreamer (ISM) (Liang et al. 2023). We use results from the official implementation of the baseline. We show generation results of different methods with different seeds in the last row.
20

---

Published as a conference paper at ICLR 2025
(our)
"A squirrel knitting scorf in cazy living room"
SDS
(ours)
CFD
"A teapot shaped a toy car"
(aurs)
"A highly detailed 3D model of sand castle"
Figure 11: Comparison with SDS. We distill MVDream Shi et al. 2024 and Stable Diffu- sion (Rombach al. 2022) this experiment. We first generate coarse shape by distilling MV- Dream using SDS and CFD. then distill Stable Diffusion refined the color with SDS and CFD, respectively. In this figure, the only difference between two methods is the noise function used by SDS and CFD. We 4 different seeds for each methods in this figure. SDS trends to generate oversmoothed textures and identical simple shapes. CFD outperforms SDS with better diversity and fidelity.
3D-CLIP
B16
B32
L14
L14-336
SDS
36.30
35.99
31,82
32.42
VSD
36.58
36.27
31.97
32.67
CFD (ours)
36.79
36.32
32.44
33.10
Table Comparison with baselines on prompt alignment. We random seed for each of the 128 prompts. B16, B32, L14, L14-336 denote different versions of CLIP backbones. We observe that CFD is competitive or outperform SDS and VSD on prompt alignment.
points at (Ip, Up 2p) to spherical coordinates (rp, 0p, 6p). For simplicity, we only present the case when &b (0, )). The point is then mapped to (xr,. 1r) E Preef where
cos Op:
(14)
dr 1). cos 8p (2
yr
21

---

Published as a conference paper at ICLR 2025
"A plate piled high with chocolate chip cookies"
"A ripe strawberry"
"A baby bunny sitting on top of stack pancakes'
"A car made out of sushi"
*A bagel filled with cream cheese and lox'
"A hotdog in a tutu skirt"
'A marble bust of mouse"
"A delicious croissant"
"A small saguaro cactus planted in clay pot"
Figure 12: NeRF results of CFD distilling Stable Diffusion (Rombach et al., 2022).
Under this mapping function, one can verify that dz,dy, ((s..ve) |d8,dóp sin ddp a(0,.
So points uniformly scattered on the sphere in 3D space Eworld will remain uniform after being mapped to the reference 2D space Eref. This design helps improve the faimess of Algorithm so that we can use lower resolution reference space while keeping most of the warped triangles covering enough area in the reference space Eref- Notably, two different triangles could overlap with the warping defined by Eq. 14 resulting in correlations across the pixels of the computed noise function é(0, e) in the same camera view. This overlap occurs only when the surface of the 3D object intersects the radius of sphere centered at the origin of the Emorld more than once. However, we not observe the destructive effects seen in other interpolation methods that can lead to correlation between pixels (as in Fig. 5 (b)) in our experiments, and we believe it is unnecessary to find a warping function that avoids such overlapping completely.
Reference space Erey resolution We reference space with resolution of 2048 2048 in most of our experiments. This will only introduce 8.1% computation overhead to our training (tested on RTX-3090 GPU). The teacher model Stable Diffusion represents the whole object with latent at 64 resolution and MVDream (Shi et al. 2024) 32, so noise map with 2048 resolution is sufficient. We also observe the quality is similar with noise map resolutions from 512 to 2048.
22

---

Published as a conference paper at ICLR 2025
SDS
VSD
FSD
CFD (our))
"A zoomed DSLR photo of model adorable cottage with thatched roof, high resolution, sharp"
"A highly detailed DSLR photo model of historical stone castle"
DSLR photo of model of treasure chest full gold coins and jewels, high resolution, sharp"
Comparison with SDS, VSD and FSD. We distill Stable Diffusion (Rombach et al, 2022) with different score distillation methods in this experiment. CFD outperforms SDS, VSD and FSD with better visual quality, geometry and has richer details.
Figure 13:
E ADDITIONAL ABLATIONS
E.1 ABLATION ON THE DESIGN SPACE
We ablate our proposed improvement step by step in this section. Timestep annealing (Wang et al.,
2024a Zhu al. 2024 Huang et al. 2024) is helpful for forming finer details. Adding negative prompts (Shi et al. 2024 Katzir et al. 2024 McAllister et al. 2024) helps to improve generation styles. We also find that adding negative prompts is crucial when timestep t(T) is small. With- out negative prompts, the color of samples will become unnatural during the optimization at small timesteps. In this work, we apply negative prompts by directly replacing the unconditional predic- tion of the diffusion model with prediction conditioned on negative prompts. Finally, by changing random sampled noise SDS with our multi-view consistent Gaussian noise, the generated samples can form much richer details and are more diverse. We visualize this ablation in Fig. 15
We propose utilizing CFD to distill the multi-view diffusion model, MVDream, in Stage as shape observation that both bascline
initialization for complex prompts. This decision is based methods and our CFD can experience multi-face issues when solely distilling SDv2.1 (Fig. [4(a) and Fig. 14(b). However, distilling only MVDream produces low-quality results (Fig. 14()). To address these issues, we adopt a two-stage pipeline in our complete method, where Stage initializes the shape using MVDream, and Stage 2 refines by distilling SDv2.1. This approach effectively mitigates the challenges identified above, as illustrated in Fig. 14(d)
E.2 COMPARE THE PIPELINE DIFFERENT METHODS
We list the differences between the pipelines of different baseline methods in Tab.
23

---

Published as a conference paper at ICLR 2025
Algorithm CFD
1: Input: 3D representation parameter 0, prompt y, pretrained diffusion model 6, (, y), render
go (c), annealing sime-schedule ((7), learning rate Ir.
2: Output: 3D representation parameter 0.
3: for from 0 to Tend do
4: Sample camera view
5: Render image (c), depth map Depth(c), and opacity map Opacity(c)
6: Get diffusion timestep ((T)
7: Compute 3D Consistent Noise E(0,c) e)
> Refer to Algorithm
8: 3y f- ango(c) + o,E(0,
9: 0 Ir (I) t(T), č(0, c))
@gn(e) 88
end for
Algorithm 2 Computing 3D Consistent Noise
1: Initialization: Noise background Ebg. high resolution noise Eref in reference space Eref, opac-
ity threshold oth, noise injection rate 7.
2: Input: Depth map Depth (c), opacity map Opacity(c).
3: Output: ε(0, c) Cout-
4: Triangulate the pixels to p
5: Project those triangles to the surface ctw(p) world space Eworld according to Depth(c)
6: Warp the triangles from world space Eworrd to reference space Eref TT (ctw, (p))
7: Rasterize and aggregate the noise values on Eref corvered by the trangles
8: cont t- Ebg
9: coat (Opacity(c) Ochlp Vn E(.,)) covered by the triangle (p) Cref (zr, v]
10: if y then
11: Ebg f vI Ebgg v randn_like (ekg)
SDE noise injection
12: Ere/ v1 7eref v randn like(Eref)
13: end if
14: Return Cout
SDS (SDv2.1)
(b) CFD (SDv2.1)
(c) CFD (MVDream)
(d) CFD stage)
Figure 14: Ablation on the pipeline stages. Prompt: "A bear playing electric bass".
E.3 COMPARISON ON NOISE METHODS
We list the differences between the noising methods of different baseline methods in Tab. 6 Con- current work FSD Yan al. 2024) also employs a deterministic, view-dependent noising function and can therefore be considered a special case of our CFD with O. The noise of FSD is aligned on a shpere independent of the 3D object surface. However, this noise design can still lead to over- smoothed textures, and the misalignment of noise with the 3D object surface in suboptimal geometry (see Fig. 13). The noise design of FSD is inferior to ours when the 3D object shape is nearly formed. Gradient consistency essential for accurately constructing geome- try in differentiable 3D representations like NeRF Aligning noise in 3D space independently of the object surface lead to deviations from the original geometry, even when relatively good shape
sometimes result
24

---

Published as a conference paper at ICLR 2025
(a) random timesteps
(b) + annealing timesteps
(c) + negative prompts
(d) + consistent noise
Figure 15: Ablation on the proposed improvements.
VSD
ISM
CFD (ours)
Timestep schedule
(sample t ~ U (lmin bmax))
maa
Emin
False Abrupt decrease Fixed
False
True
imax
Linearly decrease (ill
Linearly decrease (maile stage)
Eyin
Fixed
Linearly decrease (mult-sttge)
Noise Noisc type
Random
Inversion
Consistent
3D representation
Shape initialization Representation
Stable Diffusion--NDD
point-e point cloud-+3DGS
MVDreamecrDD NeRF(-+Mesh)
NeRF-+Mesh
Uncond prompt
LoRA network
True False
False
False
Negative prompt
True
True
Table 6: Comparison between pipelines of VSD, ISM and CFD.
formed, highly consistent region may be located away from the surface. In contrast, our noise design, which aligns with the object surface, avoids such issues.
Notably, the object surface can slowly change during the generation process, the noise for the same view in CFD is not strictly fixed even when = 0 in Eq. 11
F GRADIENT VARIANCE
We compare the gradient variance of different methods during training. We compute the scaled gradient variance by taking Exponential Moving Average parameters ú,, m,, from Adam optimizer for convenience. We report the scaled gradient variance on the parameters of nerf hash encoding with 10 seeds for each of the noising methods. a was calculated according to (where is the gradient):
mr R B[gh],
è. ~ E grl
(15)
sum(0 - m
sum Varí ))
sum(
sum(dr)
We report the gradient variance in training for VSD Wang et al. 2024a). SDS (Poole et al. 2023 Wang et al. 2023a), FSD (Yan et al. 2024) and methods in Tab.
25

---

Published as a conference paper at ICLR 2025
SDS
FSD
CFD (ours, when ? = 0)
Timestep schedule
Random
Annealing
Annealing
Same view noise
Random
Fixed
Mostly fixed (surface-dependent)
Different views noise
Independent
Aligned on sphere
Aligned on object surface
Table Comparision between SDS, FSD, and CFD.
VSD
SDS
FSD
CFD (ours)
(4)
5.165+
0.458
4.660-00.066
4.580+00.081
4.521++0.00
Table 8: Scaled Gradient Variance. Our CFD has the lowest gradient variance.
G CLEAN FLOW SDE
G.1 BACKGROUND
Song et al. (Sog et al. b021b) presented a SDE that has the same marginal distribution pa(a;) as the forward diffusion process (Eq. D. EDM (Karras et al. 2022) presented a more general form of this SDE, and the SDE corresponds to forward process defined in Eq. takes the following form:
V = logp((#))d
2 + Be or Vay at p + /23, dw;
-
(16)
O2
(d( Ot a1
F Tt 3,df)
t, + 23
dwn.
(17)
(x(e)
where dw,, is the standard Wiener process. If we set ar for all E [0, T). Eq. 17 will become the same SDE in EDM (Karras et al. (0022). The initial condition for the forward process is x ~ Pt. (x+4 1 L. (hg is small enough but 4, > to avoid numerical issues), and for the reverse process, it is x_ ~ (N I at t = T (Note that we also let or be small number but or > 0 to avoid numerical issues).
G.2 CLEAN FLOW SDE
The clean flow SDE takes the following form:
d=s. (d(:;) F ai 3,dt) (colar0 occt,t,y) E+), dé+ 3e + 22,,ww
(18)
where dw; the standard Wiener process. For the forward process, the initial condition t = t, is Sc ~ Po (x44), E. NNO, and as and ē. are independent. For the reverse process, the initial condition at t = 7 is t and é I).
Proposition (Clean flow SDE equivalent to diffusion SDE). In Eq. 18 if we define a new variable z'y according to
x' o + FoC.1
(19)
then x' and x+ in Eq. 17 have the same law (probability distribution) for all E [t.. i.e. Eq. and Eq. 17 are equivalent.
18
proof. We prove the equivalence by showing that the initial conditions and dynamics for x and a identical.
Initial conditions. For the forward process of Eq. 18 at t = t., x'_ = a,, Φc + o ē.. Thus,
2'1 Pt, (x')) according the definition of forward diffusion process (Eq. For the reverse
process of Eq. at T, æ ar oTê σrê_. So x' NN
26

---

Published as a conference paper at ICLR 2025
Dynamics. The dynamic of 244 can be derived according to:
d
=d =d - +
0z
at
Tt dd==
_ + +d(
4 +
an
(d( 01
S,dt) (Eslarāī -onuiitoteee +orE2, -2+) + +d(-0
é= +
of dēz
a
a1
at
(d(
02
at
d,dr)
(co(2"_, 1, - E+) + + 0t (d()))2+ + @t
dé+
a1
a1
(20)
(d(
3,dl) csl...) d( ) )+
+ BrEi +d -
del
at
at
at
at
ai
at
(d 01
F at
drd
col...) + 01
3,24dt +
0t dē
at
=d((
01
B,dl) csl...) +
01 Brejdl
F
0t
Bedl /2,3( (1) )dn,
11
at
at
an
04
-
-
Øt
Bidd) 6,(22_,4,) + 23r
dvv
So 44 and x+ follow the same dynamics.
We present a stochastic sampler in Algo. 3 that is equivalent Algorithm 2 in EDM (Karras et al. 2022 to show practice implementation of Fq. 18 for sampling.
G.3 PROPERTIES OF $5
G.3.1
&e ARE CLEAN IMAGES FOR ALL é [t,.7]
Lemma (Sample predictions are non-noisy images). The sample prediction of the diffusion model
Dt OzE 6 D1 1o t,
(21)
is weighted average of images in the target distribution (o)):
1 = 20 [c.]
(22)
Thus, 5. are non-noisy images. Furthermore,
Co (x)
Ft
(23)
27

---

Published as a conference paper at ICLR 2025
proof.
OzEa v)
a1
(xt + 07 V
log Pe (x:))
a1
o
+
V mc Pc(
Pı x1)
of
Væ, /
+
at
Pt x14
of
p(z m. log (
a1
(x())
of
(20) (x) a,au) )²
(24)
+
po (20
at
(x++)
207
p(ar)/20)( - o4 (aro claoo)
pt (xx)
S p( (20 + oss I 2o
plac ( p poo dao))
(x:
(2()
(x()))
[x at + Ons
wop(20o ac cdoon)
E (20 [ea)
Thus,
3t ar (x,,,,,,))
(25)
Ot
σt
Algorithm A SDE sampler that is equivalent to Algorithm 2 in EDM (Karras et al. 2022)
1: Input: Diffusion model (sample prediction) D&, tie (0.".,N) Tke (0..", .N-1): Smoise-
2: Output: ENN-
3: Initialize 60 ~ NO0 I). a6 = 0
4: for i E {0,.... N
1) do
5: Sample 6, ~ (O,, $2
6:
4 + 4i + Pili
7:
2i+1 2t2, v (4)Ps
8:
d, + (#;-- -DD + 40)/4
9:
E + + + fj)d
if if # 0then
11:
d + ( (1 + 144
12:
4c of + (li+1 - (+)( dd)
Apply 2nd order correction
13: end if
end for Return 2
15:
Proposition 2 (#C non-noisy images). 44 in Eq. 18 non-noisy images for all t (t.,, TT.
proof. Since the initial conditions of e (# for reverse process and t5. (x for forward process) implics & are initialized as non-noisy images, we only need to show that the dynamic of is will not introduce Gaussian noise into _
28

---

Published as a conference paper at ICLR 2025
The dynamic of _ can be reformulated as:
1
(dd(
St O1
F
St B,dt)
(Es(an0) + ofE1,t - =1],
at
a,a9 + +0 ar R + +, 0,64
-(d( (
F a
B,dt)
(26)
Oks
at
Tt F dtt
=(d(log
(#1 - E|zo + ozē=)).
As Eq. 26 shows that % is always moving towards non-noisy sample prediction at E [ for all / E [ts, 7), #% will be non-noisy for all E TT.
We also visualize e at random timestep & E [0, of Stable Diffusion (Rombach et al. 2022) sampling processes Appx. Fig. 17 to show that they are visually clean (non-noisy). We clean variable to refer to 1 in this work since it is always non-noisy.
G.3.2 INITIALIZTIONN OF 2
The initial condition of reverse-time clean flow SDE (Eq. 18 is given by x_ 0 Ondd NO I). This is consistent with a typical initialization of NeRF: the whole scene of the NeRF being all grey.
When we set _ 0 as the initial condition for the clean flow SDE, it corresponds the initial condition x_ (0, oz. KKrrra et al. 2022 in the diffusion SDE (Eq. 17). However, since we set small nonzero or at the beginning, the strict initial condition of the dillusion SDE should be pr (xT), which slightly different from (O, of I). In this case, we should ø ~ Po(x)) in the clean flow SDE to make the initial condition of the two SDE identical. Prior works usually ignore the small difference between (#rr) and V(O.of..1 and starts from pure noise when sam- pling (Lin et al. 2024), and from our practical observation, given different initial x j 0 but the same e_, clean flow SDE will yield almost identical outputs (given the same seeds), which implies the endpoints of øi are not sensitive initialization of 29. So we choose to set ã 0 in this work as the initial condition.
G.3.3 ENDPOINTS OF &f
At the end of the reverse-time clean flow SDE, a = æn po(o))) So कt also ends sample in the target distribution po po (2o) as w6 in the reverse-time diffusion SDE.
G.4 PROPERTIES OF é
E can be seen as the "pure noise" part in the clean flow SDE (Eq, [8). Notably, the evolution of é does not depend on áě and has closed-form solution. The dynamic of é is given by
dē F B,dt 22
(27)
The initial condition for é in both the forward and reverse process are é NN(0,))
G.4.1
CLOSED-FORM SOLUTIONS
For the forward process,
dlel' 8. 2,dr24) edó B.da + + b.d
eJo p.de 2,3ddu: ē4 Brete dode + ē Breto B.de
(28)
eló B.ds 223,dw.-
Integral on both side of Eq. 28 we have
dl D,ds 24 - ēo 1' 22 Prdr ardd
(29)
29

---

Published as a conference paper at ICLR 2025
Thus, obtain the solution of ē+:
E+ = e Jo 0,ds Eu + e e B,ds S:' 2B B.dr dw..
(30)
Similarly, we can obtain the solution of é=:
č_ = - 1 9,ds Eτ e S: 3,da [ V2 B,dr drD.. B.drd
(31)
Specifically, we can derive a closed-form formulation to compute é+ (t) given é- (t') fort' < from Eq. 30, which takes the following form:
(t) = 53 B.ds 2.ddd (t) V
(32)
= -- yé- (t) Vic,
(33)
where
7 e 8,ds 6 6~ 100
(34)
For é (1) and 1' 4,
غ_ (t) = 1i 2 (f') + V e 21" B
(35)
= =VI -1- vTc,
(36)
where
1= B,ds c c~ ((,,1).
(37)
G.4.2 SPECIAL CASE SOLUTION OF DDPM
Ho et al. 2020) corresponds to special choice of 3,, where B, = d(a./o.)/dt Karras
DDPM
/./ar
al. 2022). We present the solution of Eq. 35 when Be corresponds to the choice of DDPM the
following:
01/ar
ot/as (0-aaa oT arr/or E.
2
(t)
ēT + V 11
(38)
σT 0T/aT
Assuming a designed schedule such that k-step DDPM has a constant y in two consecutive steps as in Eq. 11 We get é_ (I) - (1 (1 - ) Thus, we obtain value of y in
Eq. that corresponds to DDPM:
log or/ar 0z/an
Te 0zt
7 = 1
(39)
σT/ oT
k
Putting a typical parameter configuration in our experiments with Stable Diffusion (DDPM sampler) into Eq. 39 where 0.212, 0 zs 0.60, T 0.974, o//or 12.59 and 25000, we get
~ 0.00024.
G.4.3 VARIANCE OF é
All vector components of ě are of unit variance for E [0, T):
+e 25 3, (' 28,8² 2 la 8.
Var(ē+,))= 25 B,ds
= = -25 B.ba I 23,0² Sú B s.dr ds)
(40)
-26 9,do (1 + L de Brdr)
- -25 B.ds (1 2 56 Bndr -1)
1,
30

---

Published as a conference paper at ICLR 2025
Figure 16: Visualization of noisy variable
Figure 17: Visualization of clean variable a;-
Va)(ē_,))= = -2 = Bdds te 25 8,ds [ T
23,e² 12 8, ds
=c- Bid0 { 2B, S2 Brdrds)
(41)
-2 B,d511 {¹ de2 B.dr)
2 8.do (1 - 1-+ o Brarr
=1 =1
G.5 CLEAN FLOW ODE
When we set 33 in clean flow SDE (Eq, 18). becomes determined and changes an ODE
(Eq. 8h. Furthermore, dē+ = 0 and thus é+ will become constant é. This ODE is the same ODE presented in FSD (Yan al. 2024). also equivalent the signal-ODE presented in BOOT Guu
et al. 2023 when the diffusion model is changed to sample cemple-prediction.
H
DISCUSSION ON THE CHOICE OF THE VARIABLE SPACE
H.1 GROUND-TRUTII VARIABLE
Apart from the clean variable at. FSD (Yan al. 2024) also defined another variable space that is visually clean, which is the ground- truth variable a.. ab is defined by
# 6,6 (a.. (a. (...,,))
(42)
a4 a is also known as the "sample prediction" of the diffusion model. The ODE on of is given by:
d -
de (x)),,t))
(43)
att
Concurrent work SDI (Lukojanov et al. 2024) shares an insight similar to ours by also using ren- dered images replace the "non-noisy variables" to guide 3D generation. The difference between SDI (Lukoianov al. 2024) and our method is that SDI replaced the ground-trutt truth variable a: with rendered image ge (c) but we replace the clean variable & with (c).).
Theoretically speaking, if it's just solve the OOD problem when using image PF-ODE a guid- ance for 3D generation, think it's both reasonable replace a and as with rendered images,
31

---

Published as a conference paper at ICLR 2025
since they are both non-noisy throughout the diffusion process (Lemma I and Proposition 2 How- ever, it's difficult to exactly compute the update rule in Eq. 43 since 2; is required on right hand side of Eq. 43 In order to recover 1t given o SDI needs to solve fixed point equation, which is hard to be solved (Lukoianov et al. 2024). In practice, SDI use loss gradient similar to ISM. SDI interpret the DDIM inversion as the approximated solution of the fixed point equation. Difficulties also appear in works that attempt to apply guidance on the ground-truth variable &, for conditional image generation, we can compute the evolution of oi exactly according to Eq. 18 without the need to solve a fixed
in UGD (Bansal et al. (023) and FreeDoM (Yu et al.) (023). In contrast,
point equation.
Additionally, another recent work ISM (iang et al. 2023) can also be viewed as replacing the ground-truth variable a: as discussed in SDI (Lukoianov et al. 2024). since the main difference between ISM and SDI loss is whether to apply text condition when computing DDIM inversion.
H.2 COMPARISON WITH CONSISTENT3D
Consistent3D (Wu et al. 2024b) introduced the Consistency Distillation Sampling (CDS) loss by modifying the consistency training loss within the score distillation framework. Their insights into the connection between SDS and Diffusion SDE align closely with ours. However, their CDS loss stems from the consistency model training loss, similar to how SDS is derived from the diffusion model training loss, disregarding the Jacobian term (Poole et al., 2023). In contrast, our CFD loss directly follows the principles of diffusion model sampling through the ODE/SDE formulation. The image rendered from specific camera view corresponds directly to a point on the ODE/SDE tra- jectory, resulting in distinct final training losses that differ from their CDS loss. Furthermore, our approach integrates a multiview consistent noising strategy, enhancing both consistency and robust- ness.
rigorous mathematical connection between score distillation and diffusion sampling compared with Consistent3D. Specifically: (i) While Con- form of SDE sampling, their proof relies sistent3D suggests that SDS can be interpreted approximating the diffusion process by assuming optimal training at each step, an assumption that may not hold practical experiments. In contrast, our approach does not rely on optimal training at every step. Additionally, our theory (Eq. 10 in our paper) covers broader range of diffusion SDEs, including EDM (Karras al., 2022) and PF-ODE as special case. (ii) The CDS approach lacks a direct correspondence to a probability flow ODE trajectory. while our interpretation establishes direct mapping between rendered images and points on the ODE/SDE trajectory.
From theoretical perspective, work provides
H.3 PROPERTIES OF CLEAN VARIABLE
Since clean flow ODE is special of clean flow SDE when 3e 0, 2 in the ODE also maintains the "clean properties" discussed in Appx. G.3
32


Reviews:
[{"review_id": "OiYRmSJuvM", "reviewer": "Reviewer_vQut", "summary": "This paper proposes the 'Consistent Flow Distillation (CFD)' for text-to-3D generation. It extends the success of SDE into 3D domain, and with its novel multi-view consistent Gaussian noise sampling, it demonstrates a simple yet effective ways to enhance the visual quality and diversity in 3D generation. Extensive quantitative and qualitative comparisons demonstrates its effectiveness compared to previous methods.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "[**Novelty**] \n- The adaptation of probability flow ODE (PF-ODE) with clean flow gradient from 2D images to guide 3D generation is innovative\n- The adaptation of multi-view consistent Gaussian noise ensures a unified appearance from all angles, which is the key to high-fidelity texture generation\n\n[**Significance**]\n- The propose design of multi-view consistent noise is useful for the whole community, its performance boost in 3D-FID and 3D-CLIP scores compared to SDS, ISM, and VSD baselines, and exhibits richer details and more photorealistic textures, providing a considerable improvement to text-to-3D generation.\n\n[**Completeness & Clarity**]\n- I like its various qualitative comparisons with different baselines and ablations, also its examples of diverse generation for the same prompt. \n- It is well-organized, and effectively explains advanced technical concepts, including clean flow gradients, PF-ODE, and the SDE-based sampling process.", "weaknesses": "- On significance and novelty, I think based on the current progress in the field of 3DGen AI, although CFD introduces innovative noise techniques, it doesn’t propose entirely new model architectures or evaluation metrics beyond standard score distillation approaches. This is more fundamental concern when existing 3D-generative models can generate high-quality 3D assets within minutes, which this approach can still take hours. \n- Limited Qualitative Examples for long and complex Prompts: although the paper includes various qualitative comparisons, additional examples, especially for complex prompts, could further enhance understanding of CFD’s limitations\n- I also feel that some example results are not sharp enough, like around line 820", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RcmXx4B4Ey", "reviewer": "Reviewer_DP49", "summary": "This work proposes Consistent Flow Distillation (CFD) to enhance generation diversity and quality in SDS-based text-to-3D generation task. By treating the SDS as a trajectory of SDE, the authors propose guiding the optimization process via consistent 2D clean flow gradients. A key insight is maintaining consistent 2D image flows across different viewpoints for generating high-quality 3D outputs. To achieve this, they present an algorithm for computing multi-view consistent Gaussian noise, aligning noise textures precisely on the 3D object's surface. Extensive experiments showcase the effectiveness of CFD over related methods like VSD, ISM.", "questions": "- The primary questions are raised in the Weaknesses part, which related to raising score.\n- Could the paper provide a detailed comparison regarding memory usage and training time costs to existing methods?\n- While MVDream is introduced to mitigate the Janus problem, there is a concern that it might overfit to the 3D training set, potentially resulting in object omission issues. Is there potential for CFD to address this drawback effectively? Typical prompts such as \"A squirrel playing the guitar,\" \"A pig wearing a backpack,\" and \"a bear playing an electric bass\" could shed light on this aspect.\n- An open question: amidst various efforts to enhance SDS optimization for improved quality, can the authors assert that their formulation stands out as the best in Table 1? \n- The significance of Figure 10 in the appendix, which likely demonstrates the effectiveness of CFD, suggests that its inclusion in the main body could boost the paper's impact and clarity.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper stands out for its robust presentation of results, thorough experimental analysis, and compelling evidence.\n- Introducing Consistent Flow Distillation, the paper leverages 2D clean flow gradients and multi-view consistent noise to elevate the diversity and quality of 3D generation.\n- Through empirical results, it is evident that the proposed CFD effectively enhances the diversity of generated outputs, showcasing its potency in improving the quality and variety of 3D-generated content.", "weaknesses": "- The stated contributions appear to overlap with existing methodologies.\n    - The utilization of SDE formulations mirrors the approach outlined in \"Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior.\" While Consistent3D emphasizes addressing the unpredictability inherent in SDE sampling by introducing a deterministic sampling prior, the rationale behind employing image PF-ODE to steer 3D generation remains ambiguous.\n    - The concept of multi-view consistent Gaussian noise is the same in \"Geometry-Aware Score Distillation via 3D Consistent Noising and Gradient Consistency Modeling.\" Despite the advancements in quality seen in this work, a detailed comparative analysis is warranted. These approaches all seem to draw inspiration from Integral Noise.\n\n- Introducing CFD could potentially inject more diversity into the 3D generation process. However, in SDS-based 3D generation, each iteration of inconsistent content distillation may exhibit the Janus problem. It remains uncertain whether CFD might improve multi-Janus issues, prompting the incorporation of MVDream for distillation. It could be beneficial to present results distilled from SDV2 or DeepFloy-IF to strengthen their argument.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cmZmqZaRVW", "reviewer": "Reviewer_kcGa", "summary": "This paper proposes Consistent Flow Distillation (CFD), which leverages gradients from 2D image flows to achieve better consistency across views, thereby improving 3D generation quality and diversity.", "questions": "See the weaknesses section. I would consider raising my socre if the author can address my concerns above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The generation quality of this proposed method is very high. The textures are realistic and detailed, providing a high level of visual fidelity that closely resembles real-world materials.\n2. This paper is well-written, with well-organized sections that guide the reader through theory and methodology.", "weaknesses": "1. Concurrent work. I believe it is necessary for the authors to clarify the distinction between their approach and “Consistent Flow Distillation for Test-to-3D Generation” within the main body of this paper. The ODF-based optimization and multi-view consistent Gaussian noise used here are quite similar to those in FSD, which warrants a more explicit comparison.\n2. Experimental Setup. (a) It’s unclear whether CFD utilizes MVDream in the comparison experiments, and if so, this may introduce an unfair advantage. (b) Only ten prompts are used in the comparative experiments. A broader evaluation set would improve the robustness of the evaluation. (c) FSD also should be included in the comparison for a more comprehensive evaluation. (d) The generation diversity hasn't been well evaluated. I also suggest that aesthetic evaluation metrics, e.g., LAION Aesthetics Predictor [1] and PickScore [2], can provide a more holistic assessment of the generated textures. (e) The ablation study lacks visualizations, which would help in understanding the impact of different components of the proposed method.\n\n[1] https://laion.ai/blog/laion-aesthetics. [2] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1QnxuatEkU", "reviewer": "Reviewer_a4cX", "summary": "The authors propose consistent flow distillation (CFD) strategy which can replace existing score distillation sampling (SDS) which leverages pre-trained 2D diffusion models for 3D generative models. The authors propose to guide 3D generation with 2D clean flow gradients operating jointly on a 3D object. They identify that a key in this process is to make the flow guidance consistent across different camera views.", "questions": "1. What happen if the proposed CFD is applied on DreamFusion pipeline which only replaces SDS to CFD while keeping all the other components the same?\n\n2. What happen if the proposed CFD is applied to image-to-3D models like Wonder3D? Please show some results more than text-to-3D task.\n\n3. Please give some results on user study for text-to-3D generative models where most of the recent text-to-3D generative models also show this results (or show SSIM or LPIPS results on image-to-3D tasks where there exist GT images on other camera views).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors well compare other proposed SDS variants including VSD and ISM.\n2. The author demonstrates the qualitative results of the proposed method compared to other variants.", "weaknesses": "1. The concern is that the proposed method leverages both MVDream and StableDiffusion2 as text-to-2D diffusion model, but other competitive methods, DreamFusion, ProlificDreamer, and LucidDreamer only StableDiffusion2. It means that the superiority of the proposed method might be from not the proposed CFD but from MVDream.\n\n2. The proposed method only do experiments on text-to-3D tasks where the proposed CFD can be applied to any X-to-3D models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)