DreamFlow: High-quality text-to-3D generation by Approximating Probability Flow
Authors: Kyungmin Lee
Kihyuk Sohn
Jinwoo Shin
Venue: ICLR 2024 spotlight (2024)
Keywords: Text-to-3D generation
Diffusion model
Score Distillation Sampling
Primary area: generative models

Abstract: Recent progress in text-to-3D generation has been achieved through the utilization of score distillation methods: they make use of the pre-trained text-to-image (T2I) diffusion models by distilling via the diffusion model training objective. However, such an approach inevitably results in the use of random timesteps at each update, which increases the variance of the gradient and ultimately prolongs the optimization process. In this paper, we propose to enhance the text-to-3D optimization by leveraging the T2I diffusion prior in the generative sampling process with a predetermined timestep schedule. To this end, we interpret text-to-3D optimization as a multi-view image-to-image translation problem, and propose a solution by approximating the probability flow. By leveraging the proposed novel optimization algorithm, we design DreamFlow, a practical three-stage coarse-to-fine text-to-3D optimization framework that enables fast generation of high-quality and high-resolution (i.e., 1024×1024) 3D contents. For example, we demonstrate that DreamFlow is 5 times faster than the existing state-of-the-art text-to-3D method, while producing more photorealistic 3D contents.

Published as a conference paper at ICLR 2024
DREAMFLOW: HIGH-QUALITY TEXT-TO-3D GENERA- TION BY APPROXIMATING PROBABILITY FLOW
Kyungmin Lee' Kihyuk Sohn² Jinwoo Shin'
KAIST "Google Research
(kynggmleee, (aawoss)eaait.coom ac. "yshyuks@google.com com
ABSTRACT
Recent progress in text-to-3D generation has been achieved through the utilization of score distillation methods: they make use of the pre-trained text-to-image (T2I) diffusion models by distilling via the diffusion model training objective. However, such an approach inevitably results in the use of random timesteps each update, which increases the variance of the gradient and ultimately prolongs the optimization process. In this paper, we propose to enhance the text-to- 3D optimization by leveraging the T2I diffusion prior in the generative sampling process with a predetermined timestep schedule. To this end, we interpret text-to- 3D optimization as multi-view image-to-image translation problem, and propose a solution by approximating the probability flow. By leveraging the proposed novel optimization algorithm, we design DreamFlow, practical stage coarse- to-fine text-to-3D optimization framework that enables fast generation of high- quality and high-resolution (i.e., 1024x 1024) 3D contents. For example, demonstrate that DreamFlow is 5 times faster than the existing state-of-the-art text-to-3D method, while producing more photorealistic 3D contents."
INTRODUCTION
High-quality 3D content generation is crucial for a broad range of applications, including entertain- reality, and robotics simulation. However, the current 3D generation process entails tedious work with 3D modeling software, which demands a lot of time and expertise. Thereby, 3D generative models (Gao et al., 2022; Chan et al., 2022; Zeng al., 2022) have brought large attention, yet they are limited by their generalization capability to creative and artistic 3D contents due to the scarcity of high-quality 3D dataset.
ment, gaming. bogmented//irtual/niin
Recent works have demonstrated the great promise of text-to-3D generation, which enables creative and diverse 3D content creation with textual descriptions (Jain al., 2022; Mohammad Khalid et al., 2022; Poole et al., 2022; Lin et al., 2023; Wang et al., 2023b; Tsalicoglou et al., 2023). Remarkably, those approaches do not exploit any 3D data, while relying the rich generative prior of text-to-image diffusion models (Nichol et al., 2021; Saharia et al., 2022; Balaji et al., 2022; Rombach et al., 2022). On this line, DreamFusion (Poole al., 2022) first proposed optimize 3D representation such that the image rendered from any view is likely to be that of sampled from reside in the high-density region of the pre-traine pre-trained diffusion model. To this end, they introduced Score Distillation Sampling, which distills the pre-trained knowledge by the diffusion training objective (Hyvärinen & Dayan, 2005). Since DreamFusion demonstrates the great potential of text-to-3D generation, subsequent studies have improved this technology by using different 3D representations and advanced score distillation methods (Lin et al., 2023; Wang al., 2023b).
However, score distillation methods exhibit a vighaneriance gradient, which requires lengthy optimization process to optimize a 3D representation. This limits the scalability to the usage of larger diffusion priors generate high-quality and high-resolutio 3D content (Podell et al., 2023). This in part due to the formulation of the score distillation method that aims to distill the diffusion prior of all noise level (i.c., Eq. 7). where the noise timesteps randomly drawn at each update. In contrast, conventional 2D diffusion generative processes implement a predetermined noise schedule
Visit project page for visualizations of method.

---

Published as a conference paper at ICLR 2024
A tiger dressed doctor
Corgi wearing top hat
A squirrel made out of fruit
A baby dragon hatching out of stone egg
Wedding dress made out of tentacles
Ficus plant on clay pot
Marble bust of fox head
Teal moped
Figure 1: Examples of 3D scene generated by DreamFlow. DreamFlow can generate photorealistic 3D models from text prompts with reasonable generation time (e.g., less than 2 hours), which have been possible by elucidated optimization strategy using generative diffusion priors.
that gradually transports the noise to the data distribution. This leads us to the following natural question: How to emulate the diffusion generative process for text-to-3D generation?
In this paper, we propose efficient text-to-3D optimization scheme that aligns with the diffusion generative process. To be specific, unlike the score distillation sampling that uses the training loss of the diffusion model an objective, method makes use of the diffusion prior in the generative (or sampling) process, by approximating the reverse generative probability flow (Song et al., 2020b;a). In particular, have framed the text-to-3D optimization problem, where we use Schrödinger Bridge problem (Schrödinger, 1932) to derive its solution with probability flow. Then, we propose match the trajectory of the probability flow to that from the pre- trained text-to-image diffusion model to effectively utilize its rich knowledge. Lastly, we approximate the probability flow cater to text-to-3D optimization and conduct amortized sampling to optimize multi-view images of a 3D representation. Here, our approach implements a predetermined noise schedule during text-to-3D optimization, as the generative 2D diffusion model does. With our new optimization algorithm, we additionally present practical text-to-3D generation framework, dubbed DreamFlow, which generates high-quality and high-resolution 3D content (see Figure 1). Our approach is conducted in coarse-to- fine manner, where generate NeRF, extract and fine-tune the 3D mesh, and refine the mesh with high-resolution diffusion prior (see Figure 3).
multi-view image-to- to-image translation
Through experiments on human preference studies, demonstrate that DreamFlow provides the most photorealistic 3D content compared to existing methods including DreamFusion (Poole et al., 2022), Magic3D (Lin et al., 2023), and Prolific (Wang et al., 2023b). We also show that DreamFlow outperforms ProlifiDDreamer with respect to CLIP (Radford et al., 2021) R-precision score (in both NeRF generation and 3D mesh fine-tuning), while being 5x faster in generation.
2

---

Published as a conference paper at ICLR 2024
2 PRELIMINARY
We briefly introduce the score-based generative models (SGMs) in the general context, which encompasses the diffusion models (DMs). Given the data distribution Pauu (x), diffusion process starts off by diffusing 2. Pasa( (Z)) with forward stochastic differential equation (SDE) and the generative process is followed by reverse SDE for time t E [o, T), given as follows:
dæ = f(x l)dl + g())dw, dæ = [f - و² (1) log (x)) dt + g())w.,
(1.2)
where f(::)) and s()) are the drift and diffusion coefficient, respectively, w is a standard Wiener process, and v. log (ar) is score function of the marginal density from Eq. 1, where po(a:)
Pdata (z). Interestingly, Song et al. (2020b) show that there is a deterministic ordinary differential equation (ODE), referred as Probability Flow ODE (PF ODE), which has same marginal density to (z():) throughout its trajectory. The probability flow ODE is given as follows:
5 (az - 2 - (1) px)) dt.
dæ
(3)
Here, the forward SDE is designed that pa+))) is sufficiently close tractable Gaussian distribution, so that one can sample from data distribution by reverse SDE or PF ODE. In particular, we follow the setup of (Karras et al., 2022) formulate the diffusion model, where we let f(x,)) t) 0. g(t) = (2) (t) for decreasing noise schedule o [0. 7] + R4, Also, we denote p(æ; ) be the smoothed distribution by adding i.i.d Gaussian noise of standard deviation a. Then the evolution of a sample 21 ~ p( ((1)) )) from time 4 to l2 yields 22 ~ p(2=2; a (12)) with following PF ODE:
dæ = (() (1) Va logp(a;; (2; (1))
where dot denotes derivateve... In practice, we solve ODE by taking finite steps over discrete time schedule, where the derivative is evaluated at each timestep (Karras et al., 2022; Song al., 2020a).
Training diffusion models. The diffusion model trains a neural network that approximates a score function of each p(z; by using Denoising Score Matching (DSM) (Hyvärinen Dayan, 2005) objective. In specific, let us denote a denoiser D(æ; o) that minimizes the weighted denoising error for samples drawn from data distribution for all a, i.e.,
Exo-pen En (A(=) (10 + +n;o) -
(5)
then we have Vo o)) (D(=;0) - /o² where X(or) is a weighting function and n is noise. The key property of diffusion models is that the training and sampling are disentangled; one can use different sampling schemes with same denoiser.
Text-to-image diffusion models. Text-to-image diffusion models (Nichol et al., 2021; Saharia et al., 2022; Rombach et al., 2022) are conditional generative models that trained with text embeddings. They utilize classifier-free guidance (CFG) (Ho & Salimans, 2022), which learns both conditional and unconditional models, and guide the sampling by interpolating the predictions with guidance scale wt D² a, y) = (1 + w) D(æ; (x; a. y) - w [(x; J), where y is a text prompt. Empirically. w > 0 controls the tradeoff between sample fidelity and diversity. CFG scale important text-to-3D generation, as for the convergence of 3D optimization (Poole et al., 2022; Wang et al., 2023b).
2.1 TEXT-TO-3D GENERATION VIA SCORE DISTILLATION S SAMPLING
Text-to-3D synthesis aims to optimize a 3D representation resembling the images generated from text-to-image diffusion models when rendered from any camera pose. This can be viewed as learning a differentiable image generator g(f, c) where 9 is a parameter for generator and c a condition for image rendering (e.g., camera pose). For 3D application, we consider optimizing Neural Radiance Fields (NeRFs) (Mildenhall et al., 2021) and 3D meshes with differentiable rasterizers (Laine et al., 2020; Munkberg et al., 2022). Throughout the paper, we omit and write g(0), unless specified.
Score Distillation Sampling (SDS) (Poole et al., 2022). SDS is done by differentiating the diffusion training objective (Eq. 5) with respect to the rendered image x g(0). Formally, the gradient of SDS loss is given as follows:
V LoLSDS (0; x = (g()) = =Ey, ((1)(x - D2(x + n: (1)) dx
00
3

---

Published as a conference paper at ICLR 2024
Intuitively, the SDS perturbs the image with noise scaled by randomly chosen timestep t, and guide the generator so that the rendered image moves to the higher density region. One can interpret the SDS the gradient of probability density distillation loss (Oord et al., 2018), which is equivalent to a weighted ensemble of variational inference problem with noise scales given as follows:
arg min E,,, ((tt)xx (((=;)(()) p(:; (t)))] for = =(t),
0
where we denote ((x) as an implicit distribution of the rendered image x = g(0). has been shown that SDS requires higher CFG scale than conventional diffusion sampling (e.g., w 100), which often leads to blurry and over-saturated output.
Variatioanl Score Distillation (VSD) (Wang et al., 2023b). To tackle the limitation of SDS, VSD solves Eq. 7 with particle-based variational inference (Chen et al., 2018) method principled by Wasserstein gradient flow (Liu, 2017). To approximate the Wasserstein gradient flow, it involves assessing the score function of rendered images Va log q( σ). thus they propose to train auxiliary diffusion model for approximation. In practice, they use low-rank adapter (Hu et al., 2021) to fine- tune the diffusion model during the optimization by minimizing Eq. 5 with respect to the rendered images. Let us denote the fine-tundd une-tuned diffusion model Dz(:: r) then the gradient of VSD loss is given by
dx
V VeLVsD (0; = g(0) = En,n (Dt) + n;)(()) - Dp(x + n; σ
(8)
00
Note that Eq. is equivalent to Eq. 6 if we consider an ideal denoiser, i.., D2(2 + nz; 0) x. In contrast to SDS, VSD allows CFG scale as low as conventional diffusion samplers (e.g.. = 7.5). thus resulting in highly-detainn considerably long time to generate high-quality 3D content.
and diverse 3D scene. Despite its promising quality, VSD requires a
2.2 SCHRODINGER BRIDGE PROBLEM
Schrödinger Bridges (SB) problem (Schrödinger, 1932; Léonard, 2013), which generalizes SGM
to nonlinear structure (Chen et al., 2021), aims at finding the most likely evolution of distribution between Fg ~ (=)) and Øy ~ (BTT with following forward and backward SDEs:
dæ = + - log (z)]dt + g(t)dw, dæ [f g2V = log 4 (z)]dt + g(t) (9, 10)
where t,, 6 are Schrödinger factors satisfying the boundary conditions ((x, 0) (z, 0) PA (m) and (xx T) f2x T) = Pn (ax). Eq. 9 and Eq. 10 induce the same marginal density Pe (a) almost surely and the solution of evolutionary trajectory satisfies Nelson's 's duality (Nelson, 2020): pt (x:)
(x))) (xx). Note that SGM is special case of SB if we set (xx) = 1 and (xx)) = (x()) for t E [0, Ti. SB is used for solving various image-to-image translation problems (Su et al., 2022; Meng al., 2021; Liu et al., 2023).
3 DREAMFILOW: ELUCIDATED TEXT-TO-3D OPTIMIZATION
We present DreamFlow, an efficient text-to-3D optimization method by solving probability flow ODE for multi-view 3D scene. At high level, we aim emulating the generative process of diffusion models for text-to-3D optimization, instead of utilizing diffusion training objective as in SDS. This is done by approximating the solution trajectory of probability flow ODE to transport the views of a 3D scene into the higher density region of data distribution learned by diffusion model. This, in turn, provides faster 3D optimization than SDS, while ensuring high-quality.
Challenges in solving probability flow ODE in 3D optimization. However, it is not straight- forward to utilize PF ODE for text-to-3D optimization as is. because of the inherent differences between 2D sampling and 3D optimization. First, unlike 2D diffusion sampling which starts off from Gaussian noise, the multi-view images of a 3D scene are initialized by the 3D representation. As such, it is natural to cast text-to-3I optimization as a multi-view Schrödinger Bridge problem. Second, text-to-3D generation requires transporting multi-view images to the data distribution, which pre- cludes the application of well-known image-to-image translation method (e.g., SDEdit (Meng et al., 1)).)). To overcome those challenges, we first derive a PF ODE that solves SB problem (Section 3.1), and present a method that approximates probability flow for text-to-3D optimization (Section 3.2). Finally, we introduce DreamFlow, which leverages proposed optimization algorithm for efficient and high-quality text-to-3I) generation (Section 3.3).
4

---

Published as a conference paper at ICLR 2024
3D Model
at ((()
add noise vn'
fine-tune
min | + =
render
D
add noise
update
a(t) x
Dol2 +
Dp
for
8
probabiitt How dz//dt
Figure 2: Proposed 3D optimization method APFO. APFO use predetermined timestep schedule for efficient 3D optimization. At each timestep i, we sample L multi-view images from 3D scene, and update 3D representation by approximation of probability flow computed by Eq. 13.
3.1 PROBABILITY FLOW ODE FOR SCHRÓDINGER BRIDGE PROBLEM
We consider the Schrödinger Bridge problem of transporting rendered image the data distribution, while using the diffusion prior. Let (()) be the distribution of rendered image æ = g(0). Our goal to solve SB problem that transports æ q(x) to Pdata by solving the reverse SDE in Eq. 10. To leverage the rich generative prior of pre-trained diffusion model, our idea is to match the marginal density of the evolution of SB by the density pr. where its score function can be estimated by pre- trained diffusson model Dp- Then from Nelson's duality, we have $(x,, (ar, t) = Pe(2)), where is equivalent to Vx log (xr) V pr()) - V log (xc)). By plugging it into Eq. 10, we have our SDE and the corresponding PF ODE given as follows:
1 f 1) 29 (1) ~V log (ar) V. log (2))) de.
dæc
(11)
While the score function Va lo (xx) can be estimated by pre-trained diffusion model, V log (ar)
is intractable in general. One can simply let ((x, t) 1, which is indeed equivalent to SDEdit (Meng al., 2021), I). but this might results in convergence a low-density region due to its high variance as similar to SDS. Instead, we directly approximate V. log (x(x) by leveraging SGM. Liu et al. (2023) showed that V log (x)) is indeed a score function of the marginal density of forward SDE (Eq. 1) for x q(x). Thus, we approximate the score V. log (x(x) by learning a SGM with the sample ~ q(x). Following (Wang et al., 2023b), this is done by fine-tuning the diffusion model Dø. initialized from Dy. using low-rank adapter (Hu et al., 2021).
3.2 APPROXIMATE PROBABILITY FLOW ODE (APFO)
Since the forward SDE has a non-linear drift, Eq. 11 is no longer diffusion process. Thus, we approximate the score functions V. log Pa (x) and V. log (x)) by adding noise n to the sample æ, and use diffusion models Dp and D&, respectively. Then the Eq. 11 with diffusion models is given by
(1) (D + n: (1) - (p + n; n;o(())) dt, dæ = (t)
(12)
where we note that the same noise n is passed to diffusion models to lower the variance. Also, we use decreasing sequence of noise levels (t) to ensure convergence to the data distribution.
Amortized sampling. The remaining challenge is to update the multi-view images within a 3D scene. To this end, we simple amortized sampling (Feng et al., 2017). Given an image renderer g(8, c) with camera pose parameter c, we randomly sample multiple camera poses to update a whole scene at each timestep. Formally, given decreasing noise schedule att) with timesteps {t,) N (i.e., (41) > ... > (),))), we randomly sample &, different views at each timestep 4; and update using the target obtained by Eq. 12. For / l,,...,44 AN and for e 1,.... li we do
arg min (lg(0, c) ag(æ + (le41 1:) where 9(0, c). and
d, a(li1)) ) a(4i) (Do( + n;o(t.)) Dz(x + n;o(()))). a(t+)
(13)
5

---

Published as a conference paper at ICLR 2024
"a 3d model of adorable cottage with thatched roof"
NeRF 256 256
512 512
mesh 1024 1024
Denoiser
Denoiser
Demotace
timestep
timestep
timestep
0.5
0.1
0.0
Stage
NeRF optimization
Stage 2 Mesh linetuning
Stage 3 Mesh Relinement
Figure 3: Coarse-to-fine text-to-3D optimization framework of DreamFlow. Our text-to-3D generation is done in coarse-to-fine manner; we first optimize NeRF, then extract 3D mesh and fine-tune. We use same latent diffusion model (denoiser 1) for first and second stage. Lastly, we refine 3D mesh with high-resolution latent diffusion prior (denoiser 2). At each stage, optimize with different timestep schedule, which effectively utilize the diffusion priors.
Here sg is a stop- grod-ggt operator. We refer our optimization method as approximate probability flow ODE (APFO), which we provide visual explanation in Figure 2, and the algorithm in Algorithm 1. Remark that we set different timestep schedules for different 3D optimization stages. For example, when optimizing NeRF from scratch, we start from large timestep fully exploit the generative knowledge of diffusion model. When fine-tune the 3D mesh we set smaller initial timestep to add more details without changing original contents.
Comparison with score distillation methods. The key difference of our method and score distilla- tion methods their objective in leveraging generative diffusion prior. Score distillation methods directly differentiate through diffusion training loss (i.e., Eq. 5), while our approach aim at matching the density of sampling trajectory. We remark that Eq. 12 similar to Eq. 8 as they both contain subtraction of score functions, but the difference occurs in its scaling of gradient flow. Empirically, our approach gradually minimizes the loss, while score distillation methods results in fluctuating loss (and norm of gradients) (see Figure 6). This often results in content shifting during optimization, resulting in bad geometry as well. See Appendix C for more discussion.
3.3 COARSE-TO-FINE TEXT-TO-3D OPTIMIZATION FRAMEWORK
We present coarse-to-fin fine text-to-3D optimization framework DreamFlow, which utilizes APFO in training 3D representations. DreamFlow is consists of three stages; first we train the NeRF from scratch. Second, extract 3D mesh from the NeRF and fine-tune. Finally, refine 3D) mesh with high-resolution diffusion prior to enhance aesthetic quality. Our framework is depicted in Figure 3.
NeRF optimization. We optimize NeRF from scratch by using multi-resolution hash grid encoder from Instant NGP (Müller et al., 2022) with MLPs attached to predict RGB colors and densities. We follow the practice from the prior works (Poole et al., 2022; Lin et al., 2023; Wang et al., 2023b) for density bias initialization, point lighting, camera augmentation, and NeRF regularization loss (see Appendix D for details). We use the latent diffusion model (Rombach et al., 2022) diffusion prior. During training, we render 256x 256 images and use APFO with initial timestep 1 1.0 decrease to t 0.2 with f views. We use the total of 800 dense timesteps unless specified.
Stage 1:
Stage 2: 3D Mesh fine-tuning. We convert the neural field into the Signed Distance Field (SDF) using the hash grid encoder from the first stage. Following (Chen et al., 2023), we disentangle the geometry and texture by optimizing the geometry and texture in sequence. During geometry tuning, do not fine-tune Da and let (= For geometry tuning, we use timesteps from 4 = 0.8 to IN = 0.4, and for texture tuning, we use timesteps from t 0.5 in 0.1. We all use = 5.
as done in PromificDreamerr (Wang al., 2023b).
Stage 3: Mesh refinement. We refine the 3D mesh using nigh-resolution diffusion prior, SDXL refiner (Podell et al., 2023), which is trained to enhance the aesthetic quality of an image. During refinement, we render the image with resolution of 1024x 1024, and use t 0.3 to tty = 0.0 with é = 10. Here we use timestep spacing of 10, which suffices to enhance the quality of 3D mesh.

---

Published as a conference paper at ICLR 2024
Dream Fusion
Ours
DreamPusion
Ours
liger easing an ice crewn co1te
orrchid flower planted in. clay pot
Magic3D
Ours
Magic3D
Ours
siher plotter piled high with frusts
beaudsful dress made out of garbege begs, on mennepuin ProliftDrreamer
ProlifiDrreamer
Ours
Ours
deimpenzze dresseed like Henry VIII king of England
small saguaro azctus planted xn a clay pot
Figure 4: Qualitative comparison with baseline methods. For each baseline method, present visual examples with same text prompt is given. Our approach presents more detailed textures.
EXPERIMENT
Throughout the experiments, we use the text prompts in the DreamFusion gallery² to compare our method with the baseline methods DreamFusion (Poole al., 2022), Magic3D (Lin et al., 2023), and ProlificDreamer (Wang et al., 2023b).
Qualitative comparisons. In Figure 4, we present qualitative comparisons between the baseline methods. We take baseline results from figures or videos in respective papers websites. Com- pared to SDS-based approach such as DreamFusion and Magic3D, our approach presents more photorealistic details and shapes. When compared to ProlificDreamer, our approach results more photorealistic details (c.g., eyes are more clearly represented for chimpanzee, and shadows are more clearly depicted in cactus) by exploiting high-resolution diffusion prior.
User preference study. We conduct user studies to measure the human preference compared to baseline methods. For each baseline method, we select 20 3D models from their original papers or demo websites (see Appendix A). Then we construct three binary comparison tasks (total 60 comparison) to rank between DreamFlow and each baseline method. For each pair, 10 human annotators were asked to rank two videos (or images videos were unavailable) based on following three components: text prompt fidelity, 3D consistency. and photorealism (see Appendix D). Results are in 'able Compared to DreamFusion Magic3I,, and PromificDreammer, DreamFlow consistently wins photorealism. While DreamFusion remains better, shows on par or better performance against more recent methods, Ma and ProlificDreamer, on 3D consistency and prompt fidelity.
Quantitative comparison. For quantitative evaluation, we measure the CLIP (Radford et al., 2021) R- R-precision following the practice of DreamFusion (Poole et al., 2022). We compare with Dream- Fusion and ProlificDreamer on NeRF generation, and Magic3D and ProlificDreamer for 3D mesh fine-tuning. For NeRF generation comparison, we randomly select 100 text prompts from DreamFu- sion gallery and reproduce results for PromificDreamer. For DreamFusion, we simply take videos from their gallery. For 3D mesh fine-tuning comparison, we select 50 NeRF representations from previous
hnctpe::/dreeanfusiiomaa (eattuu..o/galll htmml
7

---

Published as a conference paper at ICLR 2024
Table User preference studies. We conduct user studies measure the preference for 3D models. We present pairwise comparison between DreamFlow (ours) and baselines DreamFusion (DF) (Poole al., 2022), Magic3D (M3D) (Lin al., 2023), and ProlificDreamer (PD) (Wang et al., 2023b).
Method
tic
DF
ours
tic
M3D
ours
tic
PD
3D consistency
26.6%
31.0%
42.4%
36.9%
51.8 %
11.3%
31.9%
52.5%
15.6%
Prompt fidelity
22.8%
43.5%
33.7%
38.1%
52.4 %
9.52%
30.6%
58.8%
10.6%
Photorealism 82.1%
6.52%
11.4%
86.3%
8.93%
4.76%
43.8%
36.9%
19.4%
Table 2: Quantitative comparisons: NeRF opti- mization. We measure average CLIP R-precision scores rendered views from DreamFusion, Pro- lifieDreamer and DreamFlow.
Table 3: Quantitative comparisons: Mesh fine- tuning. We measure CLIP R-precision of ren- dered 3D mesh from Magic3D, ProlificDreamer, and DreamFlow.
Method
ViT-L/14 ViT-B/16 ViT-B/32
Method
ViT-L/14
ViT-B/16 ViT-B/32
DreamFusion 0.846
0.796
0.706
Magic3D
0.801
0.741
0.599
ProlificDreamer
0.875
0.858
0.782
PreaifiDDreamer
0.884
0.875
0,828
DreamFlow (ours)
0.905
0.871
0.798
DreamFlow (ours)
0.902
0.886
0,846
experiment, and reproduce Magic3D and ProlifiDDreamer using open-source implementation. We do
conduct mesh refinement for fair comparison. For evaluation, we render 120 views with uniform azimuth angle, and compute CLIP R-precision using various CLIP models (ViT-L/14, ViT-B/16, and ViT-B/32). Table 2 and Table presents the results, which show that our method achieves better quantitative score compared to prior methods.
Optimization efficiency. We show the efficiency of our method through comparing the generation speed. Note that we use single A100 GPU for generating each 3D content. For the NeRF optimization, we train for 4000 iterations with resolution of 256, which takes 50 minutes. For mesh fine-tuning, we tune the geometry for 5000 iterations and texture for 2000, taking 40 minutes total. Lastly, we refine mesh for 300 iterations, which takes 20 minutes. In sum, our method takes about 2 hours for a single 3D content generation. In comparison, DreamFusion takes 1.5 hours using TPU v4 chips in generating NeRF with resolution of 64, and Magic3D takes 40 minutes using 8 A100 GPUs in synthesizing 3D mesh with resolution of 512. ProlifieDreamer requires 5 hours in generating NeRF, 7 hours in mesh tuning. While Magie3D and DreamFusion could be faster using larger computing power, DreamFlow generates more photorealistic and high-resolution 3D) content.
Optimization analysis. We compare the optimization processes of APFO and VSD by comparing the values of loss and gradient norm per optimization iteration. For VSD, we compute loss in Eq. 13, which is equivalent to Eq. effective gradient. In Figure 6, we plot the loss values and gradient norms during NeRF optimization with each APFO and VSD. Remark that the loss and gradient norm gradually decrease for APFO, while they fluctuates for VSD. This is because of the randomly drawn timestep during optimization, which makes optimization unstable. Thus, we observe that VSD often results in poor geometry during optimization, or over-saturated (see Figure 16 in Appendix C).
Effect of each optimization stage. In Figure 5, provide qualitative examples the effect of coasre-to coatee-to-fine to-fine optimization. While DreamFlow generates high-quality NeRF, mesh tuning and refinement enhances the fidelity by increasing the resolution and using larger diffusion prior.
RELATED WORK
Text-to-3D generation. Recent works have demonstrated the promise of generating 3D content using large-scale pre-trained text-to-image generative models without using any 3D dataset. Earlier works used CLIP (Radford et al., 2021) image-text models for aligning 2D renderings and text prompts (Jain et al., 2022; Mohammad Khalid et al., 2022). DreamFusion (Poole et al., 2022) present
8

---

Published as a conference paper at ICLR 2024
Stage
Stagr
Stage
Ministure schioozzrr wooden scadpture
An imperial of England
Marbie bust of foz head
Figure 5: Ablation on the effect of DreamFlow coarse-to-fine text-to-3D optimization. Given stage generates high quality NeRF, stage improves the geometry and texture, and stage 3 refines the 3D mesh to add more photorealistic details.
359
APFO loss
(h) APFO gradient Norm
VSD loss
(d) VSD gradient
Figure 6: Optimization analysis. We plot the loss and gradient norm during 3D optimization for APFO and VSD. Remark that the loss and gradient norm gradually decreases for APFO, while VSD shows fluctuating loss and gradient norm.
text-to-3D generation by distilling score function of text-to-image diffusion model. Concurrently, Score Jacobian Chaining (SJC) (Wang et al., 2023a) established another approach use 2D diffusion prior for 3D generation. Magic3D (Lin al., 2023) extends DreamFusion higher resolution 3D content creation by fine-tuning on 3D meshes. Other works (Chen et al., 2023; Tsalicoglou et al., 2023) focused on designing better geometry prior for text-to-3D generation. ProlificDreamer (Wang et al., 2023b) enabled high-quality 3D generation by using advanced score distillation objective. All the prior works resort to score distillation methods, while we first demonstrate probability flow ODE based approach to accelerate the 3D optimization.
Transferring 2D diffusson priors. Due to the success of score-based generative models (Song et al., 2020b), especially diffusion models (Ho et al., 2020) have led to the success on texl-to-image synthesis (Nichol et al., 2021; Saharia al., 2022; Rombach al., 2022: Balaji et al., 2022). Many recent studies focus on transferring the rich generative prior of text-to-image diffusion models to generate manipulate various visual contents such images, 3D representations, videos, with text prompts (Brooks et al., 2023; Haque et al., 2023; Kim et al., 2023; Hertz et al., 2023). Our approach also shares the objective in exploiting the diffusion models. where utilizing APFO synthesize various visual contents is an interesting future work.
CONCLUSION
We propose DreamFlow, which enables high-quality and fast text-to-3D content creation. Our approach is built upon elucidated optimization strategy which approximates the probability flow ODE of diffusion generative models catered for 3D scene optimization. As result, significantly streamlines the 3D scene optimization, making it more scalable with high-resolution diffusion priors.

---

Published as a conference paper at ICLR 2024
By taking this benefit, propose three stage 3D optimization framework, where train NeRF from the scratch, fine-tune the mesh extracted from NeRF, and refine the 3D mesh using high- resolution diffusion prior. Through user preference studies and qualitative comparisons, show that DreamFlow outperforms prior state-of-the-ar method, while being 5x faster its generation.
Limitation. Since we are using pre-trained diffusion priors that do not have 3D understanding, results may not be satisfactory in some cases. Also, the unwanted bias of pre-trained diffusion model might be inherited.
ACKNOWLEDGEMENT
This work was supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No.2019-0-0007) Artificial Intelligence Grad- uate School Program(KAIST); No.2021-0-0228 00959, Few-shot Learning of Causal Inference Vision and Language for Decision Making). This work is in partly supported by Google Research grant and Google Cloud Research Credits program.
Artificial Intelligence Innovation Hub; No.2022-0-
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 183 18392-18402, 2023.
Eric R Chan, Connor 7. Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16123-16133, 2022.
Changyou Chen, Ruiyi Zhang, Wenlin Wang, Bai Li, and Liqun Chen. A unified particle-optimization framework for scalable bayesian sampling. arXiv preprint / 2018.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303. 13873, 2023.
Tianrong Chen, Guan-Hone Liu, and Evangelos A Theodorou. Likelihood training of schr)," odinger forward-backward backward sdes theory. arXiv preprint arXi:2111.112211 2021. 2021.
bridge using forward-
Yihao Feng, Dilin Wang, and Qiang Liu. Learning to draw samples with amortized stein variational gradient descent. arXiv preprint arXiv: /T07: 066666 2017.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01611. 2022.
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Advances Neural Information Processing Systems, 35:31841-31855 2022.
Ayaan Haque. Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2nerff: Editing 3d scenes with instructions. arXiv preprint 2xXiv:2333.12798 2023.
Amir Hertz, Kfrr Aberman, and Daniel Cohen-Or. Delta denoising score. arXiv preprint
arXiv:2304.07090. 2023.
Jonathan Ho and Tim Salimans. Classifier- free diffusion guidance. arXiv preprint arXiv:2207.125983 2022.
10

---

Published as a conference paper at ICLR 2024
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-668 0851, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1xXiv:2106.09680 2021.
Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi, Zheng-Jun Zha, and Lei Zhang. Dreamtime: An improved optimization strategy for text-to-3d content creation. arXiv preprint arXiv:2306. 12422. 2023.
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 867-876, 2022.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances Neural Information Processing Systems, 35:26565-26577. 2022.
Subin Kim, Kyungmin Lee, June Suk Choi, Jongheon Jeong, Kihyuk Sohn, and Jinwoo Shin. Collaborative score distillation for consistent visual synthesis. arXiv preprint arXiv:2307.04787, 2023.
Samuli Laine, Janne Hellsten, Tero Karras, Ycongho Scol, Jaakko Lehtinen, and Timo Aila. Modular primitives for perfrrrrffrm differentiable rendering. ACM Transactions on Graphics (TOG),
3-14) 2020.
Christian Léonard. survey of the schr\ odinger problem and some of its connections with optimal 05,,, 2013.
transport. arXiv preprint
Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 300-309, 2023.
Guan Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos Theodorou, Weili Nie, and Anima Anandkumar. I²S Image-to-image schrödinger bridge. arXiv preprint arXiv:2302.05872, 2023.
Qiang Liu. Stein variational gradient descent as gradient flow, Advances in neural information processing systems, 30, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:111100101, 2017.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.0107. 01007, 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Generating textured meshes from text using pretrained image-text models. models. In SIGGRAPH Asia 2022 conference papers, pp. 1-8, 2022.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4): 1-15, 2022.
11

---

Published as a conference paper at ICLR 2024
Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao. Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular models, materials, and lighting from images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8280-8290, 2022.
Edward Nelson. Dynamical theories of BBrowiin motion, volume 106. Princeton university press, 2020.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 2ttiv::2/22.10744 2021.
Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg. et al. Parallel wavenet: Fast high-fidelity speech synthesis. In International conference machine learning, pp. 3918-3926. PMLR, 2018.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprini arXiv:2307.01955 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22022 22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint 2rXiv:2202.005122 2022.
Erwin Schrödinger. Sur la théorie relativiste de l'électron l'interprétation de la mécanique quantique. In Annales de l'institut Henri Poincaré, volume 2, pp. 269-310, 1932.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087-6100 2021.
Kihyuk Sohn, Nataniel Ruiz, Kimin Lec, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al. Styledrop: Text-to-image generation in any style. arXiv preprint ar 2xxv:23066000888 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011 1. 134466, 2020b.
12

---

Published as a conference paper at ICLR 2024
Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image toagg translation. arXiv preprint arXiv:2203.08382, 2022.
Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from prompts. arXiv preprint arXiv:2304.12439, 2023.
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3481-5490, IEEE, 2022.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for generation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 12629, 2023a.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305. 162/3, 2023b.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeie, Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint arXiv:2210.06978. 2022.
13

---

Published as a conference paper at ICLR 2024
Appendix
A
ADDITIONAL QUALITATIVE COMPARISON
Corgi standing up drinking boba
Squirrel made out of fruit
Ficus planted clay pot
Marble bust of fox head
Wedding dress made out of tentacles
Figure 7: Qualitative Comparison with DreamFusion (Poole et al., 2022) (left) and FreamFlow (right).
14

---

Published as a conference paper at ICLR 2024
Cat lying on its side batting at ball of yarn
Teal moped
Corgi wearing a top hat
Octopus and giraffe having cheesecake
Tiger made out of yarn
Figure 8: Qualitative Comparison with Dream Fusion (Poole et al., 2022) (left) and DreamFlow (right).
15

---

3d model of adorable cottage with thatched roof Published as a conference paper at ICLR 2024
Tower bridge made out gingerbread candy
A stack of pancakes covered in maple syrup
tarantula, highly detailed
A made out of sushi
Figure 9: Qualitative Comparison with Magic3D (Lin et al., 2023) (left) and DreamFlow (right).
16

---

Published as a conference paper at ICLR 2024
rabbit, animated movie character, high detail 3d model
beautiful dress made out of garbage bags, mannequin. Studio lighting, high quality, high resolution.
baby dargon hatching out of a stone egg
small seguaro cactus planted a clay pot
"the Imperial State Crown of England."
Figure 10: Qualitative Comparison with Magic3D (Lin et al., 2023) (left) and DreamFlow (right).
17

---

Published as a conference paper at ICLR 2024
model of a house in Tudor style
An imperial state crown of England
car made out of sushi
Figure II: Qualitative Comparison with PremificDreamer (Wang et al., 2023b) (left) and Dream-
Flow (right).
18

---

Published as a conference paper at ICLR 2024
A sliced loaf of fresh bread
Michelangelo style statue of dog reading cell phone
tarantula, highly detailed
Figure 12: Qualitative Comparison with PreaificDreamer (Wang et al., 2023b) (Ieft) and Dream- Flow (right).
19

---

Published as a conference paper at ICLR 2024
Stage 2
Stage 3
silver plate platter piled high with fruits
Sushi made out of car
Stage 2
Stage 3
Cat lying its side batting at ball of yarn
Squirrel made out of fruit
Figure 13: Qualitative comparisons Stage and Stage 3 mesh fine-tuning. We provide additional qualitative results the ablation of using Stage 3 Mesh refinement. We zoomed in region of rendered view to demonstrate the effect of stage 3 mesh refinement, Remark that Stage 3 mesh results in more high-contrast images with better textures.
20

---

Published as a conference paper at ICLR 2024
(a) APFO
(b) VSD
(c) VSD+linear decrease loss
(d) VSD+DreamTinc
Figure 14: 2D experiments. We provide qualitative results of the 2D image generation with (a) APFO, (b) VSD, (c) VSD with lincarly decreasing timestep schedule, and (d) VSD with DreamTime (Huang al., 2023). Remark that APFO generates high-fidelity image with only 200 optimization steps, while VSD and its variants generate image with blurry artifacts.
B FURTHER EXPLANATION
Difference between APFO and VSD. Here we provide further details difference between APFO and VSD. First, remark that the proposed method APFO, is different to VSD in its implementation. In specific, the major difference is that APFO compute the time-derivative the noise scale, i.e.
a(t) o13) (e.g. Eq. 11), which is designed to transport the diffusion latent to be the next noise level from the derivation of probability flow ODE. On the other hand, score distillation methods, e.g. VSD, leverages the empirically designed weighting function A(t) = (t).. Those design choices of APFO and VSD are different due to their intrinsic motivation. APFO aims to approximate the probability flow ODE with decreasing timestep schedule, so that at the terminal step (i.c., as (t) + O, the optimized images sampled from data distribution. On the other hand, VSD aims to minimize the ensemble of KL. divergence on various noise levels, where particle-batt based variational inference algorithms were used.
Moreover, APFO is different from VSD with a simple timestep annealing method. This is because APFO employs dime-derivative of noise level, which is the exact amount to move to the next noise
21

---

Published as a conference paper at ICLR 2024
Stage 1. NeRF optimization
Results
Stage 3D Mesh finetuning
Stage 3. 3D Mesh refinement
Figure 15: Qualitative visualization text-to-3D optimization process of DreamFlow. We show that DreamFlow generates fast and high-quality 3D representations from text prompt by using pre- scheduled timesteps.
level. However, implementing VSD with decreasing noise-level does not involve such design choice. For better explanation, we provide an 2D image generation experiment which compares APFO with default VSD (i.e. random timestep sampling) and VSD with timestep annealing (linearly decreasing as in APFO). Also, we provide additional comparison with Dream (Huang et al., 2023), which proposed to sample timesteps according to predetermined weighting functions. For implementation, we use 200 optimization steps by using default DDIM (Song et al., 2020a) timestep schedule. For other methods, we follow the experimental setup in ProlificDreameer (Wang et al., 2023b) paper, with 500 optimization steps. For VSD with linearly decreasing timestep, we linearly decrease timestep simply from 1.0 to 0.O, and for VSD with DreamTime, we follow their implementation of weighting function.
The results are shown in Figure 14. One can observe that the image generated by APFO results in more sharp details, while other images remain blurty artifacts. Also, the blurry artifact does not diminish even when using annealing timestep schedule for VSD VSD scheduled with DreamTime. Also, APFO generates a more faithful image even when using 100 or 200 number of steps, while other methods use 500 steps. This 2D experiment show that 1) APFO is only different from simply taking VSD with timestep annealing, but also presents a more faithful image, and 2) by approximating the probability flow of pretrained diffusion model, APFO is able generate image with fewer optimization steps.
C ABLATION STUDY
Effect of predetermined timestep schedule. To further demonstrate the effect of predetermined timestep schedule, we plot the optimization processes of DreamFlow framework. In Figure 15, we visualize the interim results of NeRF and meshes during 3D optimization. We remark that our framework using scheduled timesteps are similar to the ancestral sampling of text-to-image diffusion models that we first identify the geometry with high noise scales (i.c., large timestep), then provide more details by using low noise scales (i.e., small timesteps). The refinement stage is involved to enhance the quality of 3D meshes, which a common practice in 2D image sampling, where image-to- mmage-to-image diffusion models are used. While method show how emulate the 2D -o--tigiigeee sampling procedure for text-to-3D generation, some other applications in 2D spaces be further considered, e.g., instruction-guidee editing (Brooks et al., 2023), personalization (Ruiz
22

---

Published as a conference paper at ICLR 2024
5000 iter
10000 iter
20000 iter
2000 iter
3000 iter
4000 iter
ProlifiDDreamer
DreamFlow
Figure 16: Qualitative comparison between VSD and APFO optimization process. Due to the random timestep. VSD often changes the object to bad geometry, while APFO consistently move to the data distribution. Thus is efficient and results in better 3D generation.
ProlifiDDreamer
Ours
Small lavender isometric soft lighting, unreal engine render, voxels.
DSLR photo
shopping
Figure 17: Complex NeRF generation. We compare ProlifieDreamer on scene (first row) and object-centric scene (second row) generation. Our method generates higher fidelity NeRF scene, while being more faster in its generation.
et al., 2023; Gal et al., 2022), or refcrence-guided stylization (Sohn et al., 2023), which we leave for future work.
Qualitative comparison on 3D optimization methods. As shown in Figure 6, we show that APFO gradually decreases the loss and gradient norm during its optimization, while VSD suffers from fluctuating loss. We remark that using random timestep in score distillation (i.c., VSD) inefficiently prolongs the optimization, while often results in content shift during optimization, which ultimately results in bad geometry. Here, provide qualitative examples in such cases, where show how APFO mitigates such issue by using decreasing timestep schedule. In Figure 16, we depict the optimization process of VSD and APFO. First, that even though APFO trained shorter than VSD, APFO converged to a valid 3D content. While length optimization of VSD leads to detailed texture, the geometry tilted and the undesirable features are appended throughout the optimization. On the other hand, APFO consistently update the feature with decreasing noise scales, which lead to coherent geometry and texture.
23

---

Published as a conference paper at ICLR 2024
Complex NeRF generation. Following the setup from ProlificDreamer (Wang et al., 2023b), evaluate our method in complex and object-centric scene generation. This is done by using their NeRF sene-initialization by using different spatial density bias. In Figure 17, show the results compared to PromificDreamer. Note that our approach generates scene with higher fidelity even with shorter optimization iteration.
24

---

Published as a conference paper at ICLR 2024
Algorithm Approximate probability flow ODE (APFO)
1: procedure APFO(g, o)(a;o), Do(o;o), a(t), a(t), No (O)......)))
2: Initialize 0
b 3D scene initialization
3: for E (O..... N 1) do
4: for j E ..) do
b Sample & views for amortized optimization
5: Sample 6j and and render x == c00,c))
6:
Sample n ~ -N (t.)))
@ + o no (=
Update & using DSM objective Sample another random noise
8:
Sample n ~ ~N,0
((4) + Dx +n';
9:
Evaluate dæ/d its
(f.)
10:
(1++1 1:
Take Euler step from f; f++11
11:
0 - 1/2 (ollg(0, (3) 2113
b. Update using updated target Return optimized 3D scene
12: return
D EXPERIMENTAL SETUP
D.1 3D SCENE OPTIMIZATION
Diffusion model configuration. We use Stable Diffusion 2.1 (Rombach et al., 2022) for our latent diffusion prior of stage and stage 2, and Stable Diffusson XL (SDXL) Refiner (Podell et al., 2023) for stage 3. When estimating Va log (2)),, we use Stable Diffusion 2. 1vv, a v-prediction model (Salimans & Ho, 2022) for stage and stage 2, and use same SDXL Refiner for stage 3. Following (Wang et al., 2023b), we train auxiliary camera pose embedding MLP, which is added to timestep embedding for each U-Net block. However, we find training camera embedding also works well. We use BulerDiscreteScheuur
for our noise schedule and timestep.
Scene representation. For density bias initialization, we follow Magic3D (Lin et al., 2023) and ProlificDreamer (Wang et al., 2023b). We set the camera distance from 1.0 to 1.5, bounding box size as 1.0. We use softplus activation for the density prediction and add spatial density given by (u) = A- (1 - /lll//c), where fu is a 3D coordinate, X 10 is density bias scale, 0.5 is the offset scale. We same light sampling strategy of Magic3D. We use background MLP that learns background color following (Poole et al., 2022).
NeRF optimization. We use hash grid encoder from Instant NGP (Müller et al., 2022), with 16 levels of hash dictionaries of size 219 and feature dimension of 4. We use 512 samples per ray in rendering. During NeRF optimization, we found that orientation loss (Verbin et al., 2022) helps consolidating the geometry. We do not use sparsity regularization, which we find it hurts geometry. During optimization, we use AdamW optimizer (Loshchilov & Hutter, 2017) where we train the grid encoder with learning rate le 2, color and density network with learning rate le 1---, 3, and background MLP with learning rate le 3 le- 4. We do not use shading, because it distracts learning texture.
Stage 2: 3D mesh fine-tuning. We extract 3D mesh from stage NeRF using DMTet (Shen et al., 2021). We then fine-tune the geometry by rendering normal maps (Chen al., 2023). We find that finetuning D& with APFO for geometry tuning is not that useful, thus we simply let Dz(x; σ) x for given rendered image, and use same APFO algorithm. We fine-tune the geometry with AdamW optimizer of learning rate le 2. For texture fine-tuning, we use AdamW optimizer with learning for color network.
rate le 2 for grid encoder,
Stage 3: 3D mesh refinement We continue finetuning the 3D mesh from stage 2. Remark that SDXL requires high memory and thus it takes much longer time in optimization due its size of U-Net. We find that using timestep spacing of 10 works well for SDXL, which enhances efficiency. We use Adam W optimizer with learning rate 1e for grid encoder, le for color network.
25

---

Published as a conference paper at ICLR 2024
D.2 EVALUATION METRICS
D.3 HUMAN SURVEY
For each baseline DreamFusion (Poole al., 2022), Magic3D (Lin al., 2023), and Prolific- Dreamer (Wang et al., 2023b), we construct 20 binary pairs to rank with our results. We collected 10 binary ratings for each pair, total 600 human ranking were collected. To properly evaluate the quality of 3D content, we ask the following to the human raters:
1. 3D consistency Which 3D item has more plausible and consistent geometry?
2. Prompt fidelity Which video or images best respects the provided prompt?
3. Photorealism Which video or images has more photorealistic details?
For the 3D consistency, better shape among different views. For prompt fidelity, users were asked to select the item that is more relevant the meaning of text prompt. For the photorealism, users asked to select the item that has more detailed textures that resembles the real 3D objects. The users can select unknown or tie, to refrain their selection.
asked to select the item that has more geometric consistency. i.e.,
D.4 CLIP R- R-PRECISION.
CLIP R-Precision measures the fidelity of the image in relation to its text input by using CLIP image-text retrieval. For the default CLIP R-precision measure, the distractors, i.c., the captions that distracts the prompts to be retrieved easily, were constructed. We found that the prompts in DreamFusion gallery share similar objects and attributes (e.g., object made out of salad, made out of wood, etc), use 397 prompts the full prompt set. During evaluation, images of 3D scene for 120 uniformly distributed azimuth angles with elevation angle of 15. The camera distances 1.2 default. Then compute CLIP score for the whole images and the full text prompts, and compute the precision with R
render the albedo
26


Reviews:
[{"review_id": "xE9qo4PrrK", "reviewer": "Reviewer_9SLV", "summary": "The authors propose a three-stage coarse-to-fine text-to-3D generation pipeline capable of producing high-resolution 3D assets from given text prompts. This involves the use of a variant of the SDS loss, Approximate Probability Flow ODE (APFO), to optimize a 3D representation learning from text-to-image priors. NeRFs and Meshes are used as the 3D representations at different stages, with the SDXL refiner applied to further enhance the extracted 3D meshes.", "questions": "1) In Fig.6, the stability of the APFO loss is attributed to the non-increasing timestep scheduling, as opposed to the two-stage stratified random scheduling used in the VSD loss from ProlificDreamer. However, the convergence of the loss curves may not precisely reflect the training quality of the 3D representation due to the random timestep sampling in VSD. Could the authors provide further clarification on this point?\n\n2) Based on the above question, what would be the outcome if the VSD loss were applied using the same timestep scheduling as proposed by the authors? Moreover, the predetermined timestep selection process remains unclear, particularly regarding the decreasing ratio over iterations.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and easy to follow. The perspective of considering SDS as a Schrödinger Bridges (SB) problem is insightful. The visual results appear comparable, and the quantitative scores are slightly better to the baselines. Notably, the proposed method demonstrates faster convergence than state-of-the-art methods.", "weaknesses": "1) While the authors provide insightful interpretations of the SDS method as an SB problem, its implementation is quite similar to the prior work ProlificDreamer. Additionally, timestep scheduling has also been proposed in previous works [1, 2, 3], limiting the impact of the proposed method.\n\n2) When viewed as an SB problem, the proposed method using a LoRA-fine-tuned score function seems to have difficulties in generating diverse results. Further explanations are encouraged.\n\n3) The improved numerical scores, as shown in the paper, are modest and are computed based on only 20 3D assets. Furthermore, some results contain weird textures, as observed in the example of \"A stack of pancakes covered in maple syrup.\"\n\n4) The results after the third stage exhibit color bias (Fig.5), and most generated 3D assets do not appear significantly improved after SDXL fine-tuning, except for the cottage example in Fig. 13. To explain this, additional ablation studies are recommended to fully evaluate the effectiveness of SDXL.\n\nMiscellaneous:\n\n1) An explanation of the method referred to as \"tie\" is missing in Table 1.\n\n2) There appear to be typos in Eq.13, with \"\\Phi\" possibly intended to be \"\\phi,\" and \"q\" possibly intended to be \"p.\" Otherwise, clarification regarding the transition from Eq.12 to Eq.13 would be appreciated.\n\n\n[1] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, Wang et al., NeurIPS 2023.\n\n[2] HiFA: High-fidelity Text-to-3D with Advanced Diffusion Guidance, Zhu et al., 2023.\n\n[3] DreamTime: An Improved Optimization Strategy for Text-to-3D Content Creation, Huang et al., 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8MwmVHPFiz", "reviewer": "Reviewer_pJwX", "summary": "This work designs DreamFlow, a practical three-stage coarseto-fine text-to-3D optimization framework that enables fast generation of highquality and high-resolution (i.e., 1024×1024) 3D contents. The experiments show that the framework outperforms exisiting baelines in both user-study and quantitative results.", "questions": "1. What is the CFG scale used in 1/2/3 stages?\n2. How is the GPU memory consumation of finetuning the LoRA of SDXL in stage3?\n3. Can you provide the 2D experiment of APFO (using APFO to optimize a 2D image)? A 2D experiment will demonstrate the effectiveness of APFO better.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and the experiments are sufficient.\n2. The theoretical part seems sound to me.\n3. Some results have good visual quality.", "weaknesses": "1. My main concern is that the proposed APFO is too similar with VSD. Maybe a more detailed explanation should be given to demonstrate the difference between VSD and APFO.\n2. The color of some results are kind of \"green\" or \"brown\" (including the cactus in Fig. 4, the corgi in Fig. 7, etc.), can you explain the reason?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IkH8TGtaSw", "reviewer": "Reviewer_ZxmY", "summary": "This paper proposes a new method for text-to-3D from pre-trained 2D diffusion model. It formulates the 3D generation problem as a Schrodinger Bridge problem that transports multi-view renderings to the data distribution defined by the 2D diffusion model. An amortized sampling scheme is also proposed to avoid using random time step sampling during optimization. The method has more stable optimization loss variation and yields better quality results than previous methods, according to user preference study.", "questions": "No questions.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The formulation of text-to-3D as solving a Schrodinger Bridge problem is a novel perspective and insightful.\n\n2. The proposed amortized sampling method stabilizes training loss (Figure 6).\n\n3. User study is conducted to showcase the effectiveness of the method, which is more convincing than other quantitative metrics.", "weaknesses": "Although motivated differently, the final ODE (equation 12) looks very similar to VSD (equation 8). It looks like the main benefit of more stable training (Figure 6) comes from the amortized sampling (Section 3.2) where there is an inner loop for each t and t decreases gradually. I think the same thing can be implemented for VSD as well, but no such comparison is shown. Maybe this sampling method is more naturally motivated for this method than VSD. But if the above understanding is correct, I think it would be better if the authors can be more explicit about the similarity and difference between the two algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QhrFcs0GSL", "reviewer": "Reviewer_paN1", "summary": "The paper presents a novel approach to text-to-3D generation named DreamFlow, which is an optimization-based method similar to score distillation-based methods (e.g., DreamFusion, Magic3D, ProlificDreamer). By reframing text-to-3D optimization as a multi-view image-to-image translation problem, the authors propose a new gradient update rule to drive the optimization. Besides, through the use of a predetermined timestep schedule, the algorithm improves the speed and quality. The paper claims that DreamFlow is 5 times faster and produces higher quality 3D content than existing methods.", "questions": "1. Why is the variance in the norm of the gradient (which you describe as instability) a detrimental factor? Is there a rigorous and detailed mathematical explanation for this?\n\n2. How is the time schedule related to your modeling? Is your model only applicable to a linear schedule?\nWhat distinguishes your method from the timestep annealing mentioned in works like DreamTime?\n\n3. Does your modeling approach have any broader applications? For instance, are there applications that could not be achieved in previous score distillation methods but might be feasible with your model?\n\n4. Please elaborate on the differences and similarities between your method and VSD, as the final formulations appear to be nearly identical.\n\n5. Please elaborate in detail on where the improvements in experimental results come from. For example, why is it faster, why do you think the results are better than VSD, and have you conducted independent ablation studies? (The current ablation study seems to be of limited significance, as adding each stage naturally leads to better results, but this is not the focus or contribution of the paper).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper offers a different perspective on how to model 3D generation problems. The authors interpret text-to-3D optimization as a multi-view image-to-image translation problem and propose a solution by approximating the probability flow.\n2. The final experimental results of this work are fairly good. In terms of the same quality, it is five times faster than VSD.", "weaknesses": "1. Lack of Technical Contribution: \n(1). Although the authors are developing an optimization loss from a different angle, the algorithm they ultimately present is actually the same as Variational Score Distillation (VSD). Similarly, I also noticed that the results obtained by the authors are similar to those of VSD. From this perspective, it doesn't seem sufficiently novel.\n(2). The author mentions that score distillation uses randomly drawn time steps at each update. However, there have already been many works that use a linear time schedule, which decreases from large t to small t during optimization. Additionally, there is a specialized work (DreamTime) that specifically studies this point. From this perspective, the technique does not appear to be original or novel.\n\n2. Unclear Connection Between Motivation and Technical Details:\n(1). From the abstract and introduction, it is evident that the authors' motivation is to address the issue of gradient variance. However, the methods section does not clearly explain how the proposed method or modeling perspective is related to the problem of gradient variance. \nThey only visualize the gradient norm in the experimental section at the end to indicate that it is more stable (of \"norm\", instead of \"direction\"). However, this also relates to the time schedule; a random time schedule would naturally result in varying magnitudes of the gradient because the intensity of diffusion output scores can vary based on the timestep t.\n(2). It remains unclear why this (\"norm\") variance is problematic for the optimization process. How is this demonstrated? For optimization, the direction of the gradient seems to be more important than its magnitude. In the experimental section on page 8, the author states, \"This is because of the randomly drawn timestep during optimization, which makes optimization unstable. Thus, we observe that VSD often results in poor geometry during optimization, or over-saturated.\" I am unclear about the logical relationship indicated by \"thus\" here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)