DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation
Authors: Yukun Huang
Jianan Wang
Yukai Shi
Boshi Tang
Xianbiao Qi
Lei Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Score Distillation
3D Content Creation
Diffusion Model
Primary area: generative models
TL;DR: We analyze the drawbacks of random timestep sampling in score distillation and propose a non-increasing timestep sampling strategy.

Abstract: Text-to-image diffusion models pre-trained on billions of image-text pairs have recently enabled 3D content creation by optimizing a randomly initialized differentiable 3D representation with score distillation. However, the optimization process suffers slow convergence and the resultant 3D models often exhibit two limitations: (a) quality concerns such as missing attributes and distorted shape and texture; (b) extremely low diversity comparing to text-guided image synthesis. In this paper, we show that the conflict between the 3D optimization process and uniform timestep sampling in score distillation is the main reason for these limitations. To resolve this conflict, we propose to prioritize timestep sampling with monotonically non-increasing functions, which aligns the 3D optimization process with the sampling process of diffusion model. Extensive experiments show that our simple redesign significantly improves 3D content creation with faster convergence, better quality and diversity.

Published as a conference paper at ICLR 2024
DREAMTIME: AN IMPROVED OPTIMIZATION STRAT- EGY FOR IIFFUSIONGGIDED 3D GENERATION
Yukun Huang" Jianan Wang'=;, Yukai Shi', Boshi Tang', Xianbiao Qi', Lei Zhang'
IIneernational Digital Economy Academy (IDEA)
The University of Hong Kong (huangyukun, wangj j anaa, sukyukai, wtw.lyykkiiiiiiiiaoee qixinnbiaa l leizhang dea edu .du.co
Text-to-3D Object
Image-to-3D Object
Text-to-3D Avatar
SDS
TP-SDS (Ours)
DreamFusion
Magic3D
Zero123
DreamWaltz
Text-to-3D
Scene
TP-VSD (Ours)
VSD
Prolific
Dreamer
Figure 1: DreamTime enables higher-quality 3D generation across different methods, includ-
ing: DreamFusion Poole et al. (2022), Magic3D Lin et al.] (2022), Zero123 Liu et al. (2023a), DreamWaltz Huang et (2023). and PremificDreamerr Wang al. (2023).
ABSTRACT
Text-to-image diffusion models pre-trained billions of image-text pairs have recently enabled 3D content creation by optimizing a randomly initialized dif- ferentiable 3D representation with score distillation. However, the optimization process suffers slow convergence and the resultant 3D models often exhibit two limitations: (a) quality concerns such as missing attributes and distorted shape and texture; (b) extremely low diversity comparing to text-guided image synthesis. In this paper, we show that the conflict between the 3D optimization process and uniform timestep sampling in score distillation is the main reason for these limi- tations. resolve this conflict, we propose to prioritize timestep sampling with monotonically non-inee non-increasing functions, which aligns the 3D optimization pro- cess with the sampling process of diffusion model. Extensive experiments show that simple redesign significantly improves 3D content creation with faster convergence, better quality and diversity.
INTRODUCTION
Humans situated in 3D environment. To simulate this experience for entertainment search, we require a significant number of 3D assets to populate virtual environments like games and robotics simulations. Generating such 3D content is both expensive and time-consuming, cessitating skilled artists with extensive aesthetic and 3D modeling knowledge. It's reasonable to
"Equal contribution.
I Work done during an internship at IDEA.
"orresponning author.

---

Published as a conference paper at ICLR 2024
inquire whether we can enhance this procedure make it less arduous and allow beginners to create 3D content that reflects their own experiences and aesthetic preferences.
Recent advancements in text-to-image generation (Ramesh et al. 2022; Saharia et al., 2022 Rom- bach ct al. 2022) have democratized image creation, enabled by large-seale image- text datasets,
e.g., L (Schuhmann et al., 2022), scraped from the internet. However, 3D data is not as eas- ily accessible, making 3D generation with 2D supervision very attractive. Previous works (Poole)
al. [022 Wang et al., 2022; Lin et al., 2022 Metzer et al. 2022 Chen et al. (023) have utilized pre-trained text-to-image diffusion models as a strong image prior to supervise 2D renderings of 3D models, with promising showcases for text-to-3D generation. However, the generated 3D mod- els are often in low quality with unrealistic appearance, mainly due to the fact that text-to-image models are not able to produce ddentity-consisted object across multiple generations, neither can provide camera pose-sensitive supervision required for optimizing a high-quality 3D) representation. To mitigate such supervision conflicts, later works orthogonal to ours have explored using generative models with novel view synthesis capability (Liu et al. 2023a b) adapting pre-trained model to be aware of camera pose and current generation (Wang et al. 2023). However, challenges remain for creative 3D content creation, the generation process still suffers from some combination of the following limitations as illustrated in Fig. [ (1) requiring a long optimization time to generate 3D object (slow convergence); (2) low generation quality such missing text-implied attributes and distorted shape and texture; (3) low generation diversity.
As class of score-based generative models (Ho et al., [2000] [ong & Ermon 2019 Song et al. 2021b), diffusion models contain a data noising and a data denoising process according to pre- defined schedule over fixed number of timesteps. They model the denoising score Vx log Pdata (x), which is the gradient of the log-density function with respect to the data on a large number of noise- perturbed data distributions. Each timestep (t) corresponds to a fixed noise with the score contain- ing coarse-to-fine to-fine information as t decreases. For image synthesis, the sampling process respects the discipline of coarse-to-fine content creation by iteratively refining samples with monotonically decreasing f. However, the recent works leveraging pre-trained data scores for diffusion-guide
3D) generation (Poole et al.) 2022, Wang et al., 2022 Lin et al.) 2022; Chen et al.) 2023 Qian et al., 2023) randomly sample during the process of 3D model optimization, which is counter-intuitive
In this paper, we first investigate what a 3D model learns from pre-trained diffusion models at each noise level. Our key intuition is that pre-trained diffusion models provide different levels of visual concepts for different noise levels. 3D 3D model initialization, it needs coarse high-level information for structure formation. Later optimization steps should instead focus refining details for better visual quality. These observations motivate us to propose time prioritized score distillation sampling (TP-SDS) for ceffusiionguidedd 31) generation, which aims to prioritize information from different diffusion timesteps (f) at different stages of 3D optimization. More concretely, we propose a non- increasing timestep sampling strategy: at the beginning of optimization, we prioritize the sampling of large t for guidance on global structure, and then gracefully decrease t with training iteration to get more information on visual details. To validate the effectiveness of the proposed TP-SDS we first analyze the score distillation process illustrated on 2D examples. We then evaluate TP-SDS against standard SDS on wide range of text-to-3D and image-to-3D generations in comparison for convergence speed as well as model quality and diversity.
Our main contributions are as follows:
We thoroughly reveal the conflict between diffusion-guider 3D optimization and uniform timestep sampling of score distillation sampling (SDS) from three perspectives: mathematical formulation supervision misalignment, and -ut-of--istrnbution (OOD) score estimation.
To resolve this conflict, we introduce DreamTime, an improved optimization strategy for diffusion-guided 3D content creation. Concretely, we propose to use non-increasing time sam- pling strategy instead of uniform time sampling. The introduced strategy is simple but effective by aligning the 3D optimization process with the sampling process of DDPM (Ho et al. 2020).
We conduct extensive experiments and show that our simple redesign of the optimization process significantly improves cnffiusion-guiden 3D generation with faster convergence, better quality and diversity across different foundation diffusion models and 3D representations.
2

---

Published as a conference paper at ICLR 2024
Random t-sampling in SDS: t~U(1,7)
Slow C Convergence
Misaligned Supervision
ut of Distribution
Droonassion: TTUU hours
conflicts with coarse-to fone generation nature
cannot handle low- uwwfrequency bias of early 3D renderings
Megae3D: 5.3 hours
Iaztssi33:: KTX houss
ProlinnDDemmin saveral 100 hours
Quality Concerns
Mode C Collapse
G Geometry
Texture
Semantics
incomplete geomeby
blurriness
color distortion
peacockona surthoami
chimpanzee dressed like Henry England"
Figure 2: Challenging phenomena in optimization- bassd diffusion-guide 3D generation.
RELATED WORK
Text-to-image generation. Text-to-image models such as GLIDE NNichol et al., 2021), ) un-
CLIP (Ramesh et al. 2022). Imagen (Saharia al. 2022), and Stable Diffusion (Rombach al. 2022 have demonstrated impressive capability of generating photorealistic and creative images given textual instructions. The remarkable progress is enabled by advances in modeling such as diffusion models (Dhariwal & Nichol 2021 Song et al.) 2021a Nichol & Dhariwal (021). well as large-scale web data curation exceeding billions of image-text pairs (Schuhmann al. [022] Sharma al. 2018: Changpinyo et al. 2021). Such datasets have wide coverage of general ob- jects, likely containing instances with great variety such as color, texture and camera viewpoints. As result, text-to-image diffusion models pre-trained those billions of image-text pairs exhibit remarkable understanding of general objects, good enough to synthesize them with high quality and diversity. Recently, generating different viewpoints of the same object has made significant progresses, notably novel view synthesis from a single image (Liu et al., 2023a|b) (), which can be applied to improve the quality of image-to-3D generation, orthogonal to approach.
Diffusion-guided 3D generation. The pioneering works of Dream Fields (Jain et al.) 2022) and CLIPmesh (Mohammad Khalid et al. 2022) utilize CLIP (Radford al., (021)) optimize 3D representation so that its 2D renderings align well with user-provided text prompt, without requir- ing expensive 3D training data. However, this approach tends produce less realistic 3D models because CLIP only offers discriminative supervision on high-level semantics. In contrast, recent studies (Poole al. 2022 Lin et al., 2022 Chen et al., 2023 Wang et al.) 2023] Qian et al. 2023 Tang et al. 2023) have demonstrated remarkable text-to-31 3D and image-to-3D generation results by employing powerful pre-trained diffusion models as a robust 2D prior. We build upon this line of work and improve over the design choice of 3D model optimization process to enable significantly higher-fidelity and higher-diversity diffusion-guided 3D generation with faster convergence.
METHOD
We first review diffusion-guided 3D generation modules, including differentiable 3D representa- tion (Mildenhall et al., [202]), diffusion models (Ho et al. 20200). and SDS (Poole et al. 2022) in Section 3.1 Then, we analyze the existing drawbacks of SDS in Section 3.2 Finally, alleviate the problems in SDS, we introduce an improved optimization strategy in Section
3.1 PRELIMINARY
Differentiable 3D representation. We aim at generating 3D model 0 that when rendered at any random view c, produces an image x= g( c) that is highly plausible as evaluated by a pre-trained text-to-image image-to-iman -o-imaae diffusion model. To be able to optimize such 3D model, require the 3D representation to be differentiable, such as NeRF (Mildenhall et al., 2021 Müller al. 2022). NeuS (Wang et al. 2021) and DMTet (Shen et al. 2021).

---

Published as a conference paper at ICLR 2024
Optimization based Sampling
Rendered Image
Timestep
Rendered Image x
1000
750
Timestop
500
250
Figure 3: Visualization of SDS gradients under different timesteps t. (Left) Visualization of SDS gradients throughout the 3D model (in this case NeRF) optimization process, where the green curved arrow denotes the path for more informative gradient directions NeRF optimization progresses. can be observed that non-increasing timestep is more suitable for the SDS optimization process. (Right) We provide more examples to illustrate the effects of timestep on SDS gradients. Small provides guidance on local details, while large is responsible for global structure.
Diffusion models (Ho et al. 2020, Nichol & Dhariwal 2021) estimate the denoising score Vx log Paam (x) by adding noise to clean data ~ p(x) in "imesteps with pre-defined schedule ag (0, 1) and == II 1 Ctg, according to:
x V -1- dne, where N),,,),,
then learns to denoise by minimizing the noise prediction error. In the sampling stage, one can derive x from noisy input and noise prediction, and subsequently the score of data distribution.
Score Distillation Sampling (SDS) (Poole et al. 2022 Lin et al. 2022 Metzer et al., 2022) is widely used method distill 2D image priors from a pre-traines trained diffusion model eø into differen- tiable 3D representation. SDS calculates the gradients of the model parameters 0 by:
0x
VoLsus x) = E.,, ((t)(cc(xx
(2)
00
where w(() a weighting function that depends on the timestep t and y denotes a given text or image prompt. SDS optimization is robust to the choice of w(t) as mentioned in Poole et al. 2022))
Remark. Our goal is to optimize a differentiable 3D representation by distilling knowledge from pre-trained Stable Diffusion (Rombach al.) 2022) Zerol23 (Liu al. (023a) given text or image prompt, respectively. In the training process, SDS is used to supervise the distillation process.
3.2 ANALYSIS OF EXISTING DRAWBACKS IN SDS
A diffusion model generates an image by sequentially denoising a noisy image, where the denois- ing signal provides different granularity of information at different timestep from structure to details (Choi et al. 20222] Balaji al. (022). For diffusion-guided 3D content generation, however, SDS (Poole et al. 2022) samples / from a uniform distribution throughout the 3D model optimiza- tion process, which counter-intuitive because the nature of 3D generation is closer DDPM sampling (sequential (-sampling) than DDPM training (uniform t-sampling). This motivates to explore the potential impact of uniform é-sampling on diffusion- guidedd 3D generation.
In this subsection, analyze the drawbacks of SDS from three perspectives: mathematical formu- lation, supervision alignment, and out-of-distribution (OOD) score estimation.
Mathematical formulation. We contrast SDS loss:
Sons(2, (-1((1
()) ell
(3)
4

---

Published as a conference paper at ICLR 2024
Low-frequency
Adding
Adding
Random Noise
Large Noise
Domain Misalignmenn
(a) NeRF Rendering Images under SDS Optimization
Domain Allgument
[=]
t-1
2-200
2-400
2-600
[-800
E-1000
t=1000
(c) Case Initial NeRF Small
(b) Intra-domain Distribution Diffusion Model's Training Data
(d) TP-SDS:
Initial NeRF Large
Figure 4: Illustration of OOD issue using web-data pre-trained diffusion model for denoising NeRF rendered images, in pixel and frequency domain. We provide an extreme case show the frequency domain misalignment: (c) adding small noise to NeRF's rendering at initialization. (d) illustrates that TP-SDS avoids such domain gap by choosing the right noise level.
NeRF Initaal
Seed=)
Seed=]
Seed=2
Seed-3
Normal Intial
Seed=6
Seedml
Seed=2
Seed-3
Figure 5: Low-frequency bias initial rendered image leads to low-diversity generation. To demon- strate this, we provide SDDSoptimized 2D generated results using text prompt "gingerbread man". Compared to normal initialization (Right), NeRF initialization (Left) exhibits low frequency and leads to the OOD issue, failing to produce diverse results.
with DDPM sampling process, i.e., for = +
- ar āi
cs cx v. orE,
Xt-
Xx
Var
VI
where (O, 0₁, is training noise schedule and o; is noise variance, e.g., vI ae..
Note that for SDS training, is randomly sampled as shown in Eqn. with red color, but for DDPM sampling, is strictly ordered for Eqn. as highlighted in blue. Since diffusion model is general denoiser best utilized by iteratively transforming noisy content to less noisy ones, we argue that random timestep sampling in the optimization process of 3D model is unaligned with the sampling process in DDPM.
Supervision Misalignment. For diffusion models, the denoising prediction provides different gran- ularity of information at different timestep t: from coarse structure to fine details as decreases. To demonstrate this, we visualize the update gradient [Xe, y, t) ell for 3D NeRF model renderings in Figure 3 From the left visualization, it is evident that as NeRF optimization progresses, the dif- fusion timestep that most informative NeRF update changes (as highlighted by curved arrow in Figure Bh. We provide more examples to reveal the same pattern on the right. Uniform t sampling supervision.
disregards the fact that different NeRF training stages require different granularity Such misalignment leads to ineffective and inaccurate supervision from SDS in the training process, leading to slower convergence and lower-quality generations (lack of fine details), respectively.
Out-of-distributiin (OOD). For pre-trained Co to provide well-informed score estimation, needs to close to the training data distribution which are noised natural images. However, at the early training stages, the renderings of NeRF are obviously out-of-distribution (OOD) to pre-trained Stable Diffuuson. Evident frequency difference exists between the rendered images and diffusion model's 's training data, shown in Figure 4 We further show in Figure 5 with 2D examples that the lack of high-frreuency signal at carly stage of content creation directly contributes to mode collapse flow-diversity models given the same prompt) observed in Poole et al. 2002
5

---

Published as a conference paper at ICLR 2024
3.3 TIME PRIORITIZED SCORE DISTILLATION
Drawbacks of uniform --sampling in vanilla SDS motivate us to sample i more effectively. Intu- itively, non-increasing i-sampling (marked by the curved arrow in Figure 3 is effective to the 3D optimization process, since it provides coarse-to- fine supervision signals which are more in- formative to the generation process, and initiates with large noise to avoid the OOD problem caused by low-frequency 3D renderings.
Based on this observation, we first try naive strategy that decreases t linearly with optimization iteration. However, it fails with severe artifacts in the final rendered image, as shown in App. Fig. 13 We observe that decreasing works well until later optimization stage when small dominates. We visualize the SDS gradients (lower-right box within each rendered image) and notice that at small I, variance of the SDS gradients are extremely high, which makes convergence difficult for 3D-
consistent representation. In fact, different denoising contributes differently (Choi et al. 2022) to content generation, so it is non-optimal to adopt a uniformly decreasing t. In result, we propose a weighted non-increasing t-sampling strategy for SDS.
Weighted non-increasing t-sampling aims to modulate the timestep descent process based on a given normalized weight function (t).. Specifically, (tt) represents the importance of diffusion timestep t and controls its decreasing speed. For example, a large value of W (t) corresponds to a flat decrease, while small corresponds to steep decline.
To sample the timestep ((i) of the current iteration step i, weighted non-increasing I-sampling can be easily implemented by:
t(1) = arg min
(()) -
(5)
where N represents the maximum number of iteration steps, and T denotes the maximum training timestep of the utilized diffusion model.
Prior weight function ((t) cannot be trivially derived. As shown in App. Fig. 13 a naively
designed constant W (t) (i.e., decreases linearly) causes training to diverge. To this end, we propose carefully designed weight function of the form (()) z W ()(()- W Wt))). where Wa(() and W (t) respectively takes into account the characteristics of the diffusion training and the 3D generation process, and = Σ Wa(t) (t) is the normalizing constant.
For Wa(t), we derive from an explicit form of the SDS loss:
[
LSDS (g) x) Er,
2 (1) || stop grad
(6)
where
to (z, vI o.co y. v)t) /Va.
is the estimated original image, and the SDS loss can be seen as a weighted image regression loss. For simplicity, we start our derivation for Wa(t) by setting w') 1 in Eqn. Then, by substi- tuting Eqn. and Eqn. to Eqn. 6 can reduce Eqn. to Eqn. with w(())
which
we naturally set as Wa((). Importantly, the term
0. is also the square root for reciprocal of ax
a diffusion model's signal-to-noiss ratio (SNR) (Lin et al. 2023). which characterizes the training process of a diffusion model.
For Wr(t), we notice that denoising with different timesteps (wer noise levels) focuses on the restoration of different visual concepts (Choi et al. 2022). When is large, the gradients provided by SDS concentrate on coarse geometry, while i small, the supervision on fine details which tends to induce high gradients variance. Thereby we intentionally assign less weights to such stages, to focus our model on the informative content generation stage when f is not in the extreme. For simplicity, we formulate (t(() to be simple Gaussian probability density function, namely,
(t) = e (t-=)² where m and are
controlling the relative importance of and is presented
the three stages illustrated in Fig. T (b). Analysis on the dyper-parameters in App. C The intuition behind our W,, formulation is that a large diffusion timestep t induces gradients that are of low-variance but rather coarse, lacking necessary information details,

---

Published as a conference paper at ICLR 2024
1.0
1000
Coarse
08
Diflusion Prior
(1))
Therweight W()
(())
0.6
modulate
Timestop 2(0
600
Content
Perception Prior (,)
620-
0.2
Detailed (High Variation)
1000
000
10k
Timesoop!
3D model uptimization step
(a) Prior weight function W(O.
Weighted non-increasing t-sampling function e(F).
Figure 6: The proposed prior weight function (()) to modulate the non-increasing (-sampling process ((i) for score distillation, as described in Eqn. 5 Weight functions W(I), Wa((), and W; (1)
are normalized to 0-1 for best visualization. Notice that Wa skews W to avoid small diffusion timesteps that induce high- varianer gradients. Such steps and their induced gradient variance are illustrated in (b) as the Detailed stage.
while small highly raises gradient variance that can be detrimental model training. Thus employ bell-shaped W so as to suppress model training extreme diffusion timesteps.
To sum up, normalized weight function ((t) is:
1
04
(t-m) m
W(1) 7
2x
(8)
0t
Time Prioritized SDS. To summarize, we present the algorithm for our Time Prioritized SDS (TP- SDS) in Alg. We also illustrate the functions Wa. Wp and W in F where should notice that Wa skews W to put less weight on the high-variance stage, where t is small. The modulated monotonically non-increasing timestep sampling function is illustrated in Fig. (b).
Discussion. The merits of our method include:
The prior weight function (t) assigns less weight those iteration steps with small t, thereby avoiding the training crash caused by high variance in these steps.
Instead of directly adjusting w(() from Eqn. to assign various weights to different iteration steps, which can hardly affect the 3D generation process (Poole et al. 2022). TP-SDS modifies the decreasing speed of in accordance with the sampling process diffusion models. This way, the updates from those informative and low-variance timesteps dominate the overall updates of TP-SDS, making our method more effective in shaping the 3D generation process.
4 EXPERIMENT
We conduct experiments on the generation of 2D images and 3D assets for comprehensive evalua- tion of the proposed time prioritized score distillation sampling (TP-SDS). For 2D experiments, the generator g (recall definitions in Sec. 3.1 an identity mapping while is an image representation. For 3D experiments, g is differentiable volume renderer that transforms 3D model parameters into images.
4.1 FASTER CONVERGENCE
We empirically find that the proposed non-increasing t-sampling strategy leads to a faster conver- gence, requiring ~ 75% fewer optimization steps than the uniform f-sampling. This is likely due more efficient utilization of information, e-g., wasteful seek structure information later stage of optimization when the 3D model is already in good shape. We conduct both qualitative and quantitative evaluation to demonstrate the fast convergence our TP-SDS.

---

Published as a conference paper at ICLR 2024
SDS
TP-SDS (Ours)
500 Iters.
1000 Iters.
2500 Ittrs.
5000 Iters.
7500 Iters.
10000 Iters.
Figure 7: Faster Convergence. Qualitative comparisons of the SDS baseline (first row in each exam- ple) and the proposed TP-SDS (second row in each example) under different iteration steps (from 500 to 10000). The proposed TP-SDS leads to faster content generation than the SDS baseline.
Qualitative comparison. Figure shows the 3D generation results with different max iteration steps, using vanilla SDS and our TP-SDS. is clear that with TP-SDS, the emer- gence of content (e.g., object struc- tures) is faster with better appearance and details.
0.7
0.6
R-Precisiinn
0.4
Quantitative evaluation. Given the 153 text prompts from object-centric COCO validation set, we show in Figure 8 the R-Precision scores of 2D generation results at different it- eration steps using the vanilla SDS and our TP-SDS. The growth rate of TP-SDS curves is consistently higher across various CLIP models, which implies a faster convergence requir- ing significantly fewer optimization steps to reach the same R-Precision score. This leads to the production of superior text-aligned generations at quicker pace with fewer resources.
0.3
TP-SDS.B/32
TP-SDS,B/16
1P-SDS.L/14
SDS,8/3
0.1
SDS,B/16
SDS.L/14
0.0
2500
5000
7500
10000
12500
15000
17500 20000
Steps
Figure 8: R-Precision curves of 2D generation results pro- duced by SDS and our TP-SDS, given 153 text prompts from object-centric COCO validation set. We use three CLIP models: B/32, B/16, and L/14, for R-Precision evalua- tion. The R-Precision for TP-SDS have a steeper growth rate compared to those of SDS, signifying faster convergence.
4.2 BETTER QUALITY
In this subsection, demonstrate the effectiveness of TP-SDS in improving visual quality. We argue that some challenging problems in text-to-3D generation, such as unsatisfactory geometry, degenerate texture, and failure to capture text semantics (attribute missing), can be effectively alle- viated by simply modifying the sampling strategy of timestep é.
Qualitative comparisons. We compare our method with the SDS (Poole et al. 2022) baseline, which is DeepFloyd (Saharia et al., accessible threestudio codebase [Guo al. 2023).
based DreamFusion implementation using publicly-
More accurate semanties. The contents highlighted in orange in Figure 9 shows that gener- ations align better with given texts and are void of the attribute missing problem. For example, SDS fails to generate the mantis' roller skates described by the text prompt TP-SDS does.
Better generation shape and details. In Figure 9 the contents highlighted in red and green respectively demonstrate that TP-SDS generates 3D assets with better geometry and details. noticeably alleviates shape distortion and compromised details. For example, in contrast to SDS, TP-SDS successfully generates a robot dinosaur with the desired geometry and textures.
Widely applicable. As shown in Fig. | the proposed TP-SDS is highly general and can be readily applicable to various 3D generation tasks, such as text-to-3D scene generation, text-to-3D avatar generation, etc., to further improve the generation quality.
8

---

Published as a conference paper at ICLR 2024
SDS
TP.SDS (Our)
robot dicosaur
an octopus playing the harp
praying mantis wearing roller skzttes
tiger dressed maid
an expensive office chair
Ao octopusjand giratle häving cheesecalse
squirrel giving lecture writing on a chalkboard
Figure 9: Qualitative comparisons between the original SDS (Poole et al. 2022) (upper row) and TP- SDS (lower row). Our method alleviates the problems attribute missing, unsatisfactory geometry, and compromised object details at the same time, as highlighted by the colored circles.
"a baby bunny sitting of stack of pancakes"
"an ice cream sundac"
SDS
TR-DS (Ours)
S
Sood-1
Sood-2
Sead=3
Soad=0
Sood=1
Sood-2
Soud=3
Figure 10: Higher Diversity. We compare the diversity of text-to-3D 3D generations between TP-SDS and SDS. The text prompts provided in the figure. Given different random seeds, TP-SDS is able to generate visually distinct objects, while the results produced by DreamFusion all look alike.
4.3 HIGHER DIVERSITY
A key ingredient of AIGC is diversity. In 2D scenarios, given text prompt, Stable Diffusion is able generate a countless number of diverse samples while respecting the given prompt. However, the success has not yet been transplanted to its 3D counterpart. Current text-to-3D generation ap- proaches reportedly suffer from the mode collapse problem (Poole al., 2022). i.e,, highly similar 3D assets are always yielded for the same text prompt whatever the random seed is. In Figure 5 we illustrate with 2D generations that the problem of mode collapse is largely caused by the low- frequency nature of initial NeRF renderings, and that the proposed TP-SDS is able to circumvent it by applying large noise (i.e., with a large t) during the early training process. Figure I00 further demonstrates that the 3D generations produced by TP-SDS are much more diverse visually than those from DreamFusion (Poole et al. 2022
5 CONCLUSION
Conclusion. We propose Dream'Tmme, an improved optimization strategy for diffusion-guided con- tent generation. We thoroughly investigate how the 31) formation process utilizes supervision from pre-trained diffusion models at different noise levels and analyze the drawbacks of commonly used score distillation sampling (SDS). We then propose non-increasing time sampling strategy (TP- SDS) which effectively aligns the training process of a 3D model parameterized by differentiable 3D representations, and the sampling process of DDPM. With extensive qualitative comparisons and quantitative evaluations we show that TP-SDS significantly improves the convergence speed, qual- ity and diversity of ceffuuiionguideen 3D generation, and considerably more preferable compared to accessible 3D generators across different design choices such as foundation generative models and 3D representations. We hope that with Dream Time, 3D content creation can be more accessible for creativity and aesthetics expression.
Social Impact. Social impact follows prior 3D generative works such as DreamFusion. Due to our utilization of Stable Diffusion (SD) as the 2D generative prior, TP-SDS could potentially inherit the social biases inherent the training data of SD. However, our model can also advance 3D-related industries such as 3D games and virtual reality.

---

Published as a conference paper at ICLR 2024
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Ait- tala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers. arXiv preprint arXiv:2211.01324, 2022.
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it SMPL.: Automatic estimation of 3D human pose and shape from single im- age. In Computer Vision-FCCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14. pp. 561-578. Springer, 2016.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing Web- Scale Image- Text Pre-Traie Traininin To Recognize Long-Tail Visual Concepts. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation. arXiv preprint arXiv:2303. 13873, 2023.
Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception Prioritized Training of Diffusion Models. In Proceedings of the IEEE/CVF Conference Computer Vision and Patter Recognition, pp. 11472-1148) 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs on Image Synthesis. Ad- vances in Neural Information Processing Systems, 34:8780-8794, 2021.
Yuan-Chen Guo, Ying-Tian Liu, Chen Wang, Zi-Xin Zou, Guan Luo, Chia-Hao Chen, Yan-Pei Cao. and Song- Hai Zhang. threestudio: A unified framework for 3d content generation. https: 2023.
//qithub.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Yukun Huang, Jianan Wang, Ailing Zeng, He Cao, Xianbiao Qi, Yukai Shi, Zheng-Jun Zha, and Lei Zhang. DreamWaltze Make a Scene with Complex 3D Animatable Avatars. arXiv preprint M:Xiv:2305.12529 2023.
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-Shot Text- Guided Object Generation With Dream Fields. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 867-876, 2022.
Ruixiang Jiang, Can Wang, Jingbo Zhang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control. arXiv preprint arXiv:2303. /7606, 2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng, Xun Huang. Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung- Yi Lin. Magic3D: High-Resolution Text-to-3D Content Creation. arXiv preprint arXiv:2211.
2022.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common Diffusion Noise Schedules and Sample Steps are Flawed. arXiv preprint arXiv:2305. 08891, 2023.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; Zero-shot One Image to 3D Object. arXiv preprint arXiv:2303.11322 2023a.
Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping
Wang. SyneDreamer: Generating Multiview-consistenl Images from Single-view Image. arXiv preprint arXiv:2309.03453. 2023b.
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures. arXiv preprint arXiv:2211.07600. 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Communi- cations of the ACM, 65(1):99-106 2021.
10

---

Published as a conference paper at ICLR 2024
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. CLIP-Mesh: Gen- erating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-8, 2022.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding, ACM Transactions on Graphics (ToG), 41(4): 1-15, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE; Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. arXiv preprint arXiv:2112.107441 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved Denoising Diffusion Probabilistic Mod- els. International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D Diffusion. arXiv preprint arXiv:2209.14988, 2022.
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, al. Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors arXiv preprint arXiv:2306. 10006.17844 17843, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. In International Conference Machine Learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:204.066/25, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Bureu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Pho- torealistic Imxae Diffusion Models with Deep Language Understanding. arXiv preprint arXiv:2205.114877 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402. 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-2565, 2018.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087-6101, 2021.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In In- fernational Conference on Learning Representations, 2021a. URL /tt/tttts///opprrrrier net thet/fozue
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribu- tion. Advances in Neural Information Processing Systems, 32, 2019.

---

Published as a conference paper at ICLR 2024
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- Based Generative Modeling through Stochastic Differential Equations. Interna- tional Conference Learning Representations, 2021b. URL httpss//epenreevew netteeeee/weeee/ee/eeeee
2RRH5
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make- it-3d: High-fidelity 3d creation from a single image with diffusion prior, 2023.
Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. TextMesh: Generation of Realistic 3D Meshes From Text Prompts. arXiv preprint arXiv:2304. /2439, 2023.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich. Score Ja- cobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation. arXiv preprint arXiv:22/2.00774, 2022.
Peng Wang. Lingjee Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2 106. 106,999,, 2021.
Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- Dreamer: High-Fidelity and Diverse Text-to3D Generation with Variational Score Distillation. arXiv preprint arXiv:2305.16213 2023.
Joseph Zhu and Peiye Zhuang. HiFA: High-fidelity Text-to-3D with Advanced Diffusion Guidance. arXiv preprint arX16 2023.
12

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
Our implementation are based on the publicly-accessiee accessible threestudio codebase (Guo et al. 2023), and only the timestep sampling strategy is modified. We provide the algorithm flow of TP-SDS as shown in Alg.
Algorithm 1: Time Prioritized SDS (TP-SDS).
Input: A differentiable generator g with initial parameters On and number of iteration steps N, pre-trained diffusion model é, prior weight function W(t), learning rate Ir, and text prompt for i 1,.., N do
((i) arg min DE
#
0, 0 Ir VeLsos (. t(1);;
end
Output: ON.
B ABLATION STUDY
In this section we evaluate the 3D generation ability of TP-SDS by formulating W(() various functions, to investigate the effects of the prior weight function (recall that T denotes the maximum training timestep of the utilized diffusion model):
Baseline: We provide the results produced by the random timestep schedule for reference.
Linear: ((I)= 7.. which induces a linearly decaying time schedule.
Truncated linear: (1) #. but we constrain ((i) to be at least 200, instead of 1. This is for assessing the influence of gradient variance induced by small
W, only: W(t) (t)...
W, only: W(t) Wa(t).
TP-SDS: (tt) = Wa(t) (t).).
The results shown in Fig. Evidently, naive designs such baseline, linear and linear (Irun- cated) fail to generate decent details, where valuable observation is that linear (truncated) gen- erates finer details than that of linear, validating our claim that small hinges 3D generation with high gradient variance. Moreover, applying W or Wa alone cannot produce satisfying 3D results, suffering either over-saturation or loss of geometrical details. comparison, our proposed prior weight function circumvents all the deficiencies and generates delicate 3D.
Random (Baseline)
Linear
Linear (truncated)
Ours (W, only)
Ours (Wa only)
Ours
Figure 11: Ablation study. We investigate the effects of the prior weight function by modifying its formulation and comparing the text-to-3D generation results. The text is "Chichen Itza, aerial view".
13

---

Published as a conference paper at ICLR 2024
C HYPER-PARAMETER ANALYSIS
TP-SDS improves generation efficiency, quality, and diversity compared to the SDS baseline. How- ever, the proposed prior weight function (t) parameterized by {m, introduces extra hyper- parameters. Thus we explore the influence of these hyper-parane parameters on text-to-3D generation, which can serve as a guide for tuning in practice. In Figure [2] we illustrate the impact of hyper- parameters the generated results in large search space. Specifically, increases the number of optimization steps spent on both the coarse and the high-variance detailed stages, degrading the generation quality. controls the model to concentrate different diffusion timesteps, and rule of thumb is to make m close to T 2.
large significantly
200
300
= 400
500
600
700
800
10
50
100
125
167
250
500
Figure 12: Influence the time prior configuration {m, 8} on generations. The text prompt "a DSLR photo of green monster truck". is set to 125 for the experiments on the first row while m 500 for the rest.
D TIMESTEP ANALYSIS
I+1K, 2-000
I=2K, c-800
[+3K, 7-700
[=4K, 2-600
I+5K, E-500
[+6K, 1=400
1+7K, 1+300
1-6K,
(+9K, 2-100
(=10K,
Figure 13: Visualization of NeRF optimization with SDS using naive t-sampling strategy. where f decreases linearly with the iteration step i. Severe artifacts appear the final rendered image due to large gradients variance with small
1000 (fixed)
700 (fixed)
t500 (fixed)
t00 (fixed)
t=100 (fixed)
Figure 14: Illustration of information capacity for different diffusion timesteps. Here we train 3D assets supervised by SDS with the diffusion timestep fixed throughout. Notably, for small i, e.g. 100, the gradient variance becomes too high for the model to generate 3D successfully, while a large like 900 makes the generation lack local details. Only those not in the extreme can prompt SDS to produce decent 3D generations, for which believe that such timesteps most informative and consequently make our model training concentrate on the "content" stage consisting of such timesteps. The text prompt is "a DSLR photo of baby bunny sitting pile of pancakes".
14

---

Published as a conference paper at ICLR 2024
E
QUANTITATIVE EVALUATION ON DIFFERENT METHODS
93.1
84.3
76.1
23.6
15.7
10
DreamFusion
TextMesh
Fantasia3D
Magic3D
Zero-1-to-3
Only)
SDS (Baseline)
TP-SDS (Ours)
Figure 15: User preference studies of SDS and our TP-SDS on five popular 3D generation methods:
DrcamFusion (Poole et al., 2022). TextMesh (Tallcogloou et al. 2023). Fantasia3D (Chen et al., 2023). Magic3I (Lin et al., 2022), and Zero-1-to- (Liu al. 2023a The implementation of these methods follows threestudio (Guo et al.] (023). We use 415, 100, 100, 100, and 50 prompts as evaluation sets respectively for the above hive methods, each evaluated by participants. Our
proposed TP-SDS consistently achieves higher preference scores (%).
F COMPARISONS WITH TIMESTEP SCHEDULES PROPOSED IN HIFA AND PROLIECDRRAMEER
p 0.25
p 0.5 HiFA
p = 1.0 Linear
p 2.0
p 3.0
Ours
(a) Comparison of our method with the timestep schedule proposed in HiFA (Zhu & Zhuang 2023). which adopts exponential f-annealing strategy with power 0.5. The results of HiPA variants with different powers (denoted p) are also provided for comprehensive comparison.
1-Stage Random
2- 2-Stage PromificDreamer
3- Stage
4 Stage
5-Stage
0 urs
(b) Comparison of our method with the timestep schedule proposed in ProlificDreamer (Wang et al. 2023
which adopts two-stage random t-sampling strategy. The results of ProlifiDrreamer variants with different number of stages are also provided for comprehensive comparison.
Figure 16: Comparisons of our method with the timestep schedules proposed in HiFA Zhu & Zhuang 2023) and PromificDreamer (Wang al. 2023)
15

---

Published as a conference paper at ICLR 2024
G IMPLEMENTATION DETAILS AND MORE RESULTS OF TP-VSD
To produce the VSD-based results presented in Figure and Figure we follow the Prolific- Dreamer (Wang et al. (023) implementation by threestudio (Guo et al. 2023) and modify its timestep schedule. Specifically, we adopt the random uniform I-sampling proposed timestep schedule for TP-VSD, We emphasize that we adopt the same hyper-parameter configuration {m 500, s = } for both TP-SDS and TP-VSD throughout this paper. Further tuning of syper-parameters might yield better results.
VSD, and use our
VSD "a DSLR photo of a hamburger inside restaurant"
TRVSD
VSD
TP-VSD
"a DSLR photo of table with dim sum it"
VSD
TP-SS
VSD
TP-VSD
"a DSLR photo of an ice cream sundae inside shopping mall"
"a Matte painting of a castle made of cheesecake surrounded by moat made of ice cream"
"bathroom, rustic style, realistic, 4k (Top: VSD, Bottom: TP-VSD)
Figure 17: More examples VSD WWang et al. 2023 and our proposed TP-VSD. Our method is
better at generating details and improving clarity, highlighted with red and green bounding boxes.
16

---

Published as a conference paper at ICLR 2024
H EFFECTIVENESS ON DIFFERENT 3D REPRESENTATIONS
Our proposed t-annealing schedule is improvement the diffusion-guided optimization process and therefore is universally applicable to various 3D representations, such as NeRF (Mildenhall al. 2021), NeuS (Wang et al. 2021), and DMTet (Shen et al. 202T).
Both quantitative and qualitative evaluations demonstrate the effectiveness our method on differ- ent 3D representations. For quantitative evaluations in Figure [5] DreamFusion and dero-1-to-3 NeRF-based methods, TextMesh is a BeuS-based method, Fantasia3D
DMTet- based method, Magic3D is hybrid (NeRF + DMTet) method. For qualitative evaluations, we provide NeRF-based (the
results in Figure 7. Figure
Figure T0] Figure 17 and Figure[21 (the first to third rows), JeuS-based
results in Figure
and Figure 22
DMTet-baser results Figure and Figure 21 (the fourth row).
SDS (Baseline) "a DSLR photo of baby dragon hatching out of a stone egg"
TP-SDS (Ours)
SDS (Baseline)
TP-SDS (Ours)
"a brightly colored mushroom growing on log"
Figure 18: Qualitative comparison of SDS and TP-SDS based on threestudio (Guo et 2023
implementation of Fantasia3D (Chen et al. 2023).
17

---

Published as a conference paper at ICLR 2024
SDS Salnan
TP-DD (Ours)
"a DSLR photo of a cake covered in colorful frosting with slice being taken out, high resolution"
"a crocodile playing drum set."
"a DSLR photo of baby dragon drinking boba"
ff
SDS (Basdine)
TP-SDS (Ours)
"a confused beagle sitting desk working on homework."
DSLR photo of magician making a white dove appear"
"a cute steampunk elephant"
Figure 19: Qualitative comparison of SDS and TP-SDS based on the threestudio (uo et al. 2023
implementation of TextMesh Tsalicoglou et al. 2023).
18

---

Published as a conference paper at ICLR 2024
I VISUALIZATIONS OF SDS-BASED OPTIMIZATION ON NEUS AND WITH INITIALIZED NERF
As complement to Figure 3 we further provide two cases: (a) NeuS Wang et al. (2021 optimized
by DeepFloyd-IF and (b) SMPL-initialized NeRF Mildenhall et al. (2021) optimized by Stable- Diffusion, and visualized the SDS gradients under different timesteps, as shown in Figure 20
The supervision misalignment issue described in Section 3.2 can also be observed in Figure 20 indicating that using a decreasing timestep schedule remains a good principle. Additionally, for well-initialized NeRF, optimization with large timesteps should be avoided.
Optimization Process under SDS Guidance
Optimization Process under SDS Guidance
Rendered Imagex
1000
750
Thresep
500
250
(a) NeuS optimized by DeepFloyd-1F.
(b) Initialized NeRF optimized by Stable Stable-Diffusion Diffusion.
Figure 20: Visualization of SDS gradients under different timesteps t. Two cases: (a) NeuS opti- mized by DeepFloyd-IF and (b) initialized NeRF optimized by Daibee-iiffuuio, are provided. In particular, initialized NeRF is pre-trained the images rendered from SMPL [Bogo al. (2016) mesh template. The supervision misalignment issue described in Section 332 can also be observed. We use an imaginary green curved arrow denote the path for more informative gradient directions as optimization progresses under SDS guidance.
19

---

Published as a conference paper at ICLR 2024
J EXAMPLES OF ALLEVIATING BLURRINESS AND COLOR DISTORTION
We provide more examples to show that the proposed timestep schedule can alleviate the issues of blurriness and color distortion, as shown Figure
O
SDS (Baseline)
TP-SDS (Ours)
SDS (Baseline)
TP-SDS (Ours)
"a DSLR photo of kitten standing on top of giant tortoise"
"a DSLR photo of a coryi standing up drinking boba"
SDS (Baseline)
TP-SDS (Ours)
SDS (Baseline)
TP-SDS (Ours)
"a beautiful dress made out of fruit, on mannequin. Studio lighting. high quality, high resolution"
"a DSLR photo of pair of tan cowboy boots, studio
lighting, product photography uniform"
SDS (Baseline) zoomed out DSLR photo of ladybug"
TP-SDS (Ours)
SDS (Baseline)
TPSDS (Ours)
"a DSLR photo baby elephant jumping on trampoline"
SDS (Baseline)
TP-SDS (Ours)
SDS (Baseline)
TP-SDS (Ours)
"a bcautiful rainbow fish"
"a beautiful dress made out of garbage bags, on mannequin, studio lighting, high quality, high resolution"
Examples to demonstrate that our method can alleviate blurness and color distortion issues, as highlighted with blue and green bounding boxes, respectively. The first three rows are the results of DreamFusion (Poole et al. 2022), and the fourth row is the results of Magic3D (Lin al. 2022), implemented by threestudio (Guo et al. 2023
Figure 21:
20

---

Published as a conference paper at ICLR 2024
K EFFECTIVENESS ON TEXT-TO-AVATAR GENERATION
In figure I show that our proposed TP-SDS facilitates ControlNet- text-to-avatar genera- tion work Dream Waltz (Huang et al. 2023), avoiding texture and geometry loss. Here we further demonstrate that another popular cext-to-avatee work, AvatarCraft (Jaag al. 2023). can also ben- efit from our method, achieving more detailed textures and avoiding color over-saturation, as shown in Figure 22
"Captain America"
"Princess Elsa"
SDS (Baseline)
TP-SDS (Ours)
Figure 22: Qualitative comparison of SDS and our TP-SDS based on the popular text-to-3D-avatar
generation work, AvatarCraft (Jiang et al. 2023). Compared to the SDS baseline, our method can lead to more detailed faces and avoid color over-saturation.
21


Reviews:
[{"review_id": "v3HdKFVYYi", "reviewer": "Reviewer_DnQ9", "summary": "The paper systematically studies the effect of the time schedule used for the diffusion model used in score distillation sampling for 3D generation. The key observation is that at different time steps, the denoiser focuses on different types of content -- high-frequency vs. low-frequency. It proposes a schedule to leverage this observation in order to improve the optimization quality and efficiency.", "questions": "The Score Jacobian Chaining paper has studied the scheduling of diffusion timesteps to an extent. Due to its high relevance, I think the paper should discuss the related findings in that paper and put things in context.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I really ike the idea proposed in the paper because it’s simple yet effective in improving the quality of 3D generation from text and images. It studies an aspects of using diffusion model for 3D diffusion model -- how to schedule the timestep used to condition the diffusion model, which is not systematically explored in prior literature. In the paper, the authors show both quantitative and qualitative experiments as well as detailed analysis to demonstrate the effectiveness of this simple technique. The authors also show the generality of this technique by showing results across many state-of-the-art 3D generation methods so it can be easily adopted by the community in any future work along the line.", "weaknesses": "1. Though I really like the idea, the paper studies a small aspect of 3D generation which can be seen as a trick to improve results. If more studies can be conducted such as using I to accelerate 2D image generation, the scope of the paper will be further expanded. But I don't think the authors need to conduct these experiments in the rebuttal.\n2. The quantitative experiments presented in the paper are limited. While the authors demonstrated the effectiveness of the proposed technique on many different methods with qualitative examples, quantitative evaluation will make these results more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kD3P4tedtm", "reviewer": "Reviewer_zkfR", "summary": "The authors propose a method for image-to-3D generation. Specifically, they suggest incorporating depth images during DDIM inversion and sampling to generate view-consistent novel view images. The enhanced novel view images are then used to compute a variant version of SDS loss, i.e., the RGSD loss, to optimize the 3D representation. The results appear to outperform the baselined methods used.\n\nThe authors proposed a timestep annealing scheme in SDS to enhance text-to-3D generation. Specifically, they pointed out the suboptimality of the random timestep sampling in Dreamfusion and suggested a non-increasing t-sampling approach using a Gaussian weight function W(t) in SDS. The proposed timestep annealing scheme proves to be effective and outperforms the baselines.", "questions": "I am curious if there exists a concrete theoretical deduction for the timestep annealing scheme that could explain which specific timestep annealing scheme should be employed.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and presents its ideas clearly.\n\nThe effectiveness of the proposed timestep annealing scheme in text-to-3D generation using SDS is demonstrated.", "weaknesses": "1. There are several existing works that have proposed timestep schedulings, such as two-stage samplings in ProlificDreamer [1] and the non-increasing timestep annealing scheme in HIFA [2]. These methods have been available on arXiv for a while. Given the rapidly evolving nature of this field, even if not officially published, it would be beneficial for the authors to compare the proposed Gaussian PDF with these existing approaches.\n2. The enhancement of TP-VSD is intriguing. However, the authors have presented only one example in Fig.1. It is recommended to provide additional visual results, especially for a wider range of objects and scenes as demonstrated in ProlificDreamer.\n3. While the proposed improvement is a valuable technique, its contribution as an ICLR paper may be limited.\n\n[1] ****ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, Wang et al., NeurIPS 2023. ****\n\n[2] ****HiFA: High-fidelity Text-to-3D with Advanced Diffusion Guidance, Zhu et al., 2023.****", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Urn0tZ82hw", "reviewer": "Reviewer_5Rpq", "summary": "Recent advancements in text-to-image diffusion models, trained on vast amounts of image-text pairs, have paved the way for 3D content creation by optimizing an initially random differentiable 3D representation using score distillation. Despite its promise, the method has substantial drawbacks: a prolonged optimization convergence and subpar 3D model outcomes in terms of both quality (e.g., distorted features and textures) and diversity, especially when juxtaposed with text-guided image synthesis. This paper pinpoints the crux of the problem in the misalignment between the 3D optimization and the uniform timestep sampling in score distillation. A novel approach is introduced that reprioritizes timestep sampling using monotonically non-increasing functions, seamlessly integrating the 3D optimization and diffusion model sampling. Comprehensive experiments validate the efficacy of this redesign, showcasing marked improvements in convergence speed, model quality, and diversity.", "questions": "In Fig.3, might it be beneficial to switch out NeRF for SDS-based implicit fields such as NeuS or VolSDF? Based on my experiments using CLIP guidance, each representation yields distinct visualizations as optimization unfolds. It raises the question: would these representations evolve similarly under SDS guidance? Occasionally, there might be a scenario where we initialize the neural implicit fields with a rudimentary object. How would the training process evolve in such instances?\n\nThe paper posits its objective as refining a differentiable 3D representation by leveraging knowledge distilled from either the pre-trained Stable Diffusion or Zero123, depending on whether the prompt is text or image-based. Could you clarify which outcomes are derived from the SDS loss in tandem with the pre-trained Stable Diffusion and which with Zero123? Additionally, I seem unable to locate results corresponding to image inputs.\n\nWhile Fig.2 highlights issues of blurriness and color distortion associated with the SDS loss in the context of texture generation, it appears that this paper falls short of presenting compelling evidence to demonstrate that the introduced method effectively addresses both challenges.\n\nMissing citations for AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control, which produces human avatars with SDS guidance. I'm intrigued to know if the methodology presented can be seamlessly adapted for avatar creation.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "While the paper's methodology is straightforward, its experiments are illuminating. It provides a visualization of NeRF's training steps guided by the SDS loss, uncovering a refined pathway to mitigate the training diversity and challenges associated with SDS loss. Furthermore, the study introduces a weighted function to fine-tune the significance of the t-sampling process for SDS. This not only accelerates convergence but also yields results with enhanced diversity. Moreover, this approach promotes improved semantic alignment.", "weaknesses": "The quality of the proposed method doesn't appear to significantly surpass that of Magic3D, Dreambooth3d[1], or Fantasia3D[2].\n\n[1] Raj A, Kaza S, Poole B, et al. Dreambooth3d: Subject-driven text-to-3d generation[J]. arXiv preprint arXiv:2303.13508, 2023. [2] Chen R, Chen Y, Jiao N, et al. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation[J]. arXiv preprint arXiv:2303.13873, 2023.\n\nThe method introduced seems to be a versatile technique applicable to various 3D representations. Comprehensive evaluations on diverse representations such as NeRF, NeuS, and DMTet should be undertaken.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LN72tIjZua", "reviewer": "Reviewer_iLxc", "summary": "This paper proposes to use a monotonically non-increasing time schedule in the score distillation training for 3D generation. The paper identifies the different denoising effects of different time steps, e.g. larger t helps generate coarse geometry and smaller t helps generate fine texture details. The proposed method shows much faster convergence speed than its random time schedule counterpart, and generates results with more geometric and textural details.", "questions": "1. Do you have any idea on how to design an adaptive schedule based on the information of the current instance instead of a fixed schedule?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The method is well motivated. The original SDS loss is derived from the training loss of diffusion models and thus adopts a random time schedule. But during score distillation for 3D generation, the process is more like the inference procedure of diffusion models. The optimized 3D representation goes from coarse to fine, so the time schedule should also be consistent with this.\n\n2. The presentation is great. The paper motivates the method with an easy-to-understand analysis (Section 3.2) and good illustrations (Figure 3, Figure 6).\n\n3. The proposed method is very easy to implement and yields no additional cost, but still achieves impressive results (Figure 8 and Figure 9).", "weaknesses": "1. The proposed time schedule seems to be highly heuristic. I do not think the optimal schedule would be the same for all different prompts. If the schedule is designed to be adaptive to the renderings of the current iteration it would be more robust.\n\n2. In Figure 1 the authors also show the results for \"TP-VSD\", but I cannot find any more explanation for it. Maybe it is obvious to apply the method on VSD, but I still think a small dedicated piece of text explaining it would be good. For example, do you use the same schedule for both SDS and VSD?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)