HIFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance
Authors: Junzhe Zhu
Peiye Zhuang
Sanmi Koyejo
Venue: ICLR 2024 poster (2024)
Keywords: Text-to-3D
Generative models
Diffusion models
Primary area: generative models

Abstract: The advancements in automatic text-to-3D generation have been remarkable. Most existing methods use pre-trained text-to-image diffusion models to optimize 3D representations like Neural Radiance Fields (NeRFs) via latent-space denoising score matching. Yet, these methods often result in artifacts and inconsistencies across different views due to their suboptimal optimization approaches and limited understanding of 3D geometry. Moreover, the inherent constraints of NeRFs in rendering crisp geometry and stable textures usually lead to a two-stage optimization to attain high-resolution details. This work proposes holistic sampling and smoothing approaches to achieve high-quality text-to-3D generation, all in a single-stage optimization. We compute denoising scores in the text-to-image diffusion model's latent and image spaces. Instead of randomly sampling timesteps (also referred to as noise levels in denoising score matching), we introduce a novel timestep annealing approach that progressively reduces the sampled timestep throughout optimization. To generate high-quality renderings in a single-stage optimization, we propose regularization for the variance of z-coordinates along NeRF rays. To address texture flickering issues in NeRFs, we introduce a kernel smoothing technique that refines importance sampling weights coarse-to-fine, ensuring accurate and thorough sampling in high-density regions. Extensive experiments demonstrate the superiority of our method over previous approaches, enabling the generation of highly detailed and view-consistent 3D assets through a single-stage training process.

Published as a conference paper at ICLR 2024
HIFA: HIGH-FIDELITY TEXT-TO-3D GENERATION WITH ADVANCED DIFFUSION GUIDANCE
Junzhe Zhu"l, Peiye Zhuang 1,2, Sanmi Koyejo'
Stanford University.
SSnap Inc.
(josefzhu, peiye, sanmi)estanford..o htt.://eeeerrrrrreeeeeeeeaaaeaee "pshuang@sapccaat.com
ABSTRACT
The advancements in automatic text-to-3D generation have been remarkable. Most existing methods use pre-trained text-to-image diffusion models to opti- mize representations like Neural Radiance Fields (NeRFs) via latent-space denoising score matching. Yet, these methods often result in artifacts and incon- sistencies across different views due to their suboptimal optimization approaches and limited understanding of 3D geometry. Moreover, the inherent constraints of NeRFs in rendering crisp geometry and stable textures usually lead to a two-stage optimization to attain nigh-resolution details. This work proposes holistic sam- pling and smoothing approaches to achieve qualituatty text-to-3D generation, all in single-stage optimization. We compute denoising scores in the text-to- image diffusion model's latent and image spaces. Instead of randomly sampling timesteps (also referred as noise levels in denoising score matching), we intro- duce a novel timestep annealing approach that progressively reduces the sampled timestep throughout optimization. To generate high-quality renderings in single- stage optimization, we propose regularization for the variance of z-coordinates along NeRF rays. To address texture flickering issues in NeRFs, we introduce a kernel smoothing technique that refines importance sampling weights coarse-to- fine, ensuring accurate and thorough sampling in high-density regions. Extensive experiments demonstrate the superiority of our method over previous approaches, enabling the generation of highly detailed and view-consistent 3D assets through single-stage training process.
INTRODUCTION
The task of automatic text-to-3D generation aims to create 3D assets based on a text description and has gained significant attention due to its wide-ranging applications in digital content genera- tion, film-making, and Virtual Reality (VR) (Lin et al. 2023; Chen et al., 20236). Initial efforts in this domain centered unconditional 3D asset generation, experimenting with various 3D repre-
sentation modalities presented in explicit formats such as meshes (Achlioptas et al.) 2018 Luo & well implicit formats such fields (Chen & Zhang 2019 Mittal et al. 2022 Zhuang et al. 2023). Following this, the field has progressed towards conditional 3D generative models, c.g., with text-based guidance (Cheng al. (023). How- ever, these studies have been limited to relatively simple 3D assets, primarily due to the scarcity of large-scale annotated 3D datasets.
Hu 2021 Smith & Meger 2017 Xie et al., 2018),
The availability of ample image datasets and the success of text-to-image generation have paved the way for lifting pre-trained eexttto-image models to the 3D domain. Specifically, recent studies focus on optimizing a 3D representation for an asset, using pre- tre-trained text-to-imas -moogmmaeeeeea generative models by providing denoising score for rendered images (Khalid et al. 2022) Jain et al. 2022 Poole al.)
2022 Xu et al.] 2022, Wang et al. 2023a Lin et al., 2023 Tang et al. 2023 Chen et al.) 2023b Wang et al. 2023b). Poole et al. (2022) proposed a loss from the distillation of a text-to-image diffusion model. They minimized the Kullback-Lebller (KL) divergence between family of Gaus- sian distributions based on the forward diffusion process and the denoising scores acquired from the pre-trained text-to-image diffusion model. The proposed Score Distillation Sampling (SDS) method combined with a NeRF enables 3D asset generation from given text prompts. Subsequent
"qqual contribution

---

Published as a conference paper at ICLR 2024
"A sea turtle"
"A tarantula, highly detailed"
"A parrot"
pair of pink fluffy slippers"
"A wooden buddha head"
"Robotic bee, high detail"
"A bagel filled with cream cheese and lox"
"Dragon wings and unicorn head hybrid creature"
"AA car made out of sushi"
Figure 1: Examples of multiple views of 3D objects generated by from our model given text prompts (below each object).
research has improved generation quality through various approaches including the adoption of two- stage optimization frameworks (Lin et al. 2023 Tang et al. 2023: Wang et al., 2223b Chen et al.)
2023ch, alterations the original SDS formulation (Wang et al. 2023a b). and the disentanglement
of geometry and appearance (Chen et al.) 2023b).
In this work, we revisit the integration of the SDS approach with NeRFs, aiming to achieve photo- realistic and high-quality text-to-3D generation through single-stage optimization. In contrast to existing text-to-3D generation work, we distill the score in the text-to-image diffusion model's la- tent and image spaces for enhanced supervision. Moreover, we observe that the efficacy of the diffusion prior is limited in previous works (Poole et al. 2022 Lin et al., 2023) when timesteps (also referred to as noise levels in denoising score matching) mization. Specifically, we observe that toward the end of the training process, the NeRF becomes almost determined in representing a particular 3D asset. Thus, we find that randomly sampling large timestep drives the diffusion model to produce a denoised image that is distinct and unrelated the original input rendering. This yields inconsistent distillation from the diffusion model and compromised optimization of NeRFs. To address this, we introduce a timestep annealing approach where the timestep in the forward diffusion process inversely correlates with the square root of the number of training iterations. Our empirical analysis demonstrates that the proposed timestep an- nealing approach effectively enhances generation quality. We also show that the square root timestep annealing consistently outperforms other annealing methods, such as linear and cosine ratios.
randomly sampled during opti-
Moreover, generating a detailed 3D asset through single-stage optimization is challenging. Specif- ically, explicit 3D representations, such as meshes, struggle to capture intricate topology, such as those with holes. Implicit 3D representations (Mildenhall et al., 2020; Müller et al. (002)) may lead to cloudy geometry and flickering textures. For instance, when NeRFs are employed to represent highly detailed 3D geometries like human bodies (Hong et al. 2023). Moiré patterns are notica- ble. To this end, prior works (Lin et al.) 2023 Wang et al. (023b) adopted two-stage optimization techniques. In these approaches, explicit 3D representations, such as Deep Marching Tetrahedra (DMTett) (Shen et al. 2021), are used to extract textured meshes from the implicit representations in
2

---

Published as a conference paper at ICLR 2024
the first stage and are subsequently fine-tuned in the second stage to capture high-quality geometry. However, these mesh representations forfeit the ability to produce detailed appearances such as fur- tradeoff wish to avoid. Differently, we aim maintain the flexibility and photo-realism offered by the NeRF representation while at the same time achieving high-quality text-to-3D generation through a single-stage training.
To this end, we propose two techniques to advance NeRF optimization. Specifically, to address the cloudy geometry issue in NeRIs, we propose a variance regularization that minimizes the variance of sampled z-coordinates distributed along NeRF rays. We observe that this technique enables NeRFs to more accurately represent crisp geometrical surfaces, thereby effectively mitigating the cloudiness issue. Additionally. we verify that our proposed z-variance regularization outperforms alternative spatial regularization proposed in previous methods (Barron et al. 2022).
Moreover, texture flickering or shimmer effects often result from inaccuracies in estimating the im- portance sampling weights across different rendering views. However, existing solutions, such as increasing the number of sampling points along the rays or deploying separate density estimation networks for each coarse and refined stage, come with increased computational demands. Instead, we propose kernel smoothing technique tailored for coarse-to-fine importance sampling along NeRF rays without an increase in the computational cost. This technique inspired by the inte- grated positional encoding for spatial points within a cone, previously proposed to tackle aliasing issues in a single image view (Barron et al. 2021). In our case, the goal is to mitigate flickering issues across multiple views. Specifically, (PDF) estimated in the coarse sampling stage along ray, which enables more comprehensive sam- pling near asset surface regions in the refined stage. This technique notably enhances the fidelity of importance sampling.
kernel refine the probability density function
We summarize our technical contributions for two crucial components of text-to-3D generation: (1) 3D representation and (2) optimization, which are outlined below:
To achieve photo-realistic and highly-detaieed text-to-3D generation, we propose score distillation in both the latent and image space of the pre-trained text-to-image diffusion models. Moreover, we introduce a timestep annealing approach for score distillation from text-to-image diffusion models.
To achieve sharp geometry quality through single-stage training, we present a regulariza- tion method applied to the variance of z-coordinates along NeRF rays.
To address flickering issues in NeRFs, we propose a kernel smoothing technique that refines the PDF estimation in coarse-to-fine importance sampling.
Taken together, we show how these holistic modifications address existing shortcomings and im- prove the quality of 3D synthesis.
2 RELATED WORK
Unconditional 3D asset generation involves the learning of 3D asset data distributions. Explicit ap-
proaches employ representations including point clouds (Achlioptas et al. 2018: Luo & Hul 202
voxel grids (Lin al. 2022 Smith & Meger 2017 Xie et al.1 and meshes (Zhang al. 2021
In contrast, implicit methods utilize representations such as signed distance functions (SDFs (Chen
& Zhang, 2019 Cheng al. [0233] Mittal al. 2022), tri-planes (Chen al. 2023a) multi-layer
perceptron (MLP) weights (Erkoç et al. 2023). and radiance fields (Lorraine et al. 2023). How-
ever, due to the limited availability of diverse 3D assets, these works primarily focus on generating
class-specific and small-scale 3D datasets.
Text-to-3D asset generation refers the creation of 3D assets based on text deseriptions. Instead of depending on limited text-annotated 3D datasets, the availability of ample text-image data pairs and the success of text-to-image generative models have inspired recent research to lift pre-trained text-to-image models into the 3D domain. Generally, these approaches can be categorized into two groups: (i) -uuPdg text-to-3D approaches (Khalid et al. 2022 Jain et al. 2022) that utilize pre-trained cross-modal matching models like CLIP (Radford et al. 2021), and (ii) 2D diffusion-
guided text-to-3D approaches (Poole et al.) 2022 Tang et al. 2023; Lin et al.] 2023 Chen et al.
2023b) that rely text-to-image diffusion based generative models such as Imagen (Saharia al.
2022) and StableDiffusion (Rombach et al. 2022). We follow the diffusion- -giieed methods due to
their superior performance in text-to-3D generation.
3

---

Published as a conference paper at ICLR 2024
Specifically, Poole al. (2022) first introduced a Score Distillation Sampling (SDS) approach, where noise is added to an image rendered from NeRFs and subsequently denoised by a pre-trained text-to-image generative model (Saharia et al. 2022). SDS minimizes the KL divergence between a prior Gaussian noise distribution and the estimated noise distribution. SDS is widely adopted in follow-up works (Lin al. [023] Tang al. 2023 Chen et al., 2023b). For example, Score- bacobian-Chaining (Wang et al. (023a) proposed a Perturb-and-Aveen
Scoring method to aggre- gate 2D image gradients of StableDiffusion (Rombach et al. 2022) over multiple viewpoints into a
3D asset gradient. [Wang et al. (203b) introduced a variational formulation of the SDS approach for diverse generation of 3D assets, yet needs to train a low-rank adaptation (LoRA) (Hu et al. 2022) for each individual 3D asset to provide the score function of the distribution. Moreover, two-stage optimization frameworks are proposed (Wang et al., 2023b Lin et al., 2023) that initially extract 3D meshes from implicit representations and then fine-tune them in the second stage to achieve high- resolution details. In this work, we propose refining of the SDS approach and improving the implicit representation to achieve high-quality 3D asset generation in a single-stage optimization process.
Image-to-3D reconstruction refers to 3D reconstruction from a provided single image. Typically, as proposed in prior works (Zhou& & Tulsiani 2023 Gu et al. 2023 Liu al. (a).bla), pre-trained text-to-image diffusion-
models are used to provide a 2D prior via the SDS approach plus an image reconstruction loss. In two-stage optimization process, Qian et al. (2023) employs 2D and 3D diffusion priors. In the first stage, they optimize a NeRF representation, and in the second stage, they extract a DMTet mesh from the NeRF for fine-tuning. Haque et al. (2023) propose an iterative dataset update strategy for editing NeRFs, leveraging text-to-image diffusion models. Alternatively, Liu et al. (023c) hallucinates 16 normal view images and directly optimizes a NeRF representation based on them. We extend our work on the image-to-3D reconstruction task and compare to these
methods in Sec. 5.4
PRELIMINARIES: SCORE DISTILLATION SAMPLING (SDS)
The SDS approach in diffusion models is proposed in recent work (Poole et al. 2022) using a pre- trained rext-to-image diffusion model to guide the 3D representation parameterized by 0. An image x is generated based on a given camera pose via a differeeniablee rendering function g. denoted as = g()) The pre-trained text-to-image diffusion model is employed to ensure the rendered images align with its learned image distribution. This work uses a latent diffusion model to reduce computational complexity.
Specifically, a latent diffusion model such as Stable Diffusion (SD) (Rombach et al. 2022 ), consists
of encoder E, decoder D, and denoising function cg. parameterized by Φ. The encoder & compresses the input image æ into low-resolution latent vector z, written as 2 = Exz). Conversely, the decoder D reconstructs the image from the latent vector as x D(z). The denoising score function Co estimates the given noise as ē == co(2;; v, where z; is noisy latent vector, formally written as z = 0+ t.c Here, a₂ and o, define a schedule for adding Gaussian noise (O0, I) to the latent vector z given a text embedding y at timestep t. Subsequently, the SDS loss is used to provide gradients for optimizing the 3D representation 0, written as
0z
VoLsDs z) = E., (t) 00
where wtt is weighting function.
4 APPROACH
We aim to generate high-quality 3D assets in a single-stage approach driven by text prompts. For this, we propose our method as illustrated in Fig. 2 We present our technical contributions in two parts. In Sec. 4.1 we compute the SDS loss in both the latent and image spaces of the pre-trained SD model (Rombach al. 2022). Unlike previous works (Poole et al.) [022,] Lin et al., [223) Chen et al. 2023b), we propose simple yet effective timestep annealing approach that gradually reduces timesteps (noise levels) throughout the optimization process. In Sec. 442] we introduce our variance regularization loss for z-coordinates along NeRF rays. Additionally, we present kernel smoothing technique for importance sampling, enabling NeRFs to produce crisp geometry and maintain a view- consistent appearance.
4.1 ADVANCING SDS-BASED OPTIMIZATION
Augmenting the SDS approach in both the image and latent spaces. In our work, we employ a pre-trained SD model (Rombach et al. 2022 to optimize NeRFs. We extend the score distillation
4

---

Published as a conference paper at ICLR 2024
Latent diffusion prior
x
E~N(0,11
leather shoe made from dragan skin"
Encoder
LSDS+
2t
Back propagation
20
g(0)
Denoising
Decoder
step
Figure 2: Overview of proposed method for text-to-3D generation. We aim to optimize 3D) model g()) using a pre-trained 2D latent diffusion prior To achieve this, we employ a latent diffusion model for score distillation. Specifically, the diffusion model takes rendered image as input and provides the estimate of the input rendered image, denoted as a. We utilize LsDS. loss that computes reconstruction loss both the latent and image spaces.
to both the latent and image spaces of the SD model. For this, we first reformulate the original SDS loss (as described in Eq. in terms of the latent vector residual instead of the noise residual:
0z
VoLsus (0, 2) = Ei,e ((t)(E c)
00
(2)
1
Oz 00
Oz
Ei, w(t) (z1 a1 (z) a(2)
Er,e
w<()
(z
Tt
00
where 8 1 1(z - σr represents the estimate the latent vector z using the denoising function
co. and (8 -0= =) is referred to as the latent vector residual. Note that due to the difficulty of recovering an explicit loss formulation LSDS for the gradients in Eq. Poole et (2022). directly compute the gradients to optimize 3D representations. In contrast, our reparameterization of the gradients as
shown Eq. Dallows to explicitly formulate the LsDS loss, thus simplifying the loss visualization and analysis process, Formally, we have
Lans(dd z) Rr, w([)l 211
where we incorporate those coefficients related to t into wt)..
Subsequently, we further adapt the loss by incorporating supervision for high-resolution images. Formally, we define the adapted loss LSDS+ as
LSDS+ (0, x,x) E,, wt) [|| 2112 + Augb x 2]
(4)
where & is recovered image obtained through the decoder 2, formally denoted Argb is a scaling parameter. We note that a similar image reconstruction loss is employed in recent image-to-3D reconstruction works (Zhou Tulsianil 2023). Our approach is different in two ways. First, we observe that it is inadequate only to use the image residual, i.e., | L| without the incorporation of the latent residual || 2| resulting in color bias issues in text-to-3D generation.
D(2) and
We will present the ablation experiments in Sec. 5.2 Second, using a random timestep sampling
approach in previous works (Zhou Tulsiani] 2023 Poole et al. 2022 Lin et al.) 2023) during
the denoising process limits the guidance of text-to-image diffusion models. In the following, analyze this in detail and introduce novel timestep annealing approach designed to enhance the SDS performance.
timestep annealing approach offers a more effective alternative to random timestep sampling used in previous works (Poole al.) 2022 Lin al., 2023). To be concrete, observations suggest that random sampling can introduce divergence issues in the denoised images. As training nears completion, NeRF renders images representing almost determined 3D asset. In this case,
5

---

Published as a conference paper at ICLR 2024
0.12
0:32
0.09
b09
2.4
2000
0.
2.55
2.00
3.43
3,80
2.85
2.05
2.51
2.54
2.85
2.80
2.00
2.34
(a) Coarse
(b) Refined (w/o and w/ KS)
(c) w/o KS (d) Flickering (e) w/ KS
Figure 3: Visualization of the flickering issue. We display the sampled z-coordinates along the ray for the rendered pixel (marked in the red square in (c) and (e)). Specifically, In (a), the ground- truth PDF is shown in blue, while the estimated PDF is shown without (yellow) and with (green) the kernel smoothing (KS) approach. In (b) we show the sampled z-coordinates in the refined stage, without KS (left) or with KS (right). Their corresponding rendered image is presented in (c) and (e), respectively. In (d), we overlay the difference of the two renderings (i.e. the flickering) on (c).
if a large timestep is randomly sampled, the denoising function might predict an image that is distinct and wnrelated to the given imput rendering. This can produce inaccurate gradients from the diffusion model, thereby negatively impacting the optimization of the 3D model.
To cireumvent this issue, we propose timestep annealing approach. Specifically, we use a high value of timestep t for the rendered image during the initial training iterations. This intentional noise allows the image align more closely with the distribution characterized by the text-to-image diffusion prior. As training proceeds, we gradually reduce the timestep I, thereby capturing finer details through more stable and lower variance gradients.
A question follows: what is the suitable annealing rate? We investigated several options, including linear, cosine, and square root schedules. Empirical evaluation (details in Sec.5) suggests that square root scheduling yields superior results in our scenario, formally written as
iter
Lmaxx
(fmax - fminn ) V total
(5)
Liter
where timestep decreases steeply during the initial training process and decelerates the training progresses. This scheduling allocates more training iterations to lower values of timestep f, ensuring that fine-grained details sufficiently captured in the latter iterations of training.
4.2 ADVANCING REGULARIZATIE IN NERF REPRESENTATION
We introduce two techniques to improve NeRF representations, including regularization method for the variance of z-coordinates (a.k.a. z-variance) sampled along NeRF rays and novel kernel smoothing approach for importance sampling during rendering.
A NeRF renders pixel color Č, of an image, denoted as Č, = =N= 1sC,, where vs and e; are respectively estimated weights and colors of the sampled coordinate 2; along a ray r (Mildenhall et al. (2020). N refers to the number of sampled points along the ray Accordingly. the depth value 14,, and the disparity value dz, of the ray r are written as
Per
Σ 24 Σ
and dz,
(6)
Uz.
where 24 can be considered as a sampled PDF. E.m
Regularization for z-variance aims to minimize variance in the distribution of sampled z- coordinates z; along the ray r. A reduced variance indicates crisper surfaces in geometry. For instance, in an extreme case where the rendering weights of a ray follow Dirac delta function, the z-variance will be zero, resulting in an extremely sharp surface. Formally, we denote the z-variance along the ray r as o².-
E2 [(=
12,)
E(a- M2..) Σ,
Vi
The regularization loss Lavar for the variance 03. is defined as
L = E, 8, = 1f Σ vs > 0.5, else 0.
(8)

---

Published as a conference paper at ICLR 2024
Dreamfusion
Magic3D
Fantasia3D
Ours
Figure 4: Visual comparisons to baseline methods. We visualize rendered images and extracted
meshes, and compare with Dreamfusion (Poole et al. 2022). Magic3D (Lin et al. (023), and Fan- tasia3D (Chen et al. 2023b). Prompts: "A plate piled high with chocolate chip cookies" (top) and "An ice cream sundae" (bottom).
LsDs-Image Figure 5: Ablation study of LsDS+ with (1) the full SDD loss Lsns+, (2) the SDS loss in latent space only, denoted as LSDS-Laents and (3) the SDS loss in image space only, denoted as LsDs-Image- Prompts are: (a) "A baby bunny sitting on top of a stack of pancakes" and (b) "A peacock on surfboard".
CSDS+
LaDs-Lttet
-sDs-Inage
LSDS+
LsDS-Larent
Here, 8, acts an indicator function (or binary weight) to filter out background rays. We find this loss remarkably useful for ensuring geometrical consistency and eliminating cloudy geometri- cal artifacts in our 3D model. In Fig. 6 we also compare the regularization loss L to existing regularization strategies (Barron et al. 2022).
Consequently, the total loss function is defined as,
L = LSDS+ + Cavars
(9)
where Avvar is the loss weight. We present our training procedure in the appendix, Algorithm |
Kernel smoothing for coarse-to-fine importance sampling. We observed that while integrating the v-vaiiance loss Lovar sharpens the density distribution along the rays, it also intensifies the flick- cring appearance. We consider the issue arising from the increased challenges PDF of volume density along these rays. To address this, propose simple yet effective kernel smoothing (KS) technique for coarse-to-fine importance sampling during rendering. Specifically, the KS approach involves weighted moving average of neighboring PDF values estimated during the coarse stage. The weight defined by a sliding window kernel. This ensures broader sam- pling scope the high-density regions in the refined stage. Formally, in the coarse stage, for E Ky 1X where Σx K2
estimating the
each weight v along ray the KS approach flattens the weight 02
K R the kernel. practice, we set [1, 1, 1]. Fig. we visualized in (a) the ground truth and the estimated distribution of volume density along NeRF ray in the coarse stage. In (b), we display the sampled z-coordinates the refined stage, either without (on the left) or with (on the right) the KS approach. Their corresponding rendered images shown in (c) and (e). Fig. F shows that the KS approach ensures comprehensive sampling near the peak of the density distribution, achieving multi-view consistent renderings and eliminating flickering issues.

---

Published as a conference paper at ICLR 2024
w/o Levar + distortion loss Figure 6: Ablation study of the z-variance loss Lzvar- We experiment with (1) the full loss L, (2) the loss without the v-rrinacee loss Lvar and (3) the loss where the 7-variance loss replaced with an alternative distortion loss (Barron et al. 2022). We show rendered example on the left, zoomed-in result at the top right and the corresponding depth image at the bottom right.
Full L
w/O Levar + distortion loss Full C
Sqrt (Ours)
Random
Linear
Cosine
Sqrt (Ours)
Random
Linear
Cosine
Figure 7: Ablation study of timestep annealing, We experiment various timestep annealing
schemes including square root, random sampling, linear and cosine. The results suggest that the square root annealing rate yields supiror performance w.r.t. photo-realism and reasonable geometry.
EXPERIMENTS
We evaluate our method to generate 3D assets from text prompts in Sec. 5.1 Specifically, we compare method with popular text-to-31) generation methods, Dreamfusion (Poole al. 2022 Magic3D (Lin et al. 2023). and Fantasia3D (Chen et al., 2023b). Additional comparisons to the concurrent work PromifiDDreamer (Wang et al. 2023b) are presented in Appendix A..4 Moreover, conduct extensive ablation studies in Sec. 5.2 to verify the effectiveness of each proposed technique. We present experiments with an alternative text-to-image diffusion model in Sec. [533 In Sec. we extend our method on the image-to-3D reconstruction task, compared with baseline methods (Liu
al.
2023b
Qian et al. 2023 Liu al. 2023c). Implementation details are Appendix A.
5.1
EXPERIMENTAL RESULTS
Qualitative rendered results of the 3D assets generated by our approach are depicted in Fig, | Our proposed approach generates high-fidelity 3D assets, with photo-realism and multi-view consistency. Additional results are shown in Appendix A.3
Qualitative comparisons to baseline methods are shown in Fig. 4
Specifically, in Fig.
compare our method with Dreamfusion (Poole al. 2022). Magic3D ([.in et al. (023). and Fanta- sia3D (Chen et al., 2023b) for text-to-3D generation.
observe that our rendered images exhibit enhanced photo-realism, improved texture details of the 3D assets, and more natural lighting effects. Additional comparisons are shown in Appendix A.4
5.2 ABLATION STUDY
Ablation on image-space regularization. Fig. 5 compares results of three different SDS loss set- tings: (1) the full SDS loss LSDS+, (2) the SDS loss in latent space only, denoted LsDs-Latens and (3) the SDS loss in image space only, denoted as Lsps-Image- The results indicate that incorporat- ing the image-space regularization contributes to a more natural appearance and enhanced texture details, as exemplified by the peacock images. However, relying solely on the image-space loss LSDS.Image results color bias issues, regardless of the guidance scale used.
Ablation on the z-variance loss Lzvar is shown in Fig. In this case, we compare the results obtained using (1) the full loss, (2) the loss without the z-variance loss Lzvar. and (3) the loss where z- z-variance loss Czvar is replaced with an alternative distortion loss introduced by Barron et al. 2022 originally for outdoor scene reconstruction. Notably, the absence of the z-variance loss, Lzvar, leads to the generation of assets with cloudy artifacts. While the distortion loss mitigates this cloudiness issue, it occasionally compromises appearance details and hollow geometry. comparison, our proposed z-variance loss Lavar consistently yields photo-realistic results with crisp geometry.
8

---

Published as a conference paper at ICLR 2024
SD
IF
SD
IF
SD
IF
Figure 8: Experiments with alternative Deep Floyd IF model. We experiment with the stage-1 model IF, which employs T5-XXL (Raffel et al., 2020) text encoder, and provides guidance in 64 x 64 resolution. Prompts: (a) "a tiger dressed like a doctor". (b) "a wide angle zoomed out DSLR photo of skiing penguin wearing a puffy jacket", and (c) "a roast turkey on a platter with only one pair of legs and one pair of wings".
Novel view
Input view
Zero-1-to-
Magic SyncDreamer
Ours
Figure 9: Novel view image generation given single view image. We compare method with concurrent works, including Zero-1-to-3 (Liu et al. 2023b), Magic123 (Qian et al. (023), and SyncDreamer (Liu al. 2023c
Ablation on timestep annealing. Fig. shows rendering images using different timestep sampling schemes. These include our proposed square root annealing rate, the random sampling adopted in
prior works/Pools et al. (2022) ; Lin et al. (2023), and both the linear and cosine annealing rates. We observe that the square root timestep annealing scheme outperforms the other baseline schemes in capturing detailed appearance and geometry.
5.3 OPTIMIZATION WITH GUIDANCE FROM AN ADVANCED TEXT ENCODER
In addition to using the SD model (Rombach et al. 2022). we also employ diffusion guidance from the Deep Floyd IF model | The IF model employs advanced text encoder, T5-XXL (Raffel al. 2020). In Fig. S we show difficulties in generating content for the terms "doctor" and "jacker" and the Janus problems (i.e., multi-face issues) when using the SD model. These challenges can be addressed using the IF model. Here, we only use the stage-1 I model in IF, generating images at 64 64 resolution. future direction would be to use the full model for high-resolution guidance.
5.4 IMAGE-TO- 3D RECONSTRUCTION
Our method also enables the reconstruction of 3D assets from a single image. To achieve this, we follow the concurrent work. SyneDreamer (Liu et al.) 2023c), which involves hallucinating 16-view images for auxiliary image reconstruction. Then, we incorporate an image (or latent) reconstruc- tion loss for the given view(s) and utilize our score distillation method to optimize the remaining randomly sampled views. Visual comparisons with the baseline methods are in Fig. [ We observe that our method can produce advancing photo-realistic images from novel views with reasonable de- tails. We also conduct an image- guieeeg hallucination experiment where initially the image reconstruction loss, transitioning to exclusively use our proposed loss L throughout all views. See Appendix A.5 for additional results.
6 CONCLUSION
We propose novel approach for high-quality text-to-3D generation in single-stage training. Specifically, we distill denoising scores from the pre-trained text-to-image diffusion models both the image and latent spaces, paired with novel timestep annealing approach. Moreover, we propose two general improvements for NeRFs, including z-variance loss and a kernel smooth approach, en- suring 3D representation with consistent appearance and sharp geometry.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
This work is partially supported by NSF 2046795, IIS 1909577, CCF 1934986, NIH IR0IMH111226-01A, NIFA award 2020-67021-32799, the Alfred P. Sloan Foundation, and Google Inc.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for 3d point clouds. In ICML, 2018.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single- stage diffusion nerf: A unified approach 3d generation and reconstruction. In ICCV, 2023a.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.13888 2023b.
Yiwen Chen, Chi Zhang, Xiaofeng Yang. Zhongang Cai, Gang Yu, Lei Yang, and Guosheng Lin. It3d: Improved text-to-3d generation with explicit view synthesis, 2023c.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In CVPR, 2019.
Yen-Chi Cheng, Hsin- Ying Lee, Sergey Tuyakov, Alex Schwing, and Liangyan Gui. SDFusion: Multimodal 3d shape completion, reconstruction, and generation. CVPR, 2023.
Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nicßner, and Angela Dai. Hyperdiffusion: Generat- ing implicit neural fields with weight-space diffusion, 2023.
Jiatao Gu, Alex Trevithick, Kai-En Lin, Josh Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi. Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion. ICML, 2023.
Ayaan Haque, Matthew Tancik, Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2nerff: Editing 3d scenes with instructions. In ICCV, 2023.
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207. 12598, 2022.
Fangzhou Hong, Zhaoxi Chen, Yushi Lan, Liang Pan, and Ziwei Liu. Eva3d: Compositional 3d human generation from 2d image collections. ICLR, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. ICLR, 2022.
Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. Shot text-guided object generation with dream fields. CVPR, 2022.
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Popa Tiberiu. Clip-mesh: Gen- erating textured meshes from text using pretrained image-text models. SIGGRAPH Asia 2022 Conference Papers, December 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023.
Chich Hubert Lin, Hsin- Ying Lee, Willi Menapace, Menglei Chai, Aliaksandr Siarohin, Ming- Hsuan Yang, and Sergey Tulyakov. Infinicity: Infinite-scale city synthesis. ICCV, 2022.
Hsin-Ying
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varna T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image 3d mesh in 45 seconds without per-shape optimization, 2023a.

---

Published as a conference paper at ICLR 2024
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: seoo one image to 3d object, 2023b.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Learning to generate multiview multiview-consistentt images from single-view image. arXiv preprint arXiv:2309.0344 033400,,..333... 2023c.
Jonathan Lorraine, Kevin Xic, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung-Yi Yi Lin, Ming-Yu Liu, Sanja Fidler, and James Lucas. Att3d: Amortized text-to-3d object synthesis. ICCV, 2023.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In CVPR, 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020.
Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. AutoSDF: Shape priors for 3d completion, reconstruction and generation. In CVPR, 2022.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphies prim- itives with multiresolution hash encoding. ACM Trans. Graph., 2022.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.1444 14988, 2022.
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306. 17843, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. resolution image synthesis with latent diffusion models. In CVPR, 2022.
High-
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NewrIPS, 2022.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. NeurlPS, 2021.
Edward Smith and David Meger. Improved adversarial systems for 3d object generation and reconstruction. In CoRI., 2017.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021.
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. arXiv preprint 2wXi::2333.14844, 2023.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Ych, and Greg Shakhmarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. CVPR, 2023a.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. NeurIPS, 2023b.
12

---

Published as a conference paper at ICLR 2024
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song- Ch---Chhm Zhu, and Ying Nian Wu. Learn- ing descriptor networks for 3d shape synthesis and analysis. In CVPR, 2018.
Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models. arXiv preprint arXiv:2212. 14704, 2022.
Song-Hai Zhang, Yuan-Chen Guo, and Qing- Qing-Wen Gu. Sketch2model: View-aware 3d modeling from single free-hand sketches. CVPR, 2021.
Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. In CVPR, 2023.
Peiye Zhuang, Samira Abnar, Jiatao Gu, Alex Schwing, Joshua M Susskind, and Miguel Ángel Bautista, Diffusion probabilistic fields. In ICLR, 2023.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
We present our implementation details in Sec. A.1 Further 3D asset generation results can be viewed in Sec. A.3 and additional comparisons to text- to-3D baseline methods available in Sec. A.4 In Sec. A.5 show additional image-to-3D reconstruction results and experiments with an image-guune sonngeeguieeddde hallucination task where we generate 3D assets that are hallucinated from the given input image (rather than reconstruction). Please refer to our video demo in the supplementary material for a comprehensive overview.
Our training algorithm is shown in Sec. A.2
A.1 IMPLEMENTATION DETAILS
Model setup. Our approach is implemented based a publicly available repository In this im- plementation, a NeRF is parameterized by a multi-layer perception (MLP), with instant-ngp (Müller al. 2022) for positional encoding. To enhance photo-realism and enable flexible lighting mod- eling, instead of using Lambertian shading as employed in (Poole et al. 2022), we encode the ray direction using spherical harmonics and utilize a background network that predicts background color solely based on the ray direction. We employ pre-trained SD model as diffusion prior, as well as pre-trained dense prediction model to predict disparity maps.
an input to NeRF. Additionally, we incorporate
Training setup.
We use Adam (Kingma & Ba 2015) with a learning rate of 10-2 for instant- ngp encoding, and 10-3 for NeRF weights. In practice, we choose total iter as 10 iterations. The rendering resolution 512 x 512. We employ DDIM (Song et al. 202I) with empirically chosen parameters
0.25, and r = 1 to accelerate training. We choose the hyper-parameters Agbb = 0.1, Ad = 0.1, and Azvar = 3. Similar to prior work (Poole et al. 2022 Lin et al. 2023]
Wang et al. 2023a). we classifier-fi free guidance (Ho & Salimans 2022
100 for our diffuston
model.
A.2 TRAINING ALGORITHM
We present training procedure in Algorithm | In step 5, either single-step multi-step denoising approach can be used to estimate the latent vector z. Here, the multi-step denoising refers to the iterative denoising of ε,, until f = 0.
Algorithm Training Procedure
Input: A pre-trained SD Rombach et al. (2022) consisting of an encoder E, decoder D, and a
denoising autoencoder ca; rendering g(0); latent vector {(x): ) number of total training steps total_iter; range of the diffusion time steps [fmaxx min] ] a conditioning y; scaling coefficients a₂ and d,
1: for iter = [0, total iite] do
2:
fmax (fmax (min
Her total iter
3: 2, = 0,2 + O.C, where NO. I)
4: Estimating noise è = Eg( H,t)
5: Estimating the latent vector 2 J [zz - o,ē) via either single- or multi-step denoising
6: Estimating the image å
D(
7: Compute the loss gradient V and update 9
8: end for
Return:
A.3 ADDITIONAL RESULTS TEXT-TO-3D GENERATION
We provide generated 3D assets given text prompts in Fig.
10
12
WWe use the pre-trained SD in https://github.com http://gghbbb.oomnnnggggggcccsss http://hww...oooougggggg//////mm
14

---

Published as a conference paper at ICLR 2024
A.4 ADDITIONAL COMPARISONS TO THE BASELINE METHODS
We present additional comparisons to the baseline methods in Fig 13 18 following the rendering
settings used in ProlificDreamer (Wang 2023b).
Specifically, in Fig. 13 we present results only using NeRF representation, comparing them to two baseline methods, namely ProaificDreamer (Wang al. 2023b) and DreamFusion (Poole et al. 2022). In this case, no fine-tuning stage for 3D asset generation is applied in these baseline methods illustrated in Fig. [3] our method allows the generation of high-fidelity details and natural colors through only single-stage optimization. We observe flickering issues and improper geometries when using only the NeRF representation in ProlificDreamer (Wang al.] 2023b). In contrast, our method consistently provides view and geometry-consistet results without flickering.
In Fig. 17 we present additional visual results, comparing them to the baseline methods, including ProlificDreamee (Wang et al. (023b), Fantasia3D Chen et al. (2023b), Magic3I) (Lin et al. 2023)
and DreamFusion (Poole et al. [022). In this case, the baseline methods (Wang et al. al.] 2023b; Lin
ctt al. 2023) employ the full training pipeline, which includes NeRF representation followed by fine-tuning.
Additional comparisons with Fantasia3D (Chen et al. 2023b) and Magic3D Lin et al. shown in Fig.
2023
16. and comparisons with Dream (Poole et al. 2022 in Fig. 18
In Fig. 19 integrate the z-variance loss into ProlificDreamee.. We observe that incorporating the z-variance loss results in sharper textures. 20g we present the results ProlificDreamer both without and with proposed method, which includes the z-variance loss, the image-space loss, and the square root time-step annealing schedule. From the results, our method enhances the baseline approach, enabling it to generate superior renderings with detailed textures.
A.5 ADDITIONAL RESULTS IMAGE-TO-3D RECONSTRUCTION
In Fig. 21 we present additional image-to-3D reconstruction results. Additionally, we conduct image-guided 3D hallucination experiments. Specifically, we execute image-to-3D reconstruction at early training iterations, and then optimize the NeRF representation only using our proposed distillation loss, omitting the image reconstruction loss. We show these results in Fig. 22
15

---

Published as a conference paper at ICLR 2024
"A ladybug"
"A low-poly tree"
"A model of the Eiffel Tower made out of toothpicks"
"A model of the Eiffel Tower, aerial view"
"A high detailed octopus, 4k"
"AA delicious croissant"
'An astronaut is riding a horse"
'AA broken old clay vessel"
Figure 10: Additional 3D asset generation results with the corresponding normal map given text prompts (below each object).
16

---

Published as a conference paper at ICLR 2024
"Gold skull, 4k, highest quality"
"The leaning tower of Pisa, aerial view"
"A pomeranian dog"
"A watermelon" "
'AA DSLR photo of Cthulhu"
''A wooden medieval shipping barrel"
"Pumpkin head zombie, skinny, highly detailed, photorealistic"
"A high detailed pumpkin, style cartoon, front view, 4k, orange, funny and cute, HDR, Halloween"
Figure 11: Additional 3D asset generation results with the corresponding normal map given text prompts (below each object).
17

---

Published as a conference paper at ICLR 2024
"A tulip"
"Neuschwanstein Castle, aerial view "
''A big coral"
"A pink cherry blossom shaped cake
"An ice cream sundae"
''A ripe strawberry"
"A shell"
"A stylized witch pot made of stone or mud with wide base and narrow neck, decorated with intricate carvings of runes and symbols, and is filled with bubbling, green liquid. The pot is surrounded by cloud steam"
Figure 12: Additional 3D asset generation results with the corresponding normal map given text prompts (below each object).
18

---

Published as a conference paper at ICLR 2024
Ous
Prolifi:Dreamer
DreamFosion
sitting stack pancakes"
Oas
Prolifietheamee
DreamFrsson
"A delicious croissont"
Figure 13: Additional visual comparisons using NeRF representation only. We compare visually with the baseline methods, ProlificDreamer (Wang al. 2023b) and DreamFusion (Poole et al. 2022), specifically after the first training stage. In this case, 3D assets are represented by NeRI, with no additional fine-tuning applied the baselines.
Ours
Fantasia3D
Magic3D
"A baby bunny sitting on top of stack of pancokes"
Ours
Fantasia3D
Magic3D
"A ripe strawberry"
Figure 14: Additional visual comparisons with Fantasia3D (Chen et al. 2023b) and Magic3D (Lin
al. 2023
19

---

Published as a conference paper at ICLR 2024
Ours
Fantasia3D
"a beagle eating donut"
Ours
Fantasia3D
''A blue jay standing on large basket of rainbow macarons"
Ours
Fantasia3D
Christmas with donuts as decorations"
Ours
Fantasia3D
"a brightly colored mushroom growing on a log"
Ours
Fantasia31)
"an orangutan making a clay bowl on a throwing wheel"
Figure 15; Additional visual comparisons with Fantasia3D CCen et al. 2023b)
20

---

Published as a conference paper at ICLR 2024
Ours
Fantasia3D
"a dalmatian wearing a fireman's hat"
Ours
Fantasia3D
"an unstable rock cairn in the middle of stream"
Ours
Fantasia3
"a red-eyed tree frog"
Ours
Fantasia3D
"a dragon-cat hybrid"
Ours
Fantasia3D
"an amigurumi motorcycle"
Figure 16: Additional visual comparisons with Fantasia3D (Chnn al. 2023b
21

---

Published as a conference paper at ICLR 2024
Ours
PremificDrramer
Fantasia3D
Magic3D
Dream Fusion
A car made out of sushi"
Figure 17:
Additional visual comparisons with the baseline methods, including Prolific- Dreamer (Wang et al.] (223b), Fantasia3D Chen et al. (2023b). Magic3D (Lin al., 2023 and DreamFusion (Poole et al.] (2022.. In this case, the baseline methods (Wang et al.; 2023b Lin al. (023) employ the full training pipeline, which includes NeRF representation followed by fine- tuning.
22

---

Published as a conference paper at ICLR 2024
Ours
DreamFusion
"A ladybug"
DreamFusion "A 3d model of an adorable cottage with a thatched roof"
Ours
Ours
DreamFusion
"Michelangelo style statue of dog reading news on a cellphone" "
Ours
DrcamFusion
"A lionfish"
Figure 18: Additional visual comparisons with DreamFusion (Poole et al. 2022
23

---

Published as a conference paper at ICLR 2024
4K iters
7K iters
10K iters
Figure 19: Visual results of incorporating the z-variance loss ProaificDreamer (Wang et al., 2023b) throughout the training process. We show rendered results w// (left) and w/ (right) the Z- z-variance loss after 4K, 7K and 10K training iterations.
Baseline
w/ our method
Baseline
w/ our method
Baseline
w/ our method
Figure 20: The baseline results, ProlifieDreamer (Wag et al. 2023b), without and with our proposed method. This includes the z-variance loss, the image-space loss, and the square root time-step annealing schedule.
Novel views
Input view
Novel views
Input view
Figure 21: Additional results of image-to-3D reconstruction.
24

---

Published as a conference paper at ICLR 2024
Novel views
Input view
"' toy grabber with dinosaur head"
Novel views
Input view
"Hircine's Ring from Elder Scrolls Online" "
Novel views
Input view
"A colorful painted teapot"
Figure 22: Visual results of image-guided 3D hallucination. We hallucinate the 3D asset from single given image using the prompt below the object.
25


Reviews:
[{"review_id": "t4PhQ3gHWK", "reviewer": "Reviewer_iKQ3", "summary": "This paper proposed an improved version of Score Distillation Sampling by introducing several strategies. The authors proposed to perform denoising in both image and latent space for better performance. A novel timestep annealing strategy is provided to reduce the sampling space. Besides, the authors also provide a z-coordinates regularization term to achieve high-quality rendering in a single-stage optimization. The paper is well-organized and easy to follow. The proposed strategies are effective to improve the performance.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The strategy by denoising in both image and latent space is useful to improve the details.\n2. The proposed z-variance loss alleviates cloudy artifacts and shows better performance than distortion loss.\n3. The proposed method achieves high-quality text-to-3D generation.", "weaknesses": "Basically the paper is good and I have several concerns:\n1. The contributions of the paper are discrete, which comprises several small contribution points.\n2. Maybe a user-study should be conducted to quantitatively evaluate the method.\n3. Basically the proposed strategies can be generalized to other baseline methods, for example, ProlificDreamer [1]. I’m curious to see the performance with other baseline methods.\n\n[1] Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., & Zhu, J. (2023). ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. arXiv preprint arXiv:2305.16213.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kmkF6XQuCm", "reviewer": "Reviewer_WWao", "summary": "- The paper proposes holistic sampling and smoothing approaches for high-quality text-to-3D generation in a single-stage optimization. \n- The method introduces a timestep annealing approach and regularization for the variance of z-coordinates along NeRF rays.\n- The paper also addresses texture flickering issues in NeRFs with a kernel smoothing technique.\n- Experiments show the method's superiority over previous approaches.", "questions": "I believe the technical aspects are articulated clearly and technically sound. Thus, I have no further questions.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The single-stage optimization is useful to the generation of highly detailed and view-consistent 3D assets and the proposed solution to it is impressive.\n- Compared to baseline methods like Dreamfusion, Magic3D, and Fantasia3D, the rendered images from this approach exhibit enhanced photo-realism, improved texture details of the 3D assets, and more natural lighting effects.", "weaknesses": "- The paper could benefit from a more explicit explanation in the introduction regarding why previous works were unable to achieve single-stage optimization.\n- The contributions presented in the paper seem fragmented, lacking a cohesive thread or central theme. It would enhance the paper's clarity and impact if the authors could refine the structure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Ql08fTQIe", "reviewer": "Reviewer_CJyW", "summary": "This work proposes four techniques on the task of 2D diffusion-guided text-to-3D generation, to enhance the generation quality. In particular, the authors 1) propose score distillation in both the latent and image space of the pre-trained text-to-image diffusion models; 2) introduce a timestep annealing strategy to achieve photo-realistic and highly-detailed generation; 3) present a regularization method on the variance of z-coordinates along NeRF rays to encourage crisper surfaces; 4) they also propose a kernel smoothing technique to address flickering issues in the optimized NeRFs. They conduct qualitatively ablation studies and the experimental results demonstrate the effectness of the proposed techniques.", "questions": "* How is the kernel smoothing conducted for coarse-to-fine importance sampling? Could the authors provide more details? In fact, I do not understand \"kernel smoothing\". Use an equation to explain it would be very helpful. Figure 3 seems only presents the results with/without kernel smoothing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* From the presented experimental results (mainly qualitative results), the proposed techniques are effective and improve the performance over prior methods; \n* The ablation studies also demonstrate the effectiveness of individual technique;", "weaknesses": "* The experimental results are all qualitative results. It is good to have a metric/metrics to compare quantitatively against prior methods; For example, to measure the CLIP similarity between the text prompts and the generated contents; Otherwise, it is difficult to evaluate the performance since we can deliberately select good performing prompts over prior methods for comparisons. \n\n*  The results from Fantasia3D are also very impressive (i.e. in terms of texture quality and geometry) from Figure 4 and Figure 14. Can you provide more results to show that yours is better? I provide following text prompts from DreamFusion: 1) an orangutan making a clay bowl on a throwing wheel; 2) a raccoon astronaut holding his helmet; 3) a blue jay standing on a large basket of rainbow macarons; 4) a corgi taking a selfi; 5) a table with dim sum on it; 6) a lion reading the newspaper; 7) a tiger dressed as a doctor; 8) a chimpanzee dressed like Henry VIII king of England; 9) an all-utility vehicle driving across a stream; 10) a squirrel gesturing in front of an easel showing colorful pie charts. Can you do the comparisons with those prompts?\n\n* For the kernel smoothing, you only choose [1, 1, 1] as the sliding window kernel, have you tried other choices?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)