DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior
Authors: Jingxiang Sun
Bo Zhang
Ruizhi Shao
Lizhen Wang
Wen Liu
Zhenda Xie
Yebin Liu
Venue: ICLR 2024 poster (2024)
Keywords: text to 3D
diffusion models
NeRF
neural rendering
3d synthesis
Primary area: generative models

Abstract: We present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose bootstrapped score distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation.

Published as a conference paper at ICLR 2024
DREAMCRAFT3D: HIERARCHICAL 3D GENERATION WITH BOOTSTRAPPED DIFFUSION PRIOR
Jingxiang Sun'; Bo Zhanga1 Ruizhi Shao', Lizhen Wang', Wen Liu", Zhenda Xie², Yebin Liu Tsinghua University, 2 DeepSeek AI. Zhejjang University
ABSTRACT
We present DreamCraft31,, hierarchical 3D content generation method that pro- duces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boost- ing. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via view-depent dependentd diffusion model. This 3D prior, along- side several training strategies, prioritizes the geometry consistency but compro- mises the texture fidelity. We further propose Bootstrapped Score Distillation specifically boost the texture. We train a personalized diffusion model, Dream- booth, on the augmented renderings of the scene, imbuing with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffu- sion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in train- ing the scene-specifer diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DrcamCraft3D generates coherent 3D objects with photorcalistic ren- derings, advancing the state-of-the-ar in 3D content generation. Code available at httpps:///e /htthbb. thub. com/ deepseek- aiw.aaaarraaaaaaaaaaaaeeaeaaaaaa
INTRODUCTION
The remarkable success of 2D generative modeling (Saharia al., 2022; Ramesh et al., 2022; Rom- bach et al., 2022; Gu et al., 2022) has profoundly shaped the way that we create visual content. 3D content creation, which is crucial for applications like games, movies and virtual reality, still presents a significant challenge for deep generative networks. While 3D generative modeling has shown compelling results for certain categories (Wang al., 2023a; Chan al., 2022: Zhang al., 2023b), generating general 3D objects remains formidable due to the lack extensive 3D data. Recent research effort has sought to leverage the guidance of pretrained text-to-image generative models (Poole et al., 2022; Lin et al., 2023; Tang et al., 2023) and showcases promising results.
The idea of leveraging pretrained text-to-image (T2I) models for 3D generation is initially proposed by DreamFusion (Poole et al., 2022). A score distillation sampling (SDS) loss is enforced to opti- mize the 3D model such that its renderings at random viewpoints match the text-conditioned image distribution as interpreted by powerful T2I diffusion model. DreamFusion inherits the imaginative power of 2D generative models and can yield highly creative 3D assets. To deal with the over- saturation and blurriness issues, recent works adopt sage-wise optimization strategies (Lin et al., 2023) or propose improved 2D distillation loss (Wang al., 2023b), which leads to an enhancement in photo-realism. However, the majority of current research falls short of synthesizing complex content as achieved by 2D generative models. In addition, these works are often plagued with the "Janus issue", where 3D renderings that appear plausible individually show semantic and stylistic inconsistencies when examined holistically.
WWok partially done during the internship at DeepSeck AI. Corresponding authors.

---

Published as a conference paper at ICLR 2024
Humoristic san goku body mixed with wild boar head running
A beagle detective's outfit
cute rabbit in stunning, detailed Chinese coat
A DSLR photo of spiderman
Figure 1: By lifting 2D images to 3D, DreamCraft3D achieves 3D generation with rich details and holistic 3D consistency. Please refer to the Appendix and the demo video for more results.
In this paper, we propose DreamCraft3D. an approach to produce complex 3D assets while main- taining holistic 3D consistency. Our approach explores the potential of hierarchical generation. We draw inspiration from the manual artistic process: an abstract concept first solidified into draft, followed by the sculpting of rough geometry, the refinement of the geometric details and the painting of high-fidelity textures. We adopt similar approach, breaking down the challenging 3D generation into manageable steps. Starting with a high-quality 2D reference image generated from a text prompt, we lift it into 3D via stages of geometry sculpting and texture boosting. Contrary to prior approaches, our work highlights how careful consideration of each stage unleash the full potential of hierarchical generation, resulting in superior-quality 3D creation.
The geometry sculpting stage aims produce plausible and consistent 3D geometry from the 2D reference image. On top of using the SDS loss for novel views and photometric loss at the reference view, introduce multiple strategies to promote geometric consistency. Foremost, we leverage an off-the-shelf viewpoin--onnditione image translation model, Zero-1-to-3 (Liu et al., 2023c), to model the distribution of novel views based on the reference image. Since this view-conditioned diffusion model is trained on diverse 3D data (Deitke et al., 2023), it provides rich 3D prior that complements the 2D diffusion prior. Additionally, we find annealing the sampling timestep and pro- gressively enlarging training views are crucial to further improve coherency. During optimization, we transition from implicit surface representation (Wang et al., 2021) to mesh representation (Shen al., 2021) for coarse-to-fine geometry refinement. Through these techniques, the geometry sculpt- ing stage produces sharp, detailed geometry while effectively suppressing most geometric artifacts.
We further propose Bootstrapped Score Distillation to substantially boost the texture. Existing view- conditioned diffusion models trained on limited 3D often struggle to match the fidelity of modern 2D diffusion models. Instead, finetune the diffusion model according multi-view renderings of the 3D instance being optimized. This personalized 3D-aware generative prior becomes instru- mental in augmenting the 3D texture while ensuring view consistency. Importantly, find that
2

---

Published as a conference paper at ICLR 2024
alternately optimizing the generative prior and 3D representation leads to mutually reinforcing im- provements. The diffusion model benefits from training on improved multi-view renderings, which in turn provides superior guidance for optimizing the 3D texture. In contrast to prior works (Poole et al., 2022; Wang et al., 2023b) that distill from a fixed target distribution, we learn from a distri- bution that gradually evolves according to the optimization state. Through this "bootstrapping"" our approach captures increasingly detailed texture while keeping the view consistency.
As shown in Figure I, our method is capable of producing creative 3D assets with intricate geomet- ric structures and realistic textures rendered coherently in 360 Compared to optimization-base approaches (Poole et al., 2022; Lin et al., 2023), our method offers substantially improved texture and complexity. Mcanwhile, compared to image-to-3D techniques (Tang et al., 2023; Qian et al., 2023), our work excels at producing unprecedentedly realistic renderings in 360'renderings These results suggest the strong potential of DreamCrafth) in enabling new creative possibilities in 3D content creation. The full implementation will be made publicly available.
2 RELATED WORK
3D generative models have been intensively studied to generate 3D assets without tedious manual creation. Generative adversarial networks (GANs) (Chan et al., 2021; 2022; 2021; Xie et al., 2021; Zeng et al., 2022: Skorokhodov et al., 2023; Gao et al., 2022; Tang et al., 2022; Xie et al., 2021; Sun et al., 2023; 2022) have long been the prominent techniques in the field. Auto-regressive models have been explored (Sanghi et al., 2022: Mittal et al., 2022; Yan et al., 2022; Zhang et al., 2022; Yu al., 2023), which learn the distribution of these 3D shapes conditioned on images or texts. Diffusion models (Wang et al., 2023a; Cheng et al., 2023; Li et al., 2023; Nam et al., 2022; Zhang al., 2023a; Nichol et al., 2022; Jun & Nichol, 2023; Bautista al., 2022; Gupta et al., 2023; Long et al, 2023; Zou et al., 2023; Liu et al., 2023d; Hong et al., 2023) have also shown significant recent success in learning probabilistic mappings from text images to 3D shape latent. However, these methods require 3D shapes or multi-view data for training, raising challenges when generating in- the-wild 3D assets due the scareity of diverse 3D data (Chang et al., 2015; Deitke al., 2023; Wu et al., 2023) compared to 2D.
3D-aware image generation aims to render images in novel views while offering some level of 3D consistency. These works (Sargent et al., 2023; Skorokhodov et al., 2023; Xiang et al., 2023) often rely on a pretrained monocular depth prediction model to synthesize view-consistent images. While some models achieve photo-realistic renderings for ImageNet categories, they struggle with large views. Recent attempts (Watson et al., 2022; Liu et al., 2023c) training view-dependent diffusion models on 3D data show promising results for open-domain novel view synthesis but, as inherently 2D models, can't 't ensure perfect view consistency.
Lifting 2D to 3D approaches improve 3D scene representation by seeking guidance using estb- lished 2D text-image foundation models. Early works (Jain et al., 2022; Lee & Chang, 2022; Hong et al., 2022) utilize the pretrained CLIP (Radford et al., 2021) model to maximize the similarity between rendered images and text prompt. DreamFusion (Poole et al., 2022) and SJC (Wang et al., 2022), on the other hand, propose distill the score of image distribution from a pretrained dif- fusion model and demonstrate promising results. Recent works have sought to further enhance the texture realism via coarse-to-fine fine optimization (Lin al., 2023; Chen et al., 2023), improved dis- tillation loss (Wang et al., 2023b; Liu et al., 2023e; Huang et al., 2023c), shape guidance (Metzer et al., 2023) or lifting 2D image to 3D (Deng al., 2023; Tang et al., 2023; Qian et al., 2023; Liu et al., 2023b; Huang et al., 2023a). Recently, Raj et al. (2023) proposes to finetune personalized diffusion model for 3D consistent generation. However, producing globally consistent 3D remains challenging. In this work, we meticulously design 3D priors through the whole hierarchical genera- tion process, achieving unprecedented coherent 3D generation.
3 PRELIMINARIES
DreamFusion (Poole et al., 2022) achieves text-to-3D generation by utilizing a pretrained text-to- image diffusion model Es an image prior to optimizing the 3D representation parameterized by 8. The image z = g(), rendered at random viewpoints by a volumetric renderer, is expected to represent a sample drawn from the text-conditioned image distribution p(arly) modeled by pre-
3

---

Published as a conference paper at ICLR 2024
trained diffusion model. The diffusion model is trained to predict the sampled noise c y, t) of the noisy image 2 at the noise level t, conditioned on the text prompt y. A score distillation sam- pling (SDS) loss encourages the rendered images to match the distribution modeled by the diffusion model. Specifically, the SDS loss computes the gradient:
VoLsus(o, (()) Ei,e (mt)(c c) Öa
09.
which is the per- per-pixel difference between the predicted and the added noise upon the rendered image, where (1) is the weighting function.
One way to improve the generation quality of conditional diffusion model is to use the classifier- free guidance (CFG) technique to steer the sampling slightly away from the unconditional sampling, i.e., (0( y, t) + wewp(x;; y, t) (æ t, 2), where Ø represents the "empty" text prompt. Typi- cally, the SDS loss requires a large CFG guidance weight for high-quality text-to-3D generation, yet this will bring side effects like nee-saturation and over-smoothing (Poole et al., 2022).
Recently, Wang et al. (2023b) proposed a variational score distillation (VSD) loss that is friendly to standard CFG guidance strength and better resolves unnatural textures. Instead of seeking single data point, this approach regards the solution corresponding to a text prompt as a random variable. Specifically, VSD optimizes distribution ("(olly)) of the possible 3D representations ((lly) corresponding to the text y, to be closely aligned with the distribution defined by the diffusion timestep 0, p( y), in terms of KL divergence:
LVSD = DKL (1" (aul)))lpp(aa (ll
(2)
Wang et al. (2023b) further shows that this objective can optimized by matching the score of noisy real images and that of noisy rendered images at each time so the gradient of LVsD is
dx
VaLvsolo, ((0) E., (w(() (20 y. 1) Elora (ari y, L c))
(3)
00
Here, élora estimates the score of the rendered images using a LoRA (Low-rank adaptation) (Hu et al., 2021) model. The obtained variational distribution yields samples with figh-fidelity textures. However, this loss applied for texture enhancement and is helpless to the coarse geometry initially learned by SDS. Moreover, both the SDS and VSD attempt to distill from fixed target 2D distri- bution which only assures per-view plausibility rather than a global 3D consistency. Consequently, they suffer from the same appearance and semantic shift issue that hampers the perceived 3D quality.
4 DREAMCRAFT3D
We propose a hierarchical pipeline for 3D content generation as illustrated in Figure 2. Our method first leverages a state-of-the-ar text-to-image generative model to generate a high-quality 2D image from a text prompt. In this way, can leverage the full power of state-of-the-art 2D diffusion mod- els to depict intricate visual semantics described in the text, retaining the creative freedom of 2D models. We then lift this image to 3D through cascaded stages of geometric sculpting and texture boosting. By decomposing the problem, we can apply specialized techniques at each stage. For geometry, prioritize multi-view consistency and global 3D structure, allowing for some com- promise on detailed textures. With the geometry fixed, we then focus solely on optimizing realistic and coherent texture, for which we jointly learn 3D-aware diffusion prior that bootstraps the 3D) optimization. Next, we elaborate on key design considerations for the two phases.
4.1 GEOMETRY SCULPTING
At this stage, we aim to craft a 3D model such that it matches the appearance of the reference image & at the same reference view while maintaining plausibility under different viewing angles. To achieve this, we encourage plausible image renderings for each randomly sampled view, recog- nizable by pretrained diffusion model. This is achieved using the SDS loss LSDS, as defined in Equation 1. In order to effectively utilize guidance from the reference image, we penalize the pho- tometric difference between the rendered image and the reference via Lub = ml O (3 g(0: é)) at the reference view é. The loss is computed only within the foreground region denoted by the mask m.. Meanwhile, we implement the mask loss Lmack i
g.n (0; 1) 2 to encourage scene
4

---

Published as a conference paper at ICLR 2024
LsDs, LYn-SDS
From Implicit surface (NeuS) to 3D Mesh (DMTel)
Text-to-Inne to-llage Diffusion
Noise
*an astronaut in sand beach"
O
View-condilloned Diflusion
Reference Image
Estimated Normal
Novel View
Geamelty Sculpting Result
LASD
Novel Views
Text-ta to to-Image Diffusion
DreomBopth
Reference
view
[V]
Texture Boosling Result
Augmented
rendeerings
Figure 2: DreamCraft3D leverages a 2D image generated from the text prompt and uses it to guide the stages of geometry sculpting and texture boosting. When sculpting the geometry. the view- conditioned diffusson model provides crucial 3D guidance to ensure geometric consistency. We then dedicately improve the texture quality by conducting a cyclic optimization. We augment the multi-view renderings and use them to finetune diffusion model, DreamBooth, to offer multi-view consistent gradients to optimize the scene. We term the loss that distills from an evolving diffusion prior as bootstrapped distillation sampling (CaSD in the figure).
sparsity, where 9m renders the silhouette. In addition, akin to (Deng et al., 2023), we fully exploit the geometry prior inferred from the reference image, and enforce the consistency with the depth and normal map computed for the reference view. The corresponding depth and normal loss are respectively computed as:
conv (d, d) Coppth o (
n á
Lnormal
(4)
||ra||2
||
where conv(-) and (-)) represent the covariance and variance operators respectively, and the depth d and the normal ń at the reference view are computed using the off-the-shelf single-view estima- tor (Eftekhar et al., 2021). The depth loss adopts the form of negative Pearson correlation Ldepth to account for the scale mismatch in depth.
Despite these, maintaining consistent semantics and appearance across back-views remains chal- lenge. Thus, we employ additional techniques to produce coherent, detailed geometry.
3D-aware diffusion prior. We argue that the 3D optimization with per-view supervision alone is coddr-constrained. Hence, we utilize a view-condittoned diffusion model, Zero-1-to-3, which is trained a large scale of 3D assets and offers an improved viewpoint awareness. The Zero-I-to-3 is a fine-tuned 2D diffusion model, which hallucinates the image in a relative camera pose c given the reference image T. This 3D-aware model encodes richer 3D knowledge of the visual world and allows us to better extrapolate the views given a reference image. As such, we distill the probability density from this model and compute the gradient of 3D-aware SDS loss for novel views:
c) Ox
LoLND (0. (()) Er,e E.,,lu(t)(es( 22
(5)
ag
This loss effectively alleviates 3D consistency issues like the Janus problem. However, the fine- tuning on limited categories of 3D data of inferior rendering quality impairs the diffusion model's generation capability, the 3D-aware SDS loss alone is prone to induce deteriorated quality when
5

---

Published as a conference paper at ICLR 2024
lifting general images to 3D. Therefore, employ hybrid SDS loss, which incorporates both the 2D and 3D diffusion priors simultaneously. Formally, this hybrid SDS loss provides the gradient as:
V o ayyrnna (o. g(0) = VoLsDs g(0) + VV (Ds g()),
(6)
where we choose μ we adopt the DeepFloyd IF base model (Shonenkov et al., 2023), a diffusion model that operates at 64 x 64 resolution pixel space and better captures coarse geometry.
emphasize the weight of the 3D diffusion prior. When computing LSDS.
Diffusion timestep annealing, To align with the coarse-to-fine progression of 3D optmmization we adopt diffusion timestep annealing strategy similar to Huang et al. (2023b). At the start of optimization, we prioritize sampling larger diffusion timestep t from the range [0.7, 0.85] when computing Equation 6 to provide the global structure. As training proceeds, we linearly anneal the L. sampling range to [0.2, 0.5] over hundreds of iterations. This annealing strategy allows the model to first establish a plausible global geometry in the early optimization phase before refining the structural details.
Detailed structural enhancement. We initially optimize an implicit surface representation with the corresponding volume rendering as in NeuS (Wang et al., 2021) to establish the coarse structure. Then, following Lin et al. (2023), we use this result to initialize a textured 3D mesh representation using a deformable tetrahedral grid (DMTet) (Shen et al., 2021) to facilitate high-resolution details. Moreover, this representation disentangles the learning of geometry and texture. Hence, at the end of this structural enhancement, frequency details from the reference image.
able solely refine the texture and better preserve high-
We provide a summary of the total loss in the geometry sculpting stage, denoted as Cgs, as follows:
Lgs Aegh + Amask Adepth deepph + Anormal Cnormall + Abybrid Chyhrrdd
We set Augb 10000, Amask 5000, Adepth Anommal 0.1, Anybai = 1.
4.2 TEXTURE BOOSTING VIA BOOTSTRAPPED SCORE SAMPLING
The geometry sculpting stage prioritizes the learning of coherent and detailed geometry but leaves the texture blurry. This due to our reliance on 2D prior model that operates at coarse resolution, and the limited sharpness offered by the 3D-aware diffusion model. Additionally, texture issues such as over-smoothing and over-saturation arise from excessively large classifier-free guidance.
To augment the texture realism, we use variational score distillation (VSD) loss, detailed in Equa- tion 3. We switch to the Stable Diffusion model (Rombach et al., 2021) in this stage which offers high-resolution gradients. To promote realistic rendering, we exclusively optimize the mesh texture with the tetrahedral grid fixed. In this learning stage, we do not leverage the Zero-I-to-3 model as the 3D prior since it adversely impacts the texture quality. Nonetheless, the inconsistent textures may come back, resulting in bizarre 3D outcomes.
We observe that the multi-view renderings from the previous stage, despite some blurriness, exhibit good 3D consistency. One idea is to adapt a pretrained 2D diffusion model using these rendering results, enabling the model form a concept about the scene's surrounding views. In light of this, we finetune the diffusion model with the multi-view image renderings {x). using DreamBooth (Ruiz al., 2023). Specifically, we incorporate the text prompts containing unique identifier and the subject's 'sss class name (e.g., "A [V] astronaut" in Figure 2). During finetuning, the camera parameter of each view is introduced an additional condition. In practice, we train the DreamBooth with "augmented" image renderings. E, (xx). We introduce Gaussian noises, in an amount specified by the diffusion timestep I', to the multi-view renderings, i.e., Ze = Gp + TrE (ae, > hyperparameters), which are restored using the diffusion model. Choosing a large t' reveals high-frequency details in augmented images at the expense of fidelity to original renderings. The DreamBooth model trained on these renderings
guide texture refinement as a 3D prior.
Further, we propose to alternatively optimize the 3D scene to facilitate a bootstrapped optimization (Figure 2). Initially, the 3D mesh yields blurry multi-view renderings. We adopt large diffusion t' to augment their texture quality while introducing some 3D inconsistency. The DreamBooth model trained on these augmented renderings obtains unified 3D concept of the scene to guide texture

---

Published as a conference paper at ICLR 2024
Super Salyon tessile energy wone
handla datectrols outfte
weating the datte Spidermenn
Reference Ours Figure 3: Qualitative comparison with baselines. Our method generates sharper and more plausible details in both geometry and texture. Note that our method generates rich texture detail at novel views and eliminates multi-face Janus problems.
Magic123 Mt-D-It-3D
DreamFusion Magic3D
ProlificDreamer
refinement. As the 3D mesh reveals finer textures, we reduce the diffusion noises introduced to the image renderings, the DreamBooth model learns from more consistent renderings and better captures the image distribution faithful to evolving views. In this cyclic process, the 3D mesh and diffusion prior mutually improve a bootstrapped manner. Formally, we derive the optimization gradient using the following bootstrapped score distillation (BSD) loss:
dr 00
V VoLBsD(o, g(0) Er,,,e e (t) (DrcmBBooch (zr y, {.)x c) Elora (lora y, {x), c))
The notation {ã} denotes the augmented renderings generated during the t iteration of Dream- Booth. Contrary prior works (Poole al., 2022; Wang al., 2023b) that distill the score function from a fixed 21) model, our BSD loss learns from an evolving model which becomes increasingly 3D consistent by drawing feedback from the ongoing crafted 3D model. In our experiments, alternate the optimization twice, which suffices to produce consistent textures with rich details.
The total loss in the texture stage, labeled as Lrexture, be expressed below:
Liexure Xegh + AnSD
(9)
We set Angb 10000 and ÄBSD 1, values empirically suitable for most cases.
EXPERIMENTS
5.1 IMPLEMENTATION DETAILS
Architectural details. In the geometry sculpting stage, we use NeuS and textured 3D mesh repre- sentations. We employ Instant NGP (Müller et al., 2022), optimizing from a 64 to 384 resolution. For the textured mesh, we use DMTet at 128 grid and 512 rendering resolution.
Optimization. In mesh refinement, given plausible global geometric structure, skip the of a 3D prior in texture optimization. We employ random camera radius and FOV angle sampling, alternately rendering normal maps and RGB images enhance texture and geometry.
5.2 COMPARISONS WITH THE STATE OF THE ARTS
Baselines. We conduct a comparative analysis of our technique against five baseline methods. The first three text-to-3D methods: DreamFusion (Poole et al., 2022), Magic3D (Lin et al., 2023) and

---

Published as a conference paper at ICLR 2024
LEND
23
23
23
23
(a) w/o Zerol-to-3
(b) SDS
(c) VSD
(d) 1-round BSD (c) 2-round BSD
Figure 4: Ablation study of the effectiveness of 3D prior and our proposed BSD (Bootstrapped Score Distillation)..a) Geometry sculpting stage without prior. (b) Texture optimization with SDS loss. (c) VSD loss leads to richer texture detail but inconsistent texture. (d) BSD enhances
texture consistency with one DreamBooth round. (c) Two-round BSD adds more detail to the result.
Reference
Geometry Sculpting (Neus / DMTet)
Texture
"a DSLR photo of corgi wearing baguette, holding two hind lags"
Figure Continual improvement of geometry and texture quality through multiple stages.
ProlificDreamer (Wang et al., 2023b). We also compare our method against two image-to-3d meth- ods: Make-it-3D (Tang et al., 2023) and Magic123 (Qian al., 2023). For DreamFusion, Magic3D, Magic123, and ProlificDreamer, we utilize their implementations in the Threestudio library (Guo al., 2023) for comparison. For Make-it-3D, we use its official implementation.
Datasets. We establish test benehmark that in- cludes 300 images, which a mix real pictures and those produced by Stable Diffusion (Rombach et al., 2021) and Deep Floyd. Each image in this bench- mark comes with alpha mask for the foreground. a predicted depth map, and a text prompt. For real images, the text prompts sourced from an image caption model. We intend to make this test benchmark accessible to the public.
Table 1: Quantitative comparison against prior 2D-to-3D lifting meth- ods. The metrics are measured 300 generated samples.
CLIP f Contextual
PSNR t LPIPS Training Time
Make-it-3D
0.872
1.609
18.937
0.054
1..5h
Magic123
0.843
1.628
22.838
0.053
~~2.8h
DreamCraft3D
0.896
1.579
31.801
0.005
~2.3
Quantitative comparison. To generate compelling 3D content that resembles the input image and consistently conveys semantics from various perspectives, we compare our technique with estab- lished baselines using a quantitative analysis. Our evaluation employed four metrics: LPIPS (Zhang et al., 2018) and PSNR for fidelity measurement at the reference viewpoint; Contextual Dis- tance (Mechrez al., 2018 8) and CL.IP score (Radford al., 2021) to estimate pixel and semantic- level coherence respectively. Table indicates that approach significantly surpasses the base- lines maintaining both texture consistency and fidelity. Regarding training efficiency, our method requires approximately 2.3 hours for each case, which is comparable to other methods.
Qualitative comparison. Figure 3 compares our method with the baselines. All the text-to-3D methods suffer from multi-view consistency issues. While ProlificDreamer offers realistic textures, fails to form a plausible 3D object. Image-to-33 methods like Make-it-3D create quality frontal views but struggle with geometry. Magic123, enhanced by Zerol 1-o-3, fares better in geometry
8

---

Published as a conference paper at ICLR 2024
Rendering Views
DreamBooth Times
"3D Pixar Lionel Messi artfully kicking point-filled bottles"
Reference
Figure 6: Improved view consistency and texture fidelity along bootstrapping.
regularization, but both generate overly smoothed textures and geometric details. contrast, our BSD improves imagination diversity while maintaining semantic consistency.
5.3 ANALYSIS
The effect of 3D prior. In our paper we claim that the guidance offered by 3D prior enhances the generation of globally plausible geometry. To ascertain its impact, ablation study is conducted, where the 3D prior is deactivated. Figure 4 shows that without 3D prior, the character tends to display the Janus issue and irregular geometry, highlighting the erucial role of viewpoint-aware 3D prior in maintaining global shape consistency.
The effect of BSD. Figure 4 also presents ablation study encompassing three texture optimiza- tion techniques: (1) BSD, (2) VSD, and (3) Score Distillation Sampling (SDS) with the traditional stable diffusion. The application SDS has been observed to generate novel-view textures that excessively smooth and over-saturated. In contrast, while VSD using standard stable diffusion can produce realistic textures, yields a notably high inconsistency. In contrast, our proposed approach successfully generates textures that strike a balance between realism and consistency.
Visualization of multiple stages. Figure 5 visualizes the intermediate rendering results for each stage in our hierarchical pipeline. In the geometry sculpting stage, we convert NeuS to DM'Tet to improve high-resolution geometry details. However, the improvement in texture negligible. On the contrary, in the texture stage, we significantly improve the texture quality with BSD.
DoothBooth times. Figure shows multi-view datasets for DreamBooth. Initially, noise am- plifies detail richness but leads to inconsistent denoised images. As optimization proceeds, render- ings evolve towards greater consistency and photorealism, enhancing the DreamBootth-tiloreed input dataset's is quality.
CONCLUSION
We have presented DreamCraft3I), an innovative approach that advances the field of complex 3D asset generation. This work introduces meticulous geometry sculpting phase for producing plau- sible and coherent 3D geometries and novel Bootstrapped Score Distillation strategy. The latter, by distilling from optimizing 3D-aware diffusion prior and adapting to multi-view renderings of the instance being optimized, significantly improves texture quality and consistency. DreamCraft3D produces high- high-fieelity 3D assets with compelling texture details and multi-view consistency. We be- lieve this work represents an important step towards democratizing 3D content creation and shows great promise in future applications.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the Natural Science Foundation of China under Grants No. 61827805 and No. 62125107.
REFERENCES
Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, al. Gaudi: A. neural architect for immersive 3d scene generation. Advances in Neural Information Processing Systems, 35:25102-25116, 2022.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3799-5809 2021.
Eric Chan, Connor Z. Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-awer 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 16123-16133, 2022.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 152.v:151..03012, 2015.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.1387. 2023.
Yen-Chi Cheng. Hsin- Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang Yan Gui. Sd-
fusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 4456-4465, 2023.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142-13153, 2023.
Congyue Deng, Chiyu Jiang, Charles Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, al. Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10637-20647, 2023.
Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-tast mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10786-10796, 2021.
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems, 35:31841-31854 2022.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696-10706, 2022.
Yuan-Chen Guo, Ying-Tre Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia- Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang. threestudio: A unified framework for 3d content generation. https: ///pt//tthh//// con/ threestudic o, 2023.
Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.05533 2023.
10

---

Published as a conference paper at ICLR 2024
Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. ACM Transactions on Graphics (TOG), 41(4):1-19 2022
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lim: Large reconstruction model for single image to 3d. arXiv preprint (tti:23111.44000 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 106.09685 2021.
Nan Huang, Ting Zhang, Yuhui Yuan, Dong Chen, and Shanghang Zhang. itstomize-it-3d: High- quality 3d creation from single image using subject-specific knowledge prior. arXiv preprini arXiv:2312.11555, 2023a.
Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi, Zheng-Jun Zha, and Lei Zhang. Dream- time: An improved optimization strategy for text-to-3d content creation. arXiv preprint arXiv:2306. 12422 2023b.
Zehuan Huang, Hao Wen, Junting Dong, Yaohui Wang, Yangguang Li, Xinyuan Chen, Yan-Pei Cao, Ding Liang, Yu Qiao, Bo Dai, et al. Epidiff: Enhancing multi-view synthesis via localized epipolar-constraiee diffusion. arXiv preprint arXiv:2312.06725, 2023c.
Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. 2022.
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463, 2023.
Han-Hung Lee and Angel X Chang. Understanding pure clip guidance for voxel grid nerf models. arXiv preprint arXiv:2209, 15/92, 2022.
Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdff Text-to-shape via voxelized diffu- sion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12642-12655 2023.
-suan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding. arXiv preprint urXiv:2305. 10764, 2023a.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, et al. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. arXiv preprint arXhv:2306.16928 2023b.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 1 Zero-shot one image to 3d object. arXiv preprint arXiv:2303.11332, 2023c.
Ying-Tian Liu, Guan Luo, Heyi Sun, Wei Yin, Yuan-Chen Guo, and Song-Hai Zhang. Pi3d: Efficient text-to-3d generation with pseudo-image diffusion. arXiv preprint arXiv:2312.09069, 2023d.
Zexiang Liu, Yangguang Li, Youtian Lin, Xin Yu, Sida Peng, Yan-Pei Cao, Xiaojuan Qi, Xiaoshui Huang. Ding Liang, and Wanli Ouyang. Unidream: Unifying diffusion priors for relightable text-to-3d generation. arXiv preprint arXiv:2312.0877 2023e.
Xiaoxiao Long. Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310. 15008, 2023.

---

Published as a conference paper at ICLR 2024
Roey Mechrez, Itamar Talmi, and Lihi Zelnik-Manor. The contextual loss for image transformation with non-aligned data. In Proceedings of the European conference on computer vision (ECCV), pp. 768-783, 2018.
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen- -r. Latent-nerf for shape- suape-guided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12663-12673, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 1599),99, 2021.
Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 306-315, 2022.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM Trans. Graph, 41(4):102:111022 July 2022. doi: 10.114553522233355177. URL https: /ttps:://oi... org/10.../...o 3530127.
Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono. Linqi Zhou, and Paul Guerrero. 3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint (rXii::212.00842. 2022.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 2022.
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, al. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306. 17843, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485 5551, 2020.
Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Natanicl Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al. Dreambooth3d: Subject-driven text-to-3d generation. arXiv preprint arXiv:2303.13500 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. resolution image synthesis with latent diffusion models, 2021.
High-
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
12

---

Published as a conference paper at ICLR 2024
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- fion Processing Systems, 35:36479-36494, 2022.
Aditya Sanghi, Hang Chu, Joseph Lambourne, Ye Wang, Chin- Yi Cheng, Marco Fumero, and Ka- mal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18603-18613, 2022.
Kyle Sargent, Jing Yu Koh, Han Zhang, Huiwen Chang, Charles Herrmann, Pratul Srinivasan, Jiajun Wu, and Deqing Sun. Vq3d: Learning 3d-aware generative model on imagenet. arXiv preprint arXiv:2302.06833 2023.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: bybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087-6100 -101, 2021.
Alex Shonenkov, Misha Konstantinov, Daria Bakshandaeva, Christoph Schuhmann, Ksenia Ivanova, and Nadiia Klokova, DeepFloyd IF: modular cascaded diffusion model. https:///átthn com/deep-aa www.e/loood/////eeeeeee deeeeeede.....p.. 2023.
Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin-Ying Lee, Peter Wonka, and Sergey Tulyakov. 3d generation imagenet. arXiv preprint arXiv:2303.01416, 2023.
Jingxiang Sun, Xuan Wang. Yichun Shi, Lizhen Wang. Jue Wang, and Yebin Liu. Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis. ACM Trans. Graph. 41(6), nov 2022. ISSN 0730-0301. doi: 10. 0..1145355045443555500 URL https::/p//// //o.//o...////////// org /rg/10. 1145/3550454 3555506.
Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, Yong Zhang, Hongwen Zhang, and Yebin Liu. Next3d: Generative neural texture rasterization for 3d-aware head avatars. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattrn Recognition (CVPR), pp. 20991-21002, June 2023.
Jiaxiang Tang- Stable-dreamfusion::
Text-to-3d
with stable-diffusion, 2022.
Junshu Tang, Bo Zhang, Binxin Yang, Ting Zhang, Dong Chen, Lizhuang Ma, and Fang Wen. Explicitly controllable 3d-aware portrait generation. arXiv preprint arXiv:2209.05444 2022.
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. arXiv preprint arXhv:2303.14184. 2023.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond Yeh, and Greg Shakhnarovich. Score ja- cobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:2212.00774.00 2022.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint 1ee::2106.16880 2021.
Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pp. 4563-4573, 2023a.
Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High- figh-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint 23355:2305. 13, 2023b.
13

---

Published as a conference paper at ICLR 2024
Daniel Watson, William Chan, Ricardo Martin-Brualle Jonathan Ho, Andrea Tagliasacchi, and Mo- hammad Norouzi. Novel view synthesis with diffusion models. arXiv preprint arXiv:2210.04628, 2022.
Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang. Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang. Jiaqi Wang, Chen Qian, et al. Omniobject3d: Vocab-voc 3d object dataset for realistic percep- tion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023.
Jianfeng Xiang, Jiaolong Yang. Binbin Huang, and Xin Tong. 3d-aware image generation using 2d diffusion models. arXiv preprint arXiv: 17905, 2023.
Chulin Xie, Chuxin Wang, Bo Zhang, Hao Yang, Dong Chen, and Fang Wen. Style-based point generator with adversarial rendering for point cloud completion. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 4619-4628, 2021.
Xingguang Yan, Liqiang Lin, Niloy Mitra, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Shapeformer: Transformer-based shape completion via sparse representation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 6239-6249 2022.
Wang Yu, Xuelin Qian, Jingyang Huo, Tiejun Huang, Bo Zhao, and Yanwei Fu. Pushing the limits of 3d shape generation at scale. arXiv preprint arXiv:2306.1 11510, 2023.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and
Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint arXiv:2210.06972 06978, 2022.
Biao Zhang, Matthias Nießner, and Peter Wonka. 3dilg: Irregular latent grids for 3d generative modeling. In NeurlPS, 2022.
Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape rep- resentation for neural fields and generative diffusion models. arXiv preprint arXiv:2301.1144
2023a.
Huichao Zhang, Bowen Chen, Hao Yang, Liao Qu, Xu Wang. Li Chen, Chao Long, Feida Zhu, Kang Du, and Min Zheng. Avatarverse: High-quality & stable 3d avatar creation from text and pose. arXiv preprint arXiv:2308. 03610, 2023b.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.
Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint 27X1v:2312.09147, 2023.
A APPENDIX
A.1 IMPLEMENTATION DETAILS
Algorithms for Bootstrapped Score Distillation. We provide summarized algorithm of boot- strapped score distillation in Algorithm 1. The "Bootstrapped Score Distillation" algorithm starts by initializing n
in our case) meshes and pretrained text-to-image diffusion model DDcamBoothh parameterized by Φ. The algorithm then enters an itcrative loop: in each iteration, it renders the mesh to obtain multi-view images x, augments these images with Gaussian noises to form ze 0 + dyc, and fine-tunes the pretrained diffusion model DreamBBoot based on these aug- mented image renderings. Within each iteration, there is an inner loop running for n steps, where random mesh and camera pose sampled, and 2D image is rendered from the chosen pose. Then, updates are performed on 9 and ó using gradients calculated from the difference between the pretrained score function and the predicted score function, and from the La norm between the pre- dicted score and real noise, respectively. These iterations continue until convergence, and the final refined mesh(s) are returned.
14

---

Published as a conference paper at ICLR 2024
Structure-aware latent regularization. To maintain the high-quality output produced by BSD while reducing noise and inconsistencies, we further incorporate a control net-guided inpainting diffusion model that regularizes the generated textures. Specifically, for rendered image from an arbitrary viewpoint, the visible section under the reference view is initially computed. This invariant portion during the generation process allows inpainting model fill in the remaining segments. As these remaining parts adhere to geometric constraints, we integrate geometric normal information through control net. Ultimately, this method permits us enforce view consistency and generate realistic results using ControlNet-guude high-quality generation output, we avoid utilizing this image directly as a loss against the rendered image. Instead, we subtly introduce it by constraining the norm of the latent variables:
inpainting diffusion model. To preserve the
Lreg (0.(()))))))))))))))))))))))(( = E(I
(Treg))l2)
(10)
Architectural details. In the NeuS approach (Wang al., 2021), we employ single-layer Multi- Layer Perceptron (MLP) with 32 hidden units to simultaneously predict RGB color, volume den- sity, and normal. The inputs to this MLP are the concatenated feature vectors derived from multi- resolution hash encoding sampled with trilinear interpolation. To sparsify the Instant NGP repre- sentation, we implement density-bane based pruning every 10 iterations within an octree structure, as suggested by Magic3D (Lin et al., 2023). In our experiments, we radius 2. For the density prediction, we utilize the softplus activation function and, following the approach of Poole et al. 2022, include initial spatial density bias to encourage optimization in favor of the object-centric neural field.
bounding sphere with
Camera and light augmentations. We follow Magic3D to add random augmentations to the cam- era and light sampling for rendering the shaded images. Differently, we sample the point light location such that the angular distance from the random camera center location (w.r.t. the origin) is sampled with a random point light distance Team ~ (7.5, 10), and (b) we freeze the material aug- mentation unlike Dreamfusion and Magic3D, as we found it is bad for training convergence (c) In the coarse stage, we propose fixed-random mixed camera pose strategy. Specifically, following the common practice of the current text-to-3D methods, random camera view sampling benefits scene optimization. However, Zerol-to-3 needs fixed camera intrinsic parameters. Therefore, we let half of the GPUs sample the camera distance from (3.22, 3.5), and the Field-of- from ((10, 20), while the left GPUs are fixed to the default camera intrinsic.
Time annealing. At the beginning of the geometry sculpting stage, We utilize simple two-stage annealing of time step in the score distillation objective. For the first several iterations we sample time steps | ~ (0.77, 0.85) and then anneal into ~ U(0.2, 0.50). We refer the readers to Prolific- Dreamer (Wang al., 2023b). For the left iterations, we fix time steps to (0.2,, 0.50). We also utilize a simple two-stage time annealing for the multi-view dataset generation, that is, for the first updating step, select time step i 0.5 for all rendered images and then anneal it into & = 0.1 along the later updating steps.
Progressive training. We linearly increase the sampling range of camera positions with eleva- tion angle (@em) from 0' to [-10%, 4,5), and azimuth angle (9eam) from 0 to [-180, 180"]. The progress length is set 200 iterations.
A.2 DIVERSE RESULTS
Prior studies frequently yield models of limited diversity with disproportionately smooth textures. Our approach superior text-to-3D generation initially translates the text prompt into a reference image via 2D diffusion before implementing our proprietary image-based 3D creation methodology. Figure demonstrates the proficiency of our method generating an array of diverse models from single text prompt, all characterized by their remarkable quality.
A.3 LIMITATIONS
Our approach occasionally incorporates frontal-view geometric details into texture, as depicted in Figure 8, due to depth ambiguity and inaccuracies in the depth prior. Furthermore, we do not pressly segregate material and lighting from the 2D reference image, an aspect deferred for future exploration.
15

---

Published as a conference paper at ICLR 2024
Algorithm 1 Boootstraped Score Distillation
Input: Number of particles n (2 1). Pretrained text-to-image diffusion model eDrcamBooth Learning rate 7/1 and 122 for 3D structures and diffusion model parameters, respectively. A prompt Number of images m and Camera poses {cr (c())) mm for the multi-view datasets.
1: initialize meshes {((*) Jim= noise prediction model Es parameterized by d.
2: while not converged do
3: Render the mesh to get multi-view images cr)
4: Augment image renderings with Gaussian noises: Dy = o. + ove.
5: Finetune DreamBooth on augmented image renderings Vr re (x))
6: for in 7'' steps do
7: Randomly sample & ~ 1E= and a camera pose c.
Render the 3D structure 0 pose e get 2D image an g(0, c). 8:
9: 0 0 - 7 B,,,, (w((( (eDramBBooth (Xt, (Xe, c, v) ea 1, C, c) (0,cc)
10:
n2 (xE,,elle t, c, -112-
11: end for
12: end while
13: return
"a DSLR photo of bagel filled with cream cheese and lox"
Figure 7: DreamCraft3D skillfully generates an assortment of visually compelling 3D models when provided with textual description.
A.4 ADDITIONAL RESULTS FOR ABLATION STUDIES
The effectiveness of BSD. We conduct quantitative and qualitative experiments to assess the ef- fectiveness of the proposed BSD. Table 2 summarizes the evaluation of 3D generation quality on four metrics: LPIPS (Zhang et al., 2018) and PSNR for fidelity measurement at the reference view; Contextual Distance (Mechrez et al., 2018) and CLIP score (Radford et al., 2021) for pixel and semantic-level coherence respectively. The use of SDS loss in texture refinement results in view- coherent texture but fails restore high-quality textural details. In contrast, VSD significantly improves the texture generation quality but tends to degrade view consistency on texture. How- ever, when employing BSD, we obtain sharp and view-consistent textured meshes. Furthermore, the results improve with each round of DreamBooth, indicating the effectiveness of the bootstrapping.
Figure 10 illustrates the efficacy of BSD. Consistent with Figure 4, we can conclude that BSD uniquely enables the generation of high-quality texture while preserving style consistency with the reference image.
The effectiveness of designs in the geometry sculpting stage. We conduct supplementary ablation studies on various design aspects within the geometry sculpting stage, which include loss functions, training strategies, and model choices. For these tests, we report quantitative and qualitative results for NeuS instead of DMTet to isolate the impact of changing 3D representations.
As depicted in Table disabling LRGB significantly drops reconstruction fidelity due lack of color reinforcement. Training without Lmast results in less but notable fidelity degradation. Also, omitting 3D prior (Liu et al., 2023c) during training negatively affects global semantic coherence.
16

---

Published as a conference paper at ICLR 2024
Figure 8: Failure case. Our method learns incorrect geometry for elephant nose.
Table 3 also reveals that omitting either progressive training diffusion timestep annealing during the geometry sculpting stage leads to a decrease in CLIP and contextual scores, indicating their crucial roles in early geometry modeling optimization.
For the diffusion guidance, replacing DeepFloyd with Stable Diffusion 2.1 (Rombach et al., 2022) in ablation study leads to decreased generation quality, potentially due DeepFloyd's superior view-awareness from using T5-XXL. (Raffel et al., 2020) as a text encoder. Similarly, DreamFu- sion (Poole et al., 2022), which also employs the T5-XXL encoder, exhibits fewer Janus issues than its reimplementation with Stable Diffusion (Tang, 2022).
Figure 11 visualizes the ablation of our design choices. Obviously, the default setting yields the highest fidelity and imaginative power in generating 3D contents.
Training time for individual stages. In addition the efficiency comparison with baselines in Table 1, we also provide training times for individual stages, shown in Table 4. Averaged on 300 objects our test bench, the total training time about 2.3 hours, of which 1.5 hours are dedicated to the geometry sculpting stage and 0.8 hours to the texture stage. The inference time is approximately 0.05 seconds per frame at resolution of 512. We conducted our timing tests using 8 A100 GPUs for training and a single A100 GPU for inference.
representation choice. We choose NeuS (Wang et al., 2021) instead of NeRF (Mildenhall et al., 2021) as the 3D representation in the early geometry sculpting stage. As shown in Figure 12, NeRF learns a volume density field, from which it is difficult to extract high-quality surface, e.g. hole artifacts, for DMTet initialization. On the contrary, NeuS Iears neural SDF representation, enabling subtle surface extraction. Moreover, the DeepFloyd base model offers gradient on a coarse resolution (64x64), which enforces coherent 3D geometry rather than introducing high-frequency signals that distract the geometry sculpting.
A.5 ROBUSTNESS TO OUT-OF-DOMAIN REFERENCE IMAGES
Our method adopts a 3D-aware diffusion prior, Zero123 (Liu al., 2023c) for guiding optimiza- tion in the geometry sculpting stage. Zerol23 is trained on large synthetic 3D dataset, Objaverse (Deitke et al., 2023). In order to prove the robustness of our method on images far from the data distribution of Objaverse, we select 4 reference images from showcases in our paper and use them to retrieve 3D objects within Objaverse. The retrieval algorithm is based on OpenShape (Liu et al., 2023a). As shown Figure 13, the retrieval results have a significant domain gap with the reference images, while our results show promising 3D generations for these cases. For example, photo-realistic back jersey details from the reference image though there is no similar 3D data in Objaverse.
3.73%
3.11%e
1.80%
DreamCreft30
Magic1 23 Prolificdreamer Magi:3D
91.3%
generate "Messi" with
Figure User study.
17

---

Published as a conference paper at ICLR 2024
Table 2: Quantitative ablation study of the effectiveness of proposed BSD (Bootstrapped Score Distillation).
CLIP
Contextual PSNR
LPIPS
SDS
0.775
1.894
25.258
0.036
VSD
0.792
1.833
31.426
0.005
One-round BSD
0.863
1.647
30.431
0.008
Two-round BSD
0.896
1.579
31.801
0.005
Table Quantitative ablation study of the effectiveness of design choices in the geometry sculpting stage.
CLIP f Contextual 4 PSNR f LPIPS ي
Default setting
0.745
1,936
20.836
0.031
w/o LRGB
0.619
2.212
10.562
0.095
w/o fmusk 0.736
1.949
19.307
0.057
w/o prior w/o time annealing
0.679
2.105
20.252
0.092
0.737
1.943
20.120
0,035
w/o progressive training
0.721
1.962
20.317
0.029
Stable diffusion
0.669
2.154
18.182
0.099
To substantiate the robustness and quality of our proposed model, we executed a user study em- ploying 15 distinct pairs of prompts and images. Each participant was provided with four free-view rendering videos alongside their corresponding textual inputs and asked to choose their top preferred 3D model. The study gathered 480 responses from total 32 participants, the analysis of which is depicted in Figure 9. On an average basis, our model was favored by 92% of users over alternative models, outperforming the baselines large margin. This result provides compelling evidence of the resilience and superior quality inherent to our proposed method.
A.6 ADDITIONAL QUALITATIVE RESULTS
Figure 14-Figure 17 provides more results produced by DreamCraft3D. Our method is able to produce photo-realistic 3D assets with compelling textural details. Moreover, our method shows significantly improved 3D consistency. Please find the video results the supplementary video.
18

---

Published as a conference paper at ICLR 2024
Table Training and inference time for our method.
Total training time Goometry sculpting Stage Texture Stage Inference
DreamCraft3D
2.3h
1.5h
0.8h
0.05s
(4) Reference
SDS
(c) VSD
(d) One-round BSD
(e) Two-round BSD
Figure 10: Ablation study of the effectiveness of the proposed BSD (Bootstrapped Score Distilla- tion). From left to right are: (a) reference image. (b) texture optimization with SDS loss. (c) The use of VSD loss results in richer texture detail but comes with texture inconsistency. (d) Application of BSD improves texture consistency with one round of DreamBooth. (e) Applying two rounds of BSD adds more detail to the final result.
Referenee
Definitetttin
wio Eaus
Wil Gyaok
wio 3D
wio tatte annealing
wio progressive training
Stable ditfision
Figure 11: Ablation studies of design choices in the geometry sculpting stage.
19

---

Published as a conference paper at ICLR 2024
NeRF
DMTet
NeuS
DMTet
Super saiyan goku, Full-body portrait
Figure 12: Ablation studies of the representation choice. Compared with NeRF, NeuS models clear surfaces and enables extracting higher-quality surfaces for DMTet initialization.
Refoeuue
Retiivall
Our Generation
Refevence
Retieval
Out Genention
Figure 13: Comparison between the object retrievals within Objaverse by the test bench images and our generations. Our method is robust to the out-of-domain reference images.
20

---

Published as a conference paper at ICLR 2024
Mictlantecuhtli
brightly colored mushroom growing on log
23
23
LAKERS
23
Lebron James wearing Lakers jersey, holding basketball
President Obama
A DSLR photo of delicious chocolate brownie dessert with ice cream the side
Wes Anderson style Red Panda, reading book, super cute, highly detailed and colored
Figure 14: Additional results of DreamCraft3D.
21

---

Published as a conference paper at ICLR 2024
Super Saiyan Goku unleashes massive energy wave while standing on top mountain
Portrait painting of batman with black leather armor, ultra realistic
Mech robot with large weapons on top with hexagonal bases
Super Mario
3d stylized game little building
3d render of statue of astronaut
Figure 15: Additional results of DreamCraft3D.
22

---

Published as a conference paper at ICLR 2024
A blue jay standing on large basket of rainbow macarons
A delicious hamburger
A DSLR photo of a plate of fried chicken and waffles with maple syrup on them
An ice cream cone
Nike Lebron 10 "Cork" Sneakers
Nike "Air Jordan 1" Sneakers
Figure 16: Additional results of DreamCraft3I..
23

---

Published as a conference paper at ICLR 2024
Tiger dressed as a nurse
Baby yoda the style of Mormookiee
Teddy with sunglasses and swimming trunks
Realistic photo of pistachio mousse cake
DSLR photo of bear dressed in medieval armor Neymar 3d character cartoon disney pixar render
Figure 17: Additional results of DreamCraft3ID.
24


Reviews:
[{"review_id": "HpVgAS6YII", "reviewer": "Reviewer_c9Ad", "summary": "This paper studies the task of text-to-3D generation, incorporating a series of enhancements that yield highly promising results. The introduced framework DreamCraft3D generates 3D assets with remarkable fidelity and minimal artifacts, marking a great advancement in the field. Key innovations of DreamCraft3D encompass initial reference image generation, a geometry sculpting stage, and a subsequent texture boosting stage.\n\nThrough an extensive series of experiments, DreamCraft3D emerges as a formidable contender, outperforming both optimization-based methods such as DreamFusion and single-image-based 3D reconstruction techniques exemplified by Magic123.\n\n----- After rebuttal ------\nThanks for providing more experiments and most of my concerns/doubts are addressed. I have raised my rating to 8-good paper.  I suggest the authors to include the extra ablation studies into the main paper, which can be informative to many readers.", "questions": "1. **training/inference time**: what's the training time of the entire pipeline and what's the time for each individual stage? Meanwhile, how \n1. **Training and Inference Time**: Could you provide insights into the training time for the entire pipeline and the time required for each individual stage? Additionally, how long does it take to generate a new 3D model from scratch, and could you provide a breakdown of the time allocation for this process?\n\n2. **Robustness of the System**: Given the numerous modules and training stages, how robust is the system? Are the same set of parameters applicable to different methods, and if not, what are the characteristics of failure cases? Have you observed any convergence issues in any of the stages?\n\n3. **Choice of Deepfloyd IF**: What is the rationale for not using SD in the first stage and opting for Deepfloyd IF instead? Could you elaborate on the advantages of Deepfloyd IF in this context?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper demonstrates several notable strengths, as outlined below:\n\n1. **Impressive Qualitative Results**: Unlike existing methods, the generated 3D assets in this study exhibit great fidelity and significantly fewer artifacts. Furthermore, the Janus problem is substantially mitigated, marking a substantial improvement.\n\n2. **Comprehensive Comparative Analysis**: The paper conducts comparisons with both optimization-based methods and single-image-based 3D reconstruction techniques, enhancing the credibility of the experimental findings.\n\n3. **Effective and Reasonable Solutions**: The proposed solutions and submodules are not only reasonable in design but also demonstrate practical effectiveness.\n\n4. **Quantitative Evaluation and Benchmarking**: Despite the inherent challenges in evaluating text-to-3D methods, this paper makes a commendable effort to perform quantitative comparisons and establishes a new benchmark, contributing to the field's progress.\n\n5. **Clarity in Communication**: The writing in this paper is mostly clear and easy to follow.", "weaknesses": "Even though the final results are encouraging, there are some weaknesses that need to be addressed:\n\n1. **Clarity of Approach Section**: The introduction of multiple stages and new sub-modules (loss functions, optimization stages) makes the presentation in the approach section less clear. It would be beneficial to include an overview section with a clear definition of the total loss and a pseudo-algorithm summarizing the steps in different stages.\n\n2. **Technical Contribution Clarification**: The major technical contributions are not clearly defined. It's essential to clarify which submodules or loss functions represent the most significant changes and contribute the most to the final performance. This information is crucial for a deeper understanding of the work. Overall, the proposed framework is a bit complicated and contains many modules and stages. \n\n3. **Incomplete Validation**: Some modules are not thoroughly validated in the experiments. For instance, the importance and effects of the two losses in Eq.4, the L_RGB, and L_mask losses should be elaborated upon. It's also important to explain the advantages of using NeuS compared to NeRF-variants and provide validation results. Additionally, details on the implementation of \"Progressive view training\" should be provided.\n\n4. **Prompt Engineering Clarity**: The paper needs to address discrepancies in prompt engineering. As shown in Fig.2, the prompt for generating the reference image is \"an astronaut in a sand beach,\" while for dreambooth, it's \"an [v] astronaut.\" These differences need clarification to ensure consistency and reproducibility.\n\n**Miscellaneous:**\n\n- **Figure Order**: The order of figures is confusing. Fig.4 is mentioned later in the text than Fig.5/6 but appears earlier. Consider reordering them for coherence.\n\n- **Fig.6 Text Descriptions**: Fig.6 is missing text descriptions of the corresponding stages. It's unclear which figure corresponds to what stage, making it difficult to follow.\n\n- **Fig.5 Ablation Study**: The ablation study in Fig.5 could be expanded. Since text-to-3D is challenging to evaluate, a single example may not provide enough information. More comprehensive insights can help to strengthen the research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rpgmdSExYN", "reviewer": "Reviewer_vdnk", "summary": "The authors present a framework for text-to-3D generation. This is achieved by first synthesizing a single view image from a text point and using a view conditioned diffusion model to guide the geometry generation process. In particular, the framework uses a combination of SDS losses from a view-conditioned diffusion model and a text conditioned diffusion model. Further, the renderings from the 3D representation are used as additional training data to train a personalization model to improve texture quality. State of the art results are demonstrated on text-to-3D creations compared to several recent baselines.", "questions": "What is the effect of time step annealing? and progressive viewpoint training?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Result Quality** : The quality of the generated assets are very impressive. Particularly, in examples where the back side of the object hallucinates necessary details.\n2. **Novelty** : The proposed approach is reasonably novel as it proposes a combination of 2D SDS and 3D aware SDS losses for better geometric reconstruction. The paper also compares against contemporary work with similar ideas (Magic123) and shows state of the art results against the same. \n2. **Related work**: An adequate treatment of the related work has been provided. Efforts have been made to reference and compare against contemporary unpublished work that also have compelling results. \n3. **Reproducibility**: All the implementation is based off of available open source code bases, make it easy to reproduce. Network and training details have been provided to further aid in reproducibility. \n4. **Progressive view training**: Is a simple and elegant idea to make sure that the generated geometry is not flat.\n5. **Benchmark dataset**: This work shares a evaluation benchmark of 300 prompt image pairs, which can be used to evaluate future work in text-to-3D synthesis.  \n6. **Structure aware latent regularization**: Is another interesting strategy for making sure already generated texture information is preserved. This section would benefit from shifting it from the appendix to the main section.", "weaknesses": "1. **Multi component-Combination** : Although reasonably novel, the framework uses several different diffusion networks for different components. Particularly, Deep-IF for initial image and base 3D geometry generation, SD for personalization and Zero-123 for viewpoint guided SDS. The final framework appears to be a combination of Magic123, Zero1-to-3, ProlificDreamer and Fantasia3D[1]\n1. **Claims** : The authors claim that personalizing the diffusion model based on the multiview renderings helps improve texture, however, this seems counter intuitive, since the initial texture generated from the 3D representation are expected to be worse than the high quality texture. Providing additional insights about this would be helpful.\n2. **Writing**: The manuscript contains certain syntactic and language errors (such as some of the nits indicated below) and would benefit from a thorough proof reading pass of both the main paper and the appendix.\n3. **Ablations**: Qualitative ablations are provided for some of the components. But adding some quantitative ablations that show change in quality would be helpful. Particularly, metrics for 3D consistency for number of BSD steps, texture quality with and without BSD and effect of progressive training, timestep annealing and choice of diffusion model used for SDS (SD vs DeepFloyd).  \n4. **Training and inference time costs**: A comparison of training and inference time cost and memory footprint of the proposed approach compared to the baselines is also important and would provide some insights about the inference time trade off and memory required to achieve this quality.\n\n\nNits:  \nSec 4. \"in the next...\" -> \"in the next section?\" or \"next\".  \nSec 4.2 \"multiview rendering from last stage..\" -> \"from the previous stage?\"  \n  \n \n[1] Chen et al. Fantasia3D, ICCV23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zP5MjED4T7", "reviewer": "Reviewer_GYis", "summary": "The paper proposes an image-conditioned 3D generative model and focuses on improving view consistency upon current techniques. It proposes to use a view-conditioned 3D prior, and various techniques to improve the texture fidelity. Results show empirical benefits of the proposed method compared to prior arts.", "questions": "* Do all examples shown in the paper share the same hyperparameter configuration?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "* Qualitative results selected in the paper show a large margin of advantages of the proposed method compared to the baselines. \n* The paper focuses on addressing the view consistency problem, and the examples shown in the paper provide strong evidence for this claim.", "weaknesses": "* The paper combines several existing techniques, including Zero-1-to-3, DreamBooth, VSD, and a bag of tricks including progressive view sampling and a transition between different 3D representations. The novelty of this paper is limited. \n* The pipeline depends on Zero-1-to-3 to provide a good initial shape. For scenes that Zero-1-to-3 fails on, e.g. scenes that differ greatly from the synthetic training distribution of Zero-1-to-3, it's unclear if the proposed method can still generate reasonable results. \n* Multiple loss or gradient terms are introduced in Section 4.1, but the relative weights of these terms are not described.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9E5t6g03LH", "reviewer": "Reviewer_NmnF", "summary": "This paper proposes a hierarchical way for 3D content generation from text inputs, following the line of dreamfusion.\nInstead of using 2D diffusion only, it combines zero123 to give 3D priors. \nIt also proposes a bootstrapped score sampling method, which finetunes the stable diffusion using dreamboth during the distillation  process. \nCombining carefully-tuned parameters, this paper achieves impressive results of text-to-3D.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The results of this paper is impressive. The improvement seems significant compared to previous methods.\n2. This paper combines many tricks in a convincing way, and clearly shows the effectiveness of each trick. \n3. The paper is well written.", "weaknesses": "1. The running time of methods. This method involves several stages an looks quite complicated. I am curious about how long does it take to finish the whole process? And how does it compare to other baselines?\n2. It would be more clear if the difference between the proposed bootstrapped diffusion prior and LORA updates in ProlificDreamer is elaborated, since they both update the diffusion during the distillation.  What's intuitive difference and actual different in implementation? \n3. I understand it is hard to evaluate generation task, but more details about metrics used in Table 1 would be convincing. All CLIP, Contextual ,PSNR and LPIPS are image-level evaluation, and it is not clear how to convert them to 3D level. PSNR and LPIPS are measured on reference images, which is understandable. But how is the CLIP and Contextual calculated? How many views are rendered from each scene for evaluation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)