Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation
Authors: Junyoung Seo
Wooseok Jang
Min-Seop Kwak
Hyeonsu Kim
Jaehoon Ko
Junho Kim
Jin-Hwa Kim
Jiyoung Lee
Seungryong Kim
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion models
NeRF
3D synthesis
Primary area: generative models

Abstract: Text-to-3D generation has shown rapid progress in recent days with the advent of score distillation sampling (SDS), a methodology of using pretrained text-to-2D diffusion models to optimize a neural radiance field (NeRF) in a zero-shot setting. However, the lack of 3D awareness in the 2D diffusion model often destabilizes previous methods from generating a plausible 3D scene. To address this issue, we propose 3DFuse, a novel framework that incorporates 3D awareness into the pretrained 2D diffusion model, enhancing the robustness and 3D consistency of score distillation-based methods. Specifically, we introduce a consistency injection module which constructs a 3D point cloud from the text prompt and utilizes its projected depth map at given view as a condition for the diffusion model. The 2D diffusion model, through its generative capability, robustly infers dense structure from the sparse point cloud depth map and generates a geometrically consistent and coherent 3D scene. We also introduce a new technique called semantic coding that reduces semantic ambiguity of the text prompt for improved results. Our method can be easily adapted to various text-to-3D baselines, and we experimentally demonstrate how our method notably enhances the 3D consistency of generated scenes in comparison to previous baselines, achieving state-of-the-art performance in geometric robustness and fidelity.

Published as a conference paper at ICLR 2024
LET 2D DIFFUSION MODEL KNOW 3D-CONSISTENCY FOR ROBUST TEXT-TO-3D GENERATION
Junyoung Seo
Wooseok Jang"1
Min-Seop Kwak Hyeonsu Kim
Jaehoon Ko
Junho Kim? Jin-Hwa Kim
Jiyoung Lee
Seungryong Kim
Korea University
NAVER AI Lab
3 AI Institute of Scoul National University
(se780, jws1997, mskwak01, hyecnsu0305, k fjhkim.ail jinhwa. kim, lee. i 1 |Jnavercorppcom seungryeng kim@korea.ac.ks
j69 (etp00000kkeaee.oo ac.
ABSTRACT
Text-to-3D generation has shown rapid progress in recent days with the advent of score distillation sampling (SDS), a methodology of using pretrained text-to-2D diffusion models to optimize a neural radiance field (NeRF) for zero-shot setting. However, the lack of 3D awareness the 2D diffusion model often destabilizes previous methods from generating a plausible 3D scene. To address this issue, we propose 3DFuse, a novel framework that incorporates 3D awareness into the pretrained 2D diffusion model, enhancing the robustness and 3D consistency of score distillation-bat methods. Specifically, we introduce a consistency injec- tion module that constructs 3D point cloud from the image generated from the text prompt and utilizes its projected depth map at a given view as a condition for the 2D diffusion model. The diffusion model, through its generative capability. robustly infers dense structure from the sparse point cloud depth map and gener- ates a geometrically consistent and coherent 3D scene. We also introduce a new technique called semantic coding that reduces the semantic ambiguity of the text prompt for improved results. Our method can be easily adapted to various text-to- 3D baselines. We experimentally demonstrate how our method notably enhances the 3D consistency of generated scenes compared to previous baselines, achiev- ing state-of-the performance in geometric robustness and fidelity. The project page is available at /attpa: //kuu///uuuue cvlab. github. io/ 3DFuse/
INTRODUCTION
Text-to-3D 3D generation, the task of generating 3D content through given text prompts (Sanghi al., 2022 Liu et al., 2022 Jain et al. 2022). has seen a rapid growth surge thanks to the advancements diffusion models (Ho et al. 2020 Rombacc et al., 2022). This innovation allows people to easily create 3D content without dealing with professional modeling tools, gaining significant attention across a wide range of industries, such as gaming, graphics, VR/AR, and animation.
One of the most popular methodologies currently used for text-to-3D generation tasks is score dis- tillation sampling (SDS) (Poole et al., 2022), which employs the gradients of 2D diffusion models to optimize 3D representation, i.e.. Neural radiance field (NeRF) (Mildenhall al. 2020). This approach has gained prominence due to its leveraging the generation capability of pretrained 2D diffusion models create expressive and detailed 3D scenes. However, various works (Wang al., 2023 Shi et al. 2023) show that 3D scenes generated by SDS often display distortions and artifacts depending text prompts. One notable failure case 3D inconsistency problem (dubbed the "Janus problem" (Wang et al.) 2023 Shi et al.) 2023)), where a frontal feature (such as a frontal face) pertaining to the text prompt is replicated at other sides of the generated scene. As shown in (a) of Fig. [ this problem arises from the inherent limitation of the 2D diffusion model - it lacks
"Equal contributions.
Co-corresponding authors.

---

Published as a conference paper at ICLR 2024
Diflusion Model
"o dankey"
Diflusion Model
Difision Model
3Dflue
cute doniey"
Difiuaiin Model
Pows R
(a) Naive score distillation
(b) 3D aware score distillation (Ours)
Figure 1: Motivation. (a) Previous methods (Poole al. 2022 Song al., 2020) solely use noised rendered images and text prompts for score distillation, often resulting in scenes with poor 3D consistency. Our 3DFuse addresses this issue by giving diffusion models 3D awareness through consistency injection module, drastically improving the 3D consistency of generated scenes.
awareness of the camera pose from which it views the scene. Therefore, it is prone to produce frontal geometric features across all scene directions, generating distorted and unrealistic 3D scenes.
A possible approach to solve this issue would be training diffusion model that can be conditioned explicitly on camera pose values to produce plausible images of novel viewpoints (Liu al. 2023). However, there are inherent hurdles to this approach. As the pose-conditioned diffusion model operates solely on the 2D domain and has no explicit knowledge of 3D space, it has difficulty in modeling 3D transformations as well subsequent self-occlusions, resulting in geometrically inconsistent novel view inferences (Liu et al. (2024). The training also requires ground-truu 3D data, whose creations require either human effort or expensive 3D sensors, leading to smaller data size and inferior quality compared to that using the vast 2D data (Schuhmann et al. 2022) available. This limits the generative capability of this model from reaching that of ordinary 2D diffusion models. We solve this dilemma between 2D diffusion models and explicit 3D geometry with a middle-ground approach that combines the best of both worlds pretrained 2D diffusion model imbued with 3D awareness, conditioned on coarse yet explicit 3D geometry.
We propose novel methodology, dubbed 3DFuse, improve the geometric consistency of gener- ated 3D scenes. Central to our method is a consistency injection module, which effectively imbues pretrained 2D diffusion models with 3D awareness and is casily adaptable existing SDS-based text-to-3D baselines. Specifically, our consistency injection module creates 3D coarse geometry of given text through an off -ff-tff point cloud generation model (Nichol et al., [022] Wu al.) 2023), which can be projected to arbitrary views to generate coarse depth maps. As this coarse depth map is explicitly 3D consistent and outlines the overall structure of the scene expected at the view- point, the diffusion model conditioned on it can optimize the given viewpoint in 3D-aware manner. To this end, we add a ControlNet (Zhang & Agrawala 2023) module to the diffusion model and fine- tune it, leveraging the generative capability of the pretrained diffusion model toward inferring dense structures of the scene despite the sparsity of the depth map. We also introduce semantic coding to add controllability and specificity in the generation process and show how it can be leveraged with Low-rank adaptation (LoRA) (Hu et al. 2021) for more semantically consistent results.
To demonstrate our method's adaptability and its universally powerful performance various base- lines, we employ our framework across a range of text-to-3D baseline models, i.e. Dreamfu- sion (Poole al. 2022), SJC (Wang al.] (2022a), and ProlificDreamer (Wang al. 2023). The results show sigmificant improvements in both generation quality and geometric consistency on all baselines. We extensively demonstrate the effectiveness of our framework with qualitative analyses and ablation studies. Moreover, we introduce an innovative metric for quantitatively assessing the 3D consistency of the generated scenes.
2 RELATED WORK
Diffusion models (Ho et al.) 2020 Song et al. 2021) have gained much attention as generative
models due to their stability, diversity, and scalability. Given these advantages. diffusion models
have been applied in various fields, such as image translation (Rombach et al.] 2022 Seo et al.
2022) and conditional generation (Kanizo et al. 2013 Rombach et al.] [2022). Especially, text-to-
image generation has been highlighted with the introduction of various guidance techniques (Ho &
Salimans] 2021 Dhariwal & Nichol 2021). GLIDE (Nichol al. 2021) utilizes CLII (Radford
2

---

Published as a conference paper at ICLR 2024
et al. 2021 guidance to enable text-to- text-to-image generation, followed by large-scale text-to-image
models such as DALL-E2 (Ramesh et al. 2022) and Stable Diffusion (Rombach et al. 2022). Such
emergence has led the utilization of pretrained text-to-image models for tasks such endowing
additional conditions (Wang et al. 2022b or performing manipulations (Gal et al. 2022 Kwon
al. 2022).
Text-to-3D generation models generally employ pretrained vision-and aainn-and-language models, such as CLIP (Radford et al. (202T) to generate 3D shapes and scenes from text prompts. DreamFields (Jain ct at. 2022) incorporates CLIP with neural radiance fields (NeRF) (Mildenhall et al. 2020), demon- strating the potential for zero-shot NeRF optimization using only CLIP as guidance. Recently, Dreamfusion (Poole et al. 2022) and SJC (Wang et al., 2022a) have demonstrated an impressive ability to generate NeRF with frozen diffusion models instead of CLIP. Concurrently, some ap- proaches bring performance improvement through coarse-to-fie pipeline such as Magic3D (Lin et al. 2022) and Fantasia3D (Chen et al. 2023). ProlifiDDreamer (Wang et al. 2023) demonstrates extremely high-fidelity results through utilizing variational version of SDS, named VSD. How- ever, despite their impressive performance, the 3D inconsistency problem fundamentally remains, causing them to often generate distorted geometries.
Image-to-3) generation models generate 3D scenes from conditioning image, which is analogous to the task of generation-
novel view synthesis. There are 3D native diffusion models such as Point-E Nichol et al. 2022). trained upon several million internal 3D models to generate point clouds, as well as SDS based methods such as NeRDi (Deng et al. 2023). NeuralLift-366 (Xu et al. (023) and Realfusion -Melas-yrriazi et al. 2023) utilize reconstruction loss combined with monodepth estimation and textual inversion, respectively, to strengthen the semantic alignment of 3D scene the image. Among these, concurrent work Zero123 (Liu al. 2023) hears similarity to our work as it models 3D scenes in the 2D image domain by finetuning a 2D diffusion model infer a novel view of the input image given relative camera pose. Our work diverges from this approach in that we model the coarse 3D geometry explicitly. This explicit geometry enables a view-alige aligneigeedd depth map be given as condition for the diffusion model, allowing model 3D consistency more effectively.
3 PRELIMINARY
For the task of text-to-image generation, diffusion models such as Stable Diffusion (Rombach et al., 2022) receive text prompt as an additional condition. Specifically, when a text prompt e given, a mapping model T--) maps the prompt e into the embedding e = Tc).. Then, the embedding is injected into the diffusion model with parameters 6. Formally, we denote a score prediction network of the text-to-image diffusion model as (x,,,, t, TCc)), where 2, represents noisy image with noise level added a clean image Tg. For the sake of brevity, we shall omit the variable and refer to the function eø (z,, T(c)..
Score distillation sampling (SDS) (Poole et al. 2022) optimizes NeRF parameters toward following the direction of the score predicted by frozen diffusion model towards higher-density regions. Specifically, let us denote as parameters of NeRF, and (e(T) as a rendering function given a camera pose In SDS, random camera pose is sampled, and the diffusion model is utilized to infer the 2D score of the rendered image, i.e., x = Re(TT) This score is used to optimize the NeRF parameters θ by letting the rendered image move to the high-density regions, i.e., to be realistic. can be understood as minimizing a standard noise prediction loss function (Ho et al.] (020)) with respect the NeRF parameters ④ instead of the diffusion model's parameters such that:
O' = argmin E, E.,,, [(tt]]lo T(c) x Ro(T)) - el12]
(1)
where e is a Gaussian noise and w()) is a weighting function. The Jacobian term of diffusion U-net Deal (2,, /(x) from the gradient of Eq. can be omitted, formulating the new gradient as:
Or ae
VaLsDs(< R=(=): A Ex, (1) (co (x (T))
(2)
(1) is a weighting function from the DDPM (Ho al. 2020 diffusion process.
3

---

Published as a conference paper at ICLR 2024
Consistency Injection Module
intuial Promot Embedding
Diffusion U-Net
Semantic Code
Tuning
LoRA Layers
Optimmzed Prompt Embeedding
Wriertas
Sporse Depth
Pretra ned Oiffusion Model
Injector
30 Model
Camera Pose (r)
Score Distilationn
Neural Rendering
Intiall Image Gerecation
Optimiarion
Semantic Code Sampling
NeRF Optimization
Figure 2: Overall architecture of 3DFuse. Our framework consists of semantic code sampling, followed by consistency injection module that gives 3D-aware conditions to the diffusion model through a sparse depth injector. Semantic code, along with LoRA (Hu et al. 2021) layers, helps maintain the semantic consistency of 3D generation.
4
METHOD
4.1
MOTIVATION AND OVERVIEW
The method of score distillation-ban text-to-3D generation (Poole et al. 2022), despite its effec- tiveness, currently possesses crucial problem: inconsistent and distorted geometry in generated scenes due to 2D diffusion models not having explicit awareness of the 3D space nor the camera viewpoint it optimizing from (Wang al. 2023). Previous works (Poole al. 2022 Wang al.) 2022a) attempt to circumvent this problem by adding the text prompts that roughly describe the camera viewpoint (e.g.. "side view"). However, this ad-hoc approach is severely limited, as wide range of different pose values is ambiguously represented by few text prompts, leaving the gener- ation process still vulnerable to geometric inconsistencies and severe deformations. Therefore, to overcome this problem, the objective boils down to how we can inject more specific and explicit 3D awareness into pretrained 2D diffusion models.
Our method, 3DFuse, achieves this objective by conditioning 2D diffusion model to projected depth maps coarsely generated 3D structures. Specifically, we first conduct semantic code sam- pling, in which initially generate an image & from the given text prompt to specify the semantics of the 3D scene we wish to generate (Sec. 4.2p. From this image æ, we construct a coarse point cloud using off-the-shelf point cloud generation model (Nichol et al. 2022 Wu et al. 2023 For every rendering view, we leverage its projected depth maps as explicitly 3D consistent conditions for the 2D diffusion model. As the sparse depth map contains rich 3D information describing the scene from a given viewpoint, this approach effectively enables the diffusion model to generate the 3D scene in a 3D-aware manner (Sec.44 (4.)). We further leverage the semantic code through low-rank adaptation (LoRA) (Hu et al., 2021) layers to ensure semantic consistency of the scene (Sec. 4.4)
The overall architecture of 3DFuse is described in Fig.
4.2 SEMANTIC CODE SAMPLING
We focus on the fact that when generating 3D scene from a text, inherent ambiguity exists within the text prompt. For instance, the text prompt "a cule cal" is ambiguous regarding color, it could refer to either black or a white cat. Such lack of specificity leaves the possibility for the 3D scene to be erroncously guided toward vastly different textures and semantics at different viewpoints, resulting in semantically inconsistent 3D scene overall, as further demonstrated in Sec. 5.4
We introduce simple yet effective technique to reduce such text prompt ambiguity called semantic code sampling. To specify and secure the semantic identity of the scene we want to optimize towards, 2D image & is generated from the text prompt c. Then, optimize the prompt embedding to better fit the generated image, similarly to the textual inversion (Gal et al. 2022):
é = argmin |/ee (Ze, e) el
(3)
4

---

Published as a conference paper at ICLR 2024
where 2 is noised image of the generated image è with the noise and the noise level t. We refer to this pair of the generated image ê and the optimized embedding è as a semantic code 8, i.e, (#, é), which would be an input for our consistency injection module.
4.3 INCORPORATING A COARSE 3D PRIOR
Our main objective is injecting 3D awareness into pretrained diffusion model to enhance the 3D consistency of generated scenes, while fully leveraging the 2D diffusion model's expressive gen- erative capabilities. We achieve this with novel consistency injection module, which conditions diffusion models on sparse depth projections of a constructed point cloud.
Specifically, employ an off-the-shelf 3D model (-)) to construct 3D point cloud from the initial image E included in the semantic code s. (()) can be chosen from a wide variety of models: it could be point cloud generative model such as Point-E (Nichol et al. 2022) or single-image 3D reconstruction model such as MCC (Wu et al.) 2003)). For every SDS optimization step, as an image of the 3D scene is rendered using the diffusion model at camera viewpoint x, the constructed point cloud is projected to the same viewpoint resulting in point cloud depth map p.
p = (()()), TT)
where P(:, x) is a depth-projection function with a camera pose л. Adopting the architecture of ControlNet (Zhang & Agrawala 2023), our sparse depth injector &s receives the sparse depth map and we add its output Teatures to the intermediate features within pretrained diffusion U-net of eo (5e, é) in a residual manner, which can be further formulated as eg (ăz, é, Ep)).
The effectiveness of this approach can be understood from multiple aspects: most importantly, the sparse depth map p. explicitly modeled in 3D. provides the 2D diffusion model with the 3D consis- tent outline of the scene it is expected to generate, effectively encouraging 3D geometric consistency in the generation process. This approach also adds much-needed controllability to the text-to-31) generation pipeline: because our method enables the overall shape of the scene to be decided before the lengthy SDS optimization process, the user can pick and choose from a variety of initial point cloud shapes, allowing one to more easily generate specific 3D scenes tailored to their needs.
Training the sparse depth injector. As the sparse point cloud obtained by the off-the-shelf 3D model inevitably contains errors and ar- tifacts, its depth map also displays artifacts, shown in Fig. aa). Therefore, our module must be able to handle the inherent sparsity and the errors present in the projected depth map.
To this end, we employ two training strategies for sparse depth injector (.). First, we train our sparse depth injector using a paired dataset of real-life images and their point cloud depths (Reizenstein et al. (021). Our model trained to generate real-life images given its point cloud depth as a condition, enabling it to infer dense structures from sparse geometry given in point cloud depths. To increase the ro- bustness of our model against errors and arti- facts, we impose augmentations by randomly subsampling and adding noisy points to the point clouds from which the depth maps originate.
(a)
(b)
(c)
(d)
Figure 3: Qualitative results conditioned on the sparse depth map. Given sparse depth maps (a), (b) are results of cepth-conditional Stable Diffu- sion, (c) are results of ControlNet trained on Mi- DaS depths only, and (d) are 3DFuse results. Given text prompts are "a front view of an owl" and "a majestic eagle," respectively.
Second, the injector E.(.) is also trained on dense depth maps of text-to-image pairs, predicted using MiDaS (Ranftl et al., 2020). This strengthens the model's generalization capability, enabling it to infer dense structural information from categories not included in the 3D point cloud dataset for sparse depth training. In combination, given the depth map p along with the corresponding image and caption c, the training objective of the depth injector E.p()) is as follows:
Linjece (Ø) = Ey...p.t. [Ico (ve, E - -[1
(5)
Note that only the depth injector (-) is trained while the diffusion model remains frozen, making the training process more efficient, akin finetuning.
5

---

Published as a conference paper at ICLR 2024
ProlifieDreamer
+3DFuse
"a photo of comfortable bed!"
"a womed out DSLR photo of a ceramic lion, white background"
"a lamtern with handle"
Prolific Dreamer
+3DFuse
"a DSLR photo of origami motorcycle"
"a DSLR photo of ironman figure"
"a DSLR photo of an robot tiger"
Prolific ProlifcDrreamer Dreamer
+3DFuse
"a DSLR photo of Space Shuttle"
"a full size figure of princess wearing tiara"
"a dragon-shaped teapot"
Figure 4: Qualitative comparisons. We demonstrate the effectiveness of approach Prolific- Dreamer (Wang et al.) 2023) baseline. Incorporation of 3DFuse framework drastically enhances 3D consistency and fidelity of generated scenes.
These training strategies enable model receive sparse and noisy depth maps directly input and successfully infer dense and robust structural information without needing an auxiliary network for depth completion. As shown in Fig. restricted to the domain of the point cloud dataset. Note the category images (birds) used for the experiment is not included in the category of the point cloud dataset use.
approach generates realistic results without being
4.4 IMPROVING SEMANTIC CONSISTENCY
To result in scene semantically consistent all viewpoints, the diffusion model should produce a score that maintains a certain degree of semantic consistency regardless of given viewpoints. Al- though optimized embedding partially achieves this, further enhance this effect by adopting LoRA (Hu et al. (02I)) technique motivated by (Ryu) 2022). LoRA layers with parameters & con- sist of linear layers inserted into the residual path of the attention layers in the diffusion U-net.

---

Published as a conference paper at ICLR 2024
Dreamfusion
+3DFuse
photo of cute hippo"
"black kill heels"
SJC
+3DFuse
"a gentle deer with spotted coat with peaceful expression"
cutee sheep with white fur"
Figure 5: 3DFuse adaptation to other baselines. We demonstrate the effectiveness our approach on GableeDreemmFusonn (Poole al., 2022) and SJC (Wang al., 2022a) baselines. Incorporation of 3DFuse drastically enhances 3D consistency and fidelity of generated scenes.
Training the LoRA layers in this manner instead of the entire diffusion model further helps it avoid overfitting to a specific viewpoint. practice, given an image á generated from text prompt e, we fix the optimized embedding é and tune the LoRA layers (Roich et al. 2022):
CLORA( = E., [|Ea, (&e, E,()) <1[
(6)
5 EXPERIMENTS
5.1
IMPLEMENTATION DETAILS
We conduct all experiments with a Stable Diffusion model based on LDM (Rombach et al. 2022 and train the sparse depth injector using subset of the Co3D) dataset from the weights (Zhang & Agrawala 2023) trained using the image-text pairs along with the depth maps predicted by MiDaS. We conduct experiments with two off-the-shelf modules, mainly using the Point-E and in certain experiments also leveraging MCC (Wu el al. 2023) with MiDaS. For the semantic code sampling, we adopt Karlo (Donghoon Lee et al. 2022) based on unCLIP (Ramesh al. 2022). as we find that it tends to follow user prompt more closely.
5.2 TEXT-TO-3D GENERATION
We apply 3DFuse's methodology to previous SDS-based text-to-3D frameworks DreamFu- sion, Score Jacobian Chaining (SJC), and Pro- lificDreamer. All experiments utilize the Sta- Diffusion as its diffusion model, which is publicly available large-scale text-to-image dif- fusion model. Because DreamFusion's 's official implementation uses publicly unavailable Ima- gen (Saharia et al., 2022), we instead resort to using Stable Diffusion for implementation.
Figure 6: Qualitative results of 3DFuse with
MCC et. 2023). Our 3DFuse framework yields high-fidelity results with MCC model.

---

Published as a conference paper at ICLR 2024
Figure 7: Quantitative evaluation. We compare 3D consistency with our proposed metric using the COLMAP Schonberger &
Figure 8: User study. The user study conducted by surveying 102 participants to evaluate 3D co- herence, prompt adherence, and rendering quality.
Frahm 2016).
Prompt Overall coherence adherence quality
3D
Method
Method
Variance |
SJC 3DFuse (Ours)
60.1% %
59.2%
60.9% %
SJC 3DFuse (Ours)
0.0499
Stable- Sabble-Dreaausionn
23.4%
23.4%
22.7%
SJC Wang al. (2022a)
0.0870
SJC
16.5%
17.4%
16.4%
Qualitative evaluation. We present qualitative evaluation in Fig. and Fig. 5 which clearly demonstrates the effectiveness our method in ensuring the geometric consistency and high fidelity of generated 3D scenes. While the bascline methods produce inconsistent, distorted geometry in multiple directions, combined with 3DFuse, they are empowered to generate robust geometry at every viewpoint, as evidenced by the figure as well as video results provided in the supplementary materials. In Fig. 6 we also present the qualitative results of our approach when utilizing MCC (Wu al. 2023) for the 3D prior, which also show consistent high-fidelity 3D scene generation results.
Quantitative evaluation. is difficult to conduct quantitative evaluation on a zero-shot text-to-3D generative model due the absence of ground truth 3D) scenes corresponding to the text prompts. In this light, we propose new metric that utilizes COLMAP (Schonberger & Frahm (016) to measure the 3D consistency of a generated scene. In Table we report the average variance scores over 42 generated 3D scenes. Our 3DFuse framework outperforms baseline SJC by a large margin, demonstrating that our framework achieves more geometrically consistent text-to-3D generation. The details of our metric are given in Appendix C.2
User study. We have conducted user study with 102 participants, which is shown Table. g We have asked the participants to choose their preferred result regarding 3D coherence, prompt adherence, and overall quality between Gable-DreemFusionn SJC, and our 3DFuse-combined SJC. The results show that 3DFuse generates 3D scenes that the majority of people judge having higher fidelity and better geometric consistency than previous methods. Further details are described in Appendix
5.3 VIEW-DEPENDENS TEXT-TO-IMAGE GENERATION
"A front of"
"A side view of" "A backside view
We conduct text-to-image generation experi- ments with 3DFuse to verify our framework's 's capability inject 3D awareness into 2D dif- fusion models. We observe whether the im- ages are generated in 3ww- manner when viewpoints are given as conditions through our 3DFuse framework. Fig. 9
(b)
demonstrates the effectiveness of our framework: it allows for highly precise control of the camera pose in 2D images, with even small changes in viewpoint being reflected well on generated images. Our approach shows superior performance to previous prompt-based methods (e-g.. "A front view of"") regarding both precision and control- lability of injected 3D awareness.
"a blue bird with beak and feathered wings"
(b)
black horse"
5.4 ABLATION STUDY
Figure 9:
View-dependent image generation comparison. (a) the results of naive view augmented prompting, and (b) our results with 3DFuse framework.
Comparison with Zero-123, We conduct a qualitative comparison with concurrent work Zero-123, which bears similarity to our work in that it aims to inject 3D awareness into diffusion models through view conditioning. We compare the two methods by conducting SDS-based 3D scene generation, with Zero-123 its diffu- sion model and the other with our 3DFuse framework as its diffusion backbone model, with all other settings identical. As shown in Fig. 10 notice that our 3DFuse results generally more
8

---

Published as a conference paper at ICLR 2024
Condition
Condition
Zero123
Condition
"a fancy chair"
beach chair"
3DFuse
Side
Back
Side
Back
Condition
"a simple mug" "a rusty mug"
Figure 10: Qualitative result for comparison with Zero-1-to-3 (Liu et al., (023). With ref- erence image directly given as input, 3DFuse generates expressive, high-fidelity 3D scenes than Zero-I-to-3.
Figure 11: 3D reconstruction with differ- prompts given single 3D structure. The first column represents the given point clouds as conditions, and the second and third columns show synthesized results.
expressive and high-fidelity results. We hypothesize this phenomenon related to the fact that, un- like Zero-123, which requires finetuning of the diffusion model itself for incorporation of 3D pose awareness, our framework only finetunes an external conditioning module in manner similar to ControlNet (Zhang & Agrawala, 2023). leaving the diffusion model untouched. This allows our framework to leverage the pretrained diffusion model's generative capability to the fullest, resulting in more expressive and high-fidelity scene generation. We provide additional experiment results applying Zero- 123 for text-to-3D generation, like our methodology. section of our Appendix.
Different prompts with single 3D representation. 3D representation, i.e. point cloud obtained from off-the-shelf models (Nichol et al. 2022 Wu et al. 2023) is typically coarse and sparse. To in- vestigate how the diffusion model implicitly refines the coarse 3D structure, conduct ablation study using fixed point cloud and different text prompts instead of inferring a point cloud from the initial image of semantic code Fig. 11 shows that 3DFuse is flexible in responding the егror and sparsity inherent in the point cloud, depending on the semantics of text input.
Semantic code sampling. We conduct an ablation study on semantic code sampling in our 3DFuse framework. The results in the first two columns of Fig. [2] show significant differences in geome- try (pumpkin's 's smooth left surface and bumpy right surface) and semantic details (the tank's s absence of wheel on the left) without semantic code sampling, depending viewpoints. In contrast, using seman- tic code sampling ensures both geometric and se- mantic consistency across viewpoints, demonstrat- ing its prominence in preserving the semantic iden- tity of the 3D scene.
semantic cade sampling
semantic code sampling
Left side
Right side
Left side
Right side
Figure 12: Ablation study on semantic code sampling. The given prompts are "a round orange pumpkin with stem and a tex- tured surface" (top) and "a product photo of toy tank" (bottom).
CONCLUSION
In this paper, we address the inconsistency problem within SDS-based text-to-3ID generation. We propose novel framework, 3DFuse, that incorporates 3D awareness into pretrained 2D dif- fusion model. Our method utilizes -iewpoint-specific depth maps from a coarse 3D structure passed through sparse depth injector and semantic code sampling for semantic consistency. Our work offers practical solution for addressing the limitations of current text-to-3D generation techniques and opens up possibilities for more realistic 3D scenes from text prompts. Our experimental results demonstrate the effectiveness of our framework, outperforming previous models in quantitative met- rics and qualitative human evaluation.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This research was supported by the MSIT, Korea (UTP-2022-2020--0011811, ICT Creative Con- silience Program, RS-2023-00227592, Development of 3D Object Identification Technology Robust to Viewpoint Changes).
REFERENCES
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF Inter- national Conference Computer Vision (ICCV), October 2023.
Congyue Deng, Chiyu Jiang, Charles Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, et al. Nerdi: Single-view nerf synthesis with language-guide diffusion as general image priors. In Proceedings of the IEEE/CVE Conference on Computer Vision and Patlern Recognition, pp. 20637-20647, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 34:8780-8794, 2021.
Jiseob Kim Donghoon Lee, Jongmin Kim Jisu Choi, Woonhyuk Baek Minwoo Byeon, and Sachoon Kim. Karlo-v petto-v1.0.aam 1.0.alpha on coyo-100m 100m and cc15m. htttps://githb... htt.thuub..cm/kkkaaaaaaa...
karlo 2022.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashník, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208. 016188, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 (858440-651,, 2020.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09688 2021.
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 867-876, 2022.
Yossi Kanizo, David Hay, and Isaac Keslassy. Palette: Distributing tables in software-defined net- works. In 2013 Proceedings IEEE INFOCOM, pp. 545-549. IEEE, 2013.
Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have semantic latent space. arXiv preprint arXiv:2210 /0. 10960, 2022.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:
2022.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu. and Hao Su. One- 2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; Zero-shot one image 3d object. arXiv preprint arXiv:2303. / 2023.
10

---

Published as a conference paper at ICLR 2024
Zhengzhe Liu, Peng Dai, Ruihui Li, Xiaojuan Qi, and Chi- Shi-Wing&& Fu. Iss: Image stetting stone for text-guided 3d shape generation. arXiv preprint (rXi:22099.04445, 2022.
Luke Melas- Kyriaz,, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion: 360deg reconstruction any object from a single image. In Proceedings of the IEEE/CVE Conference Computer Viston and Pattern Recognition, pp. 8446-8455 2023.
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint 1rXiv:2211007600 2022.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 12. 12. 2021.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:22/2.0877// 2022.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem. Magic123: One image high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306. 17843, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pp. 1748-87663 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transac- tions on pattern analysis and machine intelligence, 44(3):1623-1637, 2020.
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan- Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating deep learning with pytorch3d. arXiv:2007.0850/ 2020.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cat- egory reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10901-10911, 2021.
Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Transactions on Graphics (TOG), 42(1):1-13,, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Simo Ryu. Low-rank adaptation for fast text-to-image diffusion fine-tuning, https://githan
com/ caam/clonede Lloneofsimo/iie /oraa 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi. Rapha Gontijo Lopes, et al. Photorealistic eext-to-imaee diffusion models with deep language understanding. arXiv preprint arXiv:2205. 1487, 2022.
11

---

Published as a conference paper at ICLR 2024
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Aditya Sanghi, Hang Chu, Joseph G. Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In Pro- ceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18603-18613, June 2022.
Johannes L Schonberger and Jan-Michael Frahm. Structureffrommmotiinn revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4104-4113, 2016.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.021 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
Junyoung Seo, Gyuseong Lee, Scokju Cho, Jiyoung Lee, and Seungryong Kim. Midms: Matching interleaved diffusion models for exemplar-based image translation. arXiv preprint arXiv:2209./104,, 2022.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
Yang Song, Jascha fohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2020.
Jiaxiang Tang. https/hhwhhooooeesaaaweeeeeeebbe dreamuuuuin...aaaasioooono.
Stable-dreamfusion:
Text-to-3d with stable-diffusion, 2022.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond Yeh, and Greg Shakhnarovich. Score ja-
cobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:2212.00774, 2022a.
Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. arXiv preprint arXiv:2205.1299 2022b.
Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305. /6213, 2023.
Chao- Yuan Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, and Georgia Gkioxari. Multiview compressive coding for 3D reconstruction. arXiv preprint arXiv:2301.082477 2023.
Dejia Xu, Yifan Jiang, Peihao Wang. Zhiwen Fan, Yi Wang. and Zhangyang Wang. Neurallift-360: Lifting in-the-wild 2d photo to 3d object with 360deg views. In Proceedingy the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 4479-4489, 2023.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-iman -moge diffusion models. arXiv preprint arXiv:2302.00 2302.05543, 2023.
12

---

Published as a conference paper at ICLR 2024
Appendix
In Sec.
A provide additional implementation details for our proposed method, 3DFuse. In B
Sec.
we present the results of additional experiments to validate our approach. In Sec. C we provide details regarding the user study and the evaluation of our method. Sec. D we discuss the limitations of our approach.
A IMPLEMENTATION DETAILS
A.1 TRAINING DETAILS
We use the Co3D (Reizenstein et al. 2021) dataset to train our sparse depth injector. The dataset is comprised of 50 categories and 5,625 annotated point cloud videos. From these point cloud- annotated videos, we randomly subsample three frames create 16,875 pairs of RGB images and their projected depth map. We use 3k-5k points, augmented with 0-10% noise points, sampled from the dense point clouds provided in the Co3D dataset. We employ the rasterizer from Py- Torch3D (Ravi et al. 2020) library to project the point clouds and synthesize sparse depth maps. To relieve domain constraints from the Co3D dataset, we also employ text-to-image pairs along with MiDaS (Ranfil et al., 2020)- predicted dense depth maps. For case of training, we start from weights (Zhang & Agrawala 2023) trained on the text-to-image pairs with MiDaS depth and fine-tune the model using the sparse depth maps synthesized from the Co3D dataset for 2 additional epochs.
A.2 INTENSITY CONTROLS
All networks equipped with a pretrained diffusion model in our framework are designed to add ex- ternal features to the intermediate features of the diffusion U-Net in residual manner. Therefore, their intensities can be adjusted by multiplying scaling factors. This enables us to control the in- fluence of the LoRA layers, used for semantic consistency, and the sparse depth injector, used for geometric consistency, on the diffusion model. In particular, adjusting the intensities of the LoRA layers is effective in controlling how much our 3D scene overfits the initial image. We use scaling factors of 0.3 and 1.0 on the features passing through the LoRA layers and the sparse depth injector, respectively.
A.3 SEMANTIC CODE SAMPLING
To perform semantic code sampling, an initial image first generated based on the text prompt, followed by optimization of corresponding text prompt embedding. The specific method follows that of Textual Inversion (Gal et al. 2022), wherein a new word is added to the embedding space and then optimized. Additionally, this initially generated image is used input for the aff-the-shelf 3D model in the auxiliary module. To facilitate the optimization and 3D shape inference processes, add "a front view of" and ", white background" before and after the text prompt.
A.4 ARCHITECTURAL CHOICES
There are manifold text-to-image diffusion models that have been trained at large scale, including DALL-E2 (Ramesh et al. 2022), Imagen (Saharia et al. 2022), and Stable Diffusion (Rombach et al., 2022). However, as DALL-E2 and Imagen are not publicly available, we adopt Stable Diffusion, the most popular and widely used model.
To impose an additional condition on pretrained text-to-image diffusion models, various methods such as PITI (Wang et al.) 2022b) and ControlNet (Zhang & Agrawala, 2023) have been proposed. Recently, Depth-conditional Stable Diffusion (Rombach et al. 2022) has been introduced, which finetunes Stable Diffusion to receive MiDaS (Ranfil et al. 2020 depth map as condition for image translation. After conducting experiments with the above methods to find the optimal solution for our setting, find that ControlNet achieves the best efficiency in training sparse depth-conditionel diffusion model. We thus incorporate ControlNet architecture into our framework. Fig illusttraee architectural choice incorporating Nen architecture into our 3use framework.
13

---

Published as a conference paper at ICLR 2024
Gaussian noise
Rendered image
embedding e*
Sparse depth map
Rg (z)
P
Text encoder
Enc. block
Enc. block
Enc. block
Enc. block
:
Enc. block
Enc. block
Middle block
Middle block
Dec. block
A.4
Conv
Lincars
Dec.
Conv
Add
Dec. block
AdH
Conv
LoRA layers
Diffusion model f
Sparse depth injector (based on ControlNec)
(based on U-Net)
Figure 13: Architectural choices for 3DFuse framework. The sparse depth injector follows the architecture of ControlNet (Zhang & Agrawala 2023), with the encoder side comprised of encoder blocks copied from the diffusion U-Net and the decoder side comprised of convolutional layers. The sparse depth map features are added to the intermediate features in the decoder blocks of the diffusion U-Net. LoRA layers are attached every attention layer of the diffusion U-Net.
B ADDITIONAL EXPERIMENTS
B.1 COMPARISONS WITH NOVEL VIEW MODEL-BASER CONCURRENT WORKS
In Figure 14, we conduct comparison of our model with different image-to-3D models (Liu al., 2023 Qian et al. 2023), we first generate images from a single text prompt (through the Stable Diffusion (Meterr al. 2022) model) and give these images Zero-123 (Liu et al. 2023 and Magic- 123 Qian et al. (2023) for 3D scene generation. This experiment setting can be seen as observing what happens the image-to-3D component of our 3DFuse methodology replaced with other existing explicit image-to-3D models. Our 3DFuse undergoes its original generation process with the identical text prompt, and the images given each model are not curated. Also, to remove the aliasing-lite like effect in the renderings that occurred due to the feature rendering architecture of SJC, we have changed the backbone NeRF model of Zero- 123 and 3DFuse from feature-level SJC to pixel-level Instant-NGP backbone, which allows us gain clearer results.
The experiment results show that our method's generative capabilities enable our model to generate more detailed, high figh-fidelity results in comparison to image-to-3D models that attempt explicit novel view synthesis of conditioning images. We hypothesize the reason for this is that strict image-to-31 models struggle to predict novel views of an image when an image is outside the domain of training data, while our model is loosely conditioned and therefore gives the generative model more freedom to generate realistic details, fully realizing its generative capability. This agrees with what have explained in our comparison Zero- 123 in our main paper.
B.2 ROBUSTNESS
In the main paper, we present experimental results of 3DFuse with fixed random seeds for a fair comparison. Here, we conduct an additional experiment to verify the robustness of our approach. We compare method with previous works (Poole et al. 2022 Wang al. 2022a) with combinations of fixed text prompts and varying random seeds. Fig. [5| demonstrates that our approach exhibits enhanced robustness compared previous works in terms of stochasticity.
14

---

Published as a conference paper at ICLR 2024
Stable Diffusion
Zoro123
Stable Difusion
Magici2s
3DFuse (Ours)
"A 3D model of an adorable cottoge with thatched roof."
Stable Diffusion Stable Oiffusion+
Zero123
Magic123
3DFuse (Ours)
"A highly detailed sand castle." "
Stable Diffusion Stable Diffusion
Magiciz3 Zeroi
3DFuse (Ours)
"Sydney opera house, aerial view."
Figure 14: Comparisons with novel view model-based concurrent works (Liu et al. 2023 Qian
et al. 2023). The experiment results show that method's generative capabilities enable our model to generate more detailed, high-fidelity results in comparison to image-to-31 models that attempt explicit novel view synthesis of conditioning images.
B.3 SPARSITY VARIATION
We provide additional analysis of our model's robustness against sparsity and ambiguity of point cloud. We drastically change the number of points in point cloud and generate images from its depth map. As shown Fig. [6 despite varying sparsity, our model shows consistency in inferring dense structures and generates realistic images.
15

---

Published as a conference paper at ICLR 2024
Drsimionrcamaoiionn
SIC
3DFuse (Ours)
Sced 0
Sced
Sced 2
Seed
Sced 4
"a photo comfortable bed
Seed D
Seed
Seed 2
Seed
Seed
"a black borse"
Figure 15: Qualitative comparison for robustness. Qualitative results with varying random seeds show our model's robustness against stochastic factors. We visualize the results generated with the seed from 0 to 4.
B.4 ADDITIONAL QUALITATIVE RESULTS
Fig. [7 displays additional qualitative results. Our qualitative results demonstrate that the outputs of our model have superior robustness and 3D-consistency compared to previous methods.
B.5 ABLATION ON SPARSE DEPTH INJECTOR
We conducted an ablation study regarding the effectiveness of conditional depth maps. Specifically, remove the sparse depth injector from the diffusion model that receives optimized embedding
16

---

Published as a conference paper at ICLR 2024
9182 pts
4096 pts (default)
2048 pts
1024 pts
1024 pts (back)
Projected depth.
Generated 2D
Figure 16: Sparsity variation with fixed random seed.
with LoRA layers. Fig. 18 presents the result of our framework with and without the sparse depth injector, confirming its importance in achieving geometric consistency.
C EVALUATION DETAILS
C.1 USER STUDY
To qualitatively compare our 3DFuse with previous methods, DreamFusion (Poole et al. 2022) and SJC (Wang et al. 2022a), we conduct user study. As the Imagen (Saharia al. (022)) model used
in DreamFusion publicly unavailable, we utilize Stable- Saalee-Dremuusttee (Tang) (022). which uses
Stable Diffusion (Rombach et al. 2022). All methods employ Stable Diffusion vI.5 checkpoint.
Our user study questionnaire shows rendered images of 3D scenes generated by 3DFuse, SJC, and using identical prompts. We ask the participants to select the result that best fits each of the questions given below. We use 7 prompts and generate three 3D scenes per prompt using 3DFuse, SJC, and Stable-DreamFusine the camera poses identical across different models for fair comparison. We do not disclose the models used to generate the results and randomize the order of methods for each question order to keep the selection process fair. Our final user study is the statistics summarizing responses from a total of 102 participants.
We give rendered images for each 3D scene, with
The questions used in the user study as follows:
These images are rendered by rotating the camera horizontally from 0 to 360 degrees. Which result has the most natural shape? (3D coherence)
Which result seems follow the user input text the best? (prompt adherence) Which result has the best overall quality? (overall quality)
C.2 COLMAP-BASED METRIC
We provide additional details of the COLMAP (Schonberger & Frahm (016)-baser metric we pro- pose. We sample 100 uniformly spaced cameras whose origin points are on a hemisphere of fixed radius at an identical elevation angle, which is 30 degrees in our setting. All cameras are directed towards the center of the hemisphere, and we render 100 images of the 3D scene from these camera viewpoints. Subsequently, COLMAP predicts the camera pose of each image for point cloud recon- struction. We measure the difference between the predicted camera poses of image pairs that were adjacent at the previous rendering stage.
The variance between the different values is used as a metric for the 3D consistency evaluation. As initial differences between all adjacent camera pairs identical the rendering stage, a high variance in predicted difference values indicates the magnitude inaccuracy in the COLMAP op- timization process that predicts cameras. Such inaccuracies are caused by distortions and repeating
17

---

Published as a conference paper at ICLR 2024
Stable-DreamFussee
SJC
3DFuse (Ours)
"a cozy fireplace with flickering flames and crackling logs"
"a pink faminggo
"a stack of pancakes with maple syrup and butter"
"a zoomed out Colosseum"
"a black
blackffutuistt futuristic drone" "
"a playful zebra with black and white stripe"
Figure 17:
More qualitative results. Additional qualitative results show 3DFuse's superior perfor- mance in ensuring geometric and semantic consistency.
artifacts in 3D scene, whose ambiguous and repetitive nature confuses COLMAP optimization by offering multiple erroneous solutions in the camera pose prediction process. Therefore, it can be argued that this variance value corresponds to the prominence of 3D inconsistent features that make optimization for COLMAP difficult.
Fig. 19 visualizes the point cloud and the camera poses reconstructed through COLMAP optimiza- tion. The left columns show COLMAP optimization results inconsistent 3D scenes generated by SJC (Wang et al. 2022a), and the right columns show COLMAP optimization results of consistent 3D scenes generated by 3DFuse. It is noticeable that the cameras predicted from 3DFuse's 's scene more uniformly spaced and stable than those of SJC. In the left-right figure showing horse
18

---

Published as a conference paper at ICLR 2024
wio Sparse Depth Injector
w/ Sparse Depth Injector
w/o Sparse Depth Injector
w/ Sparse Depth Injector
cocy in chimney porch
Colusseum
"a colorfiul candy house"
"o photo of cute hippo"
Ablation sparse depth injector. Our results demonstrate that the absence of sparse depth injection induces the breakdown geometry. resulting in blurry and inconsistent 3D shapes, proving the prominence of the sparse depth injector in our framework.
Figure 18:
Rendered image
Reconstruction
Rendered image
Reconstruction
Figure 19: Visualization of predicted camera pose from COLMAP (Schonberger & Frahm 2016) optimization. Left columns visualize COLMAP optimization results from the renderings of SJC-generated scenes (Wang et al. 2022a), while the right columns visualize the results from 3DFuse.
generated by SJC, COLMAP displays evident difficulty in handling the ambiguous, repetitive fea- tures of the 3D-inconsisseent horse, leading it to mistakenly predict that all cameras are distributed on only one side of the hemisphere. As this phenomenon consistently observed throughout mul- tiple scenes, we demonstrate that such distortions and 3D inconsistencies increase the difficulty of COLMAP optimization and harm the camera pose prediction quality.
Please note that limitation of the COLMAP-based metric lies in its resolution. If the generated output lacks sufficient 3D consistency, the optimization process within COLMAP may fail con- verge, resulting in significantly lower scores. Consequently, it becomes challenging differentiate between such samples. However, this metric retains an advantage assessing the 3D consistency among samples of reasonably high quality.
C.3 LIMITATION OF CLIP-BASED METRIC
For 2D image generation, metrics such as FID (Heusel et al. (017)) and Inception score (Salimans et al. 2016) are used to quantify the generation quality by measuring the similarity between the distribution ground truth images and generated images. However, the same approach cannot be applied to score distillation-base sponding to the text prompts. This presents challenge for conducting a quantitative evaluation of zero-shot text-to-3I) generative models, which previous works have bypassed by showcasing var-
text-to-3D generation, due the absence of 3D scenes corre-
19

---

Published as a conference paper at ICLR 2024
(a) Average of CLIP score: 19.9
(b) Average of CLIP score: 19.2
Figure 20: Scenes evaluated by CLIP-based metric. (a) inconsistent 3D scene and (b)
consistent 3D scene. The inconsistent 3D scene achieves a higher score in the CLIP-based metric.
ious qualitative results instead of quantitative evaluation study (Lin et al. 2022).
Wang al.
2022a) conducting user
Another approach used in previous works (Jain et al. 2022 Poole et al. 2022 is to use CLIP-based metrics, such as CLIP R-precision, which measures retrivaal accuracy based on CLIP (Radford et al., (021) through projected 2D image and text input. Nevertheless, such CLIP-based metrics are ineffective in measuring the quality and consistency of generated 3D geometry. For example, when geometric failure such tures multiple times around the 3D scene, the CLIP-based metric measures all such viewpoints with frontal features as having high similarity with the text prompts. This causes such 3D scenes have misleadingly high scores despite having incorrect geometry. Fig. 20 illustrates phenomenon, where each image rendered geometric inconsistent scene (first row) achieves higher CLIP score than each image generated in a geometric consistent scene (second row).
multiple-face problem occurs, producing frontal geometric fea-
example of this
D LIMITATION
We have proposed novel 3DFuse framework that infuses 3D awareness into pretrained 2D diffu- sion model while preserving its original generalization capability. While 3DFuse stably optimizes NeRF for text-to-3D generation, demonstrating superior performance, there are still several limita- tions that need to be addressed. First, our approach inherits the difficulty in reflecting complex user prompts faithfully, it relies the ability of the pretrained diffusion model to follow text prompts. Additionally, our approach may face various societal biases inherent in the dataset (Schuhmann)
al. 2021), similar the text-to-image generation models (Rombach et al. 2022).
20


Reviews:
[{"review_id": "ogBw9GTIgo", "reviewer": "Reviewer_ftV7", "summary": "This work introduces a new framework called 3DFuse that enhances the robustness and 3D consistency of score distillation-based methods for text-to-3D generation. The framework incorporates 3D awareness into the pretrained 2D diffusion model, resulting in geometrically consistent and coherent 3D scenes. The authors also introduce a new technique called semantic coding for improved results. The effectiveness of the framework is demonstrated through qualitative analyses and ablation studies, and it is shown to outperform previous prompt-based methods in terms of precision and controllability of injected 3D awareness. The 3DFuse framework and semantic coding technique have the potential to improve the quality and controllability of generated 3D scenes, which could have applications in various fields such as virtual reality, gaming, and architecture.", "questions": "- How stable is the 3D point cloud generative model? Any failure cases of predicted 3D point cloud? How to handle it?\n\n- Why should it predict a point cloud and then convert it to a depth map? What if the proposed module directly predicts a monocular depth map?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The authors introduce a novel framework called 3DFuse that enhances the robustness and 3D consistency of score distillation-based methods for text-to-3D generation. This framework incorporates 3D awareness into the pretrained 2D diffusion model, resulting in geometrically consistent and coherent 3D scenes.\n\n- The authors introduce a new technique called semantic coding that involves generating an initial image based on the text prompt and optimizing the corresponding text prompt embedding. This technique improves the quality and controllability of generated 3D scenes.\n\n- The authors demonstrate the effectiveness of the 3DFuse framework through qualitative analyses, which show that it outperforms previous prompt-based methods in terms of precision and controllability of injected 3D awareness.\n\n- The 3DFuse framework and semantic coding technique have the potential to improve the quality and controllability of generated 3D scenes", "weaknesses": "- My biggest concern is the marginal improvement. It seems that the proposed model only shows a very limited improvement compared to each baseline method.\n\n- My understanding is that the video from supp is generated by render well-optimized NeRF, which should be inherently. But why does the video itself look aliasing?\n\n- The proposed framework breaks the text-to-3D problem into text-to-image + image-to-3D tasks. Therefore, it would be better to compare it with other image-to-3D methods [a, b] too.\n\nReferences:\n\n[a] Zero-1-to-3: Zero-shot One Image to 3D Object\n\n[b] Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wErz6hPtk4", "reviewer": "Reviewer_MdgG", "summary": "This paper introduces 3DFuse, a new method to integrate 3D consistency information into the SDS loss optimization pipeline of text-to-3D generation. The authors first fine-tune the Stable Diffusion model to understand sparse depth conditions. \nDuring SDS optimization, 3DFuse reconstructs a coarse point cloud from an image using off-the-shell methods and then renders sparse depth maps from the point clouds for depth-conditioned SDS loss supervision.\nBoth qualitative and quantitative analyses underscore the efficacy of conditioning based on coarse point clouds, resulting in enhancements in the text-to-3D generation results. Additionally, the authors have unveiled a semantic coding method to tackle inherent semantic ambiguities.", "questions": "NA", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. 3DFuse offers a robust solution to fine-tune a sparse depth-conditioned Stable diffusion model, employing the Co3D dataset. The result in Fig.3 provides a compelling argument for the performance of the depth-conditioned diffusion model. Furthermore, depth-conditioned SDS loss augments the 3D consistency of generated results, as depicted in Fig.4.\n2. The authors explore the semantic ambiguity problem, which has often been overlooked in prior studies. They introduced semantic code sampling to mitigate this challenge, and Fig.12 convincingly showcases the effectiveness of this strategy.\n3. The authors conduct comprehensive ablation studies and extensive experimental validation. Most of the proposed components are well-ablated. In addition, the authors also provide a user study in Fig.8.\n4. This paper boasts of a coherent and lucid narrative structure, facilitating easy comprehension.", "weaknesses": "1. 3DFuse relies heavily on off-the-shelf point cloud reconstruction or generation methods like Point-E, to obtain coarse point clouds. These point clouds subsequently serve as conditional information in the SDS optimization process. However, sparse-depth renderings are low-quality and may have many artifacts, resulting in degenerated and ambiguous results as shown in Fig.4.\n2. The authors provide the novel view synthesis results compared with Zero-123 in Fig.10. However, 3DFuse fails to fit the reference image and generates a different object from it. Furthermore, the predicted novel views are also blurry.\n3. The quality of 3DFuse's results lags behind contemporaneous methods. Additionally, it is noteworthy that the results for benchmark methods, such as Dreamfusion and ProlificDreamer, presented in this paper, appear to be less robust than those delineated in their original publications and open-source implementations. It would be beneficial for the authors to recalibrate their methodology based on more formidable baseline models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rRtCllL1hJ", "reviewer": "Reviewer_s3AW", "summary": "The paper investigates the geometry quality of SDS-based methods. By using an additional control net module that is aware of the sparse depth from point-e/mcc, the model grounds the 3D generation with rough geometry and therefore reduces the janus issues. The proposed pipeline can be integrated with SDS and VSD.", "questions": "What are the major differences between the proposed method and the above [1][2] on improving semantic consistency in SDS?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The depth-aware controlnet module alleviates the janus problem and improves the geometry quality of 3D generation greatly.\n\n+ LoRA is utilized for parameter-efficient fine-tuning when improving the semantic consistency. \n\n+ I played with the code of the proposed framework, and it works well.", "weaknesses": "- Missing references on image-to-3D generation (CVPR'23 papers, both released on arxiv before the original release of this submission). [1] proposes textual inversion + dreambooth method for improving semantic consistency, similar to Imagic. [2] uses textual inversion for semantic consistency. Would be nice if these approaches were discussed in the related works section, as they are relevant to the semantic consistency component in this work.\n\n[1] Xu D, Jiang Y, Wang P, et al. NeuralLift-360: Lifting an In-the-Wild 2D Photo to a 3D Object With 360deg Views[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 4479-4489.\n\n[2] Melas-Kyriazi L, Laina I, Rupprecht C, et al. Realfusion: 360deg reconstruction of any object from a single image[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 8446-8455.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)