SweetDreamer: Aligning Geometric Priors in 2D diffusion for Consistent Text-to-3D
Authors: Weiyu Li
Rui Chen
Xuelin Chen
Ping Tan
Venue: ICLR 2024 poster (2024)
Keywords: Text-to-3D
diffusion models
Primary area: generative models

Abstract: It is inherently ambiguous to lift 2D results from pre-trained diffusion models to a 3D world for text-to-3D generation. 2D diffusion models solely learn view-agnostic priors and thus lack 3D knowledge during the lifting, leading to the multi-view inconsistency problem. We find that this problem primarily stems from geometric inconsistency, and avoiding misplaced geometric structures substantially mitigates the problem in the final outputs. Therefore, we improve the consistency by aligning the 2D geometric priors in diffusion models with well-defined 3D shapes during the lifting, addressing the vast majority of the problem. This is achieved by fine-tuning the 2D diffusion model to be viewpoint-aware and to produce view-specific coordinate maps of canonically oriented 3D objects. In our process, only coarse 3D information is used for aligning. This “coarse” alignment not only resolves the multi-view inconsistency in geometries but also retains the ability in 2D diffusion models to generate detailed and diversified high-quality ob-jects unseen in the 3D datasets. Furthermore, our aligned geometric priors (AGP) are generic and can be seamlessly integrated into various state-of-the-art pipelines, obtaining high generalizability in terms of unseen shapes and visual appearance while greatly alleviating the multi-view inconsistency problem.

SWEETDREAMER: ALIGNING GEOMETRIC PRIORS IN 2D DIFFUSION FOR CONSISTENT TEXT-TO-3D
Weiyu Li Rui Chen X Chen Ping Tan 1:2f
1 Hong Kong University of Science and Technology
2 Light Illusions
3 South China University of Technology
4 Tencent AI Lab
(weiyuli.cn, suelin.chen. 3d) ggmail.com, riorui @fw.@@ff..moommmmmmmmmm pingtan @ @ust.hk
ABSTRACT
It is inherently ambiguous to lift 2D results from pre-trained diffusion models to a 3D world for text-to-3D generation. 2D diffusion models solely learn view- agnostic priors and thus lack 3D knowledge during the lifting, leading to the multi- view inconsistency problem. We find that this problem primarily stems from ge- ometric inconsistency, and avoiding misplaced geometric structures substantially mitigates the problem the final outputs. Therefore, we improve the consis- tency by aligning the 2D geometric priors in diffusion models with well-defined 3D shapes during the lifting, addressing the vast majority of the problem. This is achieved by fine-tuning the 2D diffusion model to be viewpoint-awarc and produce view-specific coordinate maps of canonically oriented 3D objects. In our process, only coarse 3D information is used for aligning. This "coarse" alignment not only resolves the multi-view inconsistency in geometries but also retains the ability in 2D diffusion models to generate detailed and diversified high-quality ob- jects unseen in the 3D datasets. Furthermore, our aligned geometric priors (AGP) are generic and can be seamlessly integrated into various state-of-the pipelines. obtaining high generalizability in terms of unseen shapes and visual appearance while greatly alleviating the multi-view inconsistency problem.
INTRODUCTION
Generative models have achieved diverse and high-quality image generation, in a highly controllable way with input text prompts (Nichol et al., 2022; Ramesh al., 2021; Saharia al., 2022b; Rom- bach et al., 2022). This remarkable achievement has been attained by training scalable generative models, particularly diffusion models, on an extensive corpus of paired text-image data. To repli- cate such success in 3D, substantial endeavor is obviously necessary to gather vast amount of high-quality text-3D pairs, which is currently receiving commendable attention (Deitke et al., 2023; Wu et al., 2023; Shrestha et al., 2022). However, it evident that the effort required to collect comprehensive 3D dataset covering highly varied subjects is considerably more significant, given the high cost associated with acquiring high-quality 3D content.
On the other end, attempts to achieve text-controlled 3D generative models have taken several routes, among which the 2D-lifting technique has emerged as particularly promising direction and is increasingly gaining momentum in the field (Poole et al., 2022). This technique lifts 2D results into a 3D world and features an optimization framework, wherein a 3D representation is updated in differentiable parameterizations with the Score Distillation Sampling (SDS) loss derived from pre-trained 2D diffusion model. By combining SDS with various suitable representations (Lin et al., 2023; Chen et al., 2023; Wang et al., 2023b; Shi et al., 2023), this technique can generate high-fidelity 3D objects and scenes for a diverse set of user- ssor-provided text prompts.
Yet lifting 2D observations into 3D is inherently ambiguous. 2D diffusion models solely learn 2D priors from individual images and therefore lack 3D knowledge for disambiguation during the
Work done during internship at Tencent AT L.b.
Corresponding author

---

Mint Parts, Time-Traveling Hourglass, glowing with temporal evergy. intricale sand pafterws, mysterious device, db asset
Zombie bust, berror 123dsculpt bust, zomble
Glant Ice Golem, towering elemenisl guardian, frozer colossus, model
bulidoy wearing black pirate hat
Explorer's Exosuit, advanced astromaut armor, HUb visor, interstellar adventure, 3b asset
Skeampunle Clockoorkk Dragon, mechanical marvel, cogs and gears, industrial foutapy, 20 model
Fire-breathing Phoenis, mythical birch, sugulfed flames, rebirth renewal, 30 render, 8K, HD
angel, blander
Sleampunk Clockworkk Dragon, mechanical marvel, and gears, industrial fanlasy. model
Closerup rendering Woody from Tay Story, capturing geometiic shapes of kis cowboy hat and vest
A ab model of Simba, tie Lion cub from The Lion King standing majestically Pride Rock, charaeter
Albert Einstein with grey is riding bicycle
A DSLR photo of peacock surfboard
Jack Sparrow wearing sunglassed head, pholorealictie, HD
ab model of mini China town, highly debailed, blender ad.
kighly delailed,
Our work can generate high-fidelity and highly diversified 3D results from various text prompts, free from the notorious multi-view inconsistency problem. We highly recommend referring the supplementary materials for a more immersive viewing experience of the 3D results.
Figure 1:
2

---

lifting, leading notorious multi-view inconsistency problems, e.g., the multi-face Janus problem. While learning robust 3D priors from extensive and comprehensive 3D datasets is seemingly the very answer, in reality, are only presented with 3D data that is rather scarce compared to plentifully available images. Hence, a currently compelling direction is to incorporate 3D priors learned from relatively limited 3D data into 2D diffusion priors that possess high generalizability, thus obtaining the best of both worlds.
In particular, the issues related to multi-view inconsistency can be primarily categorized into two types: i) geometric inconsistency issues, that are caused by the ambiguity in the spatial arrangement of geometric structures, i.e., a geometric structure can position and orient differently in 3D. Impor- tamtly, geometry inconsistency is further exacerbated during the lifting by the supervision imposed by 2D priors that lack 3D awareness, where many irrational 3D structures resulting in identical 21) projections can deceive 21 priors; ii) appearance inconsistency issues, that arise due to the ambigu- ity in the mapping from geometric structures to corresponding appearance, and again, is exacerbated by the lack of 3D awareness in 2D diffusion for disambiguation. Empirically, we found the geometry inconsistency issue is the primary cause contributing to most multi-view inconsistent results within various existing methods, whereas the appearance inconsistency issue manifests itself alone in only extreme cases and thereby holds lesser significance. This evidenced by the fact that the majority of 3D inconsistent results exhibit repetitive geometric structures, typically multiple hands, or faces, that are generated under the guidance of 2D diffusion. It is worth noting that addressing these mis- placed structures plays a significant role in mitigating 3D inconsistency in the final outcomes, as the inclusion of geometric hints from 3D consistent geometries greatly aids the appearance modeling. This holds true for both one-stage text-to-3D pipelines (Poole al., 2022), where geometry and appearance are updated simultaneously, as well as pipelines that model geometry and appearance separately (Chen al., 2023; Richardson et al., 2023). However, it should also be acknowledged that there may still be exceptional circumstances where appearance inconsistency can manifest with 3D consistent geometric structures.
These findings have motivated us to prioritize addressing geometric inconsistency issues in text-to- 3D, by equipping the 2D priors with the capability to produce 3D consistent geometric structures while retaining their generalizability. In a way analogous to (Leike & Sutskever, 2023), we enforce the 2D geometric priors in diffusion models act in a way that aligns with well-defined 3D geometries as depicted in 3D datasets during the lifting, addressing the vast majority of the inconsistency prob- lem from the origin. We refer to the resulting geometrie priors as "AGP", for Aligned Geometric Priors. Specifically,
align the geometric priors by fine-tuning the 2D diffusion model to produce coordinate maps of objects in canonical space, thereby disambiguating the geometry distribution in 3D for ease of learning, and further conditioning on additional camera specifications, thereby conferring 3D awareness eventually. Notably, in stark contrast to methods that hinge heavily upon the geometric and appearance information in 3D datasets, we only capitalize on coarse geometries, avoiding over-reliance on geometric and visual appearance details that may further introduce unde- sired inductive bias. This "coarse" alignment of geometric priors not only enables the generation of 3D objects without the multi-view inconsistency problem but also retains the ability in 2D diffusion models to generate vivid and diversified objects unseen 3D datasets.
Finally, yet importantly, our AGP possesses high compatibility that is generally lacking in compet- ing methods. We show that AGP is highly generic and can be seamlessly integrated into various state-of-the-at pipelines using different 3D representations, obtaining high generalizability in terms of unseen geometries and appearances while significantly alleviating multi-view inconsistency. represents new ofate-of-the-art performance with 85+% consistency rate in human evaluation (see qualitative results gallery in Figure I and quantitative results in Table 1).
2 RELATED WORK
In the following, we mainly review related literature that exploit 2D priors learned in text- conditioned generative image models for text-to-3D, and refer readers to Zhang et al. (2023) for more in depth survey of text-to-iman -mmge diffusion models.
Text-to-DD using 2D Diffusson. Following successful text-to-imase image diffusion models, there has been a surge of studies that lift 2D observations in diffusion models perform text-to-3D synthesis,
3

---

bypassing the need for large-scale text-3D datasets for training scalable 3D diffusion models. In particular, the pioneer work by Poole et al. (2022) introduces a key technique - Score Distillation Sampling (SDS), where diffusion priors used score functions to supervise the optimization of a 3D representation. Concurrent with Poole et al. (2022), a similar technique is proposed in Wang et al. (2023a), which applies the chain rule on the learned gradients of diffusion model and back- propagate the score of a diffusion model through the Jacobian of a differentiable renderer to optimize 3D world. An explosion of text-to-3D techniques occurred in the community since then that improves the text-to-3D in various aspects, such as improved sampling shedules (Huang et al., 2023), adopting various 3D representations (Lin al., 2023; Tsalicoglou et al., 2023b; Chen et al., 2023), new score distillation (Wang et al., 2023b), etc. Although these methods have shown the capability to circumvent the necessity for text-3D data and generate photo-realistic 3D samples of arbitrary subjects with user-provided textual prompts, more often than not, they are prone to the notorious 31) inconsistency issucs. Hence, prior works have attempted to address the inconsistency with improved score functions and/or text prompts (Armandpour et al., 2023; Hong et al., 2023). Nonetheless, these methods cannot guarantee 3D consistency and tend to fail on consistent text-to-3D synthesis.
Concurrently, MVDream (Shi et al., 2023) addresses the multi-view inconsistency problem via train- ing a dedicated multi-view diffusion model, which simultaneously generates multi-view images that consistent across a set of sparse views. Particularly, they jointly fine-tune the diffusion model on real images and synthetic multi-view images, in order to inherit the generalizability in 2D diffu- sion and obtain the multi-view consistency in 3D datasets. Instead of relying on computationally intensive renderings and fine-tuning on both synthetic and real images, our method uses only low- resolution and low-cost geometry maps, and hence the "coarse" alignment of geometric priors is computationally efficient. Besides, our AGP is generic and can be seamlessly integrated into exist- ing pipelines to confer 3D consistency. Such compatibility is obviously lacking in MVDream.
Generative Novel Views with Diffusion Models Another route of 3D generation is to model it as a view-conditioned amage-to-image translation task with diffusion models, and directly generate novel views of the 3D, detouring the need for optimizing 3D world. Watson et al. (2022) train a pose-conditional image-to-image diffusion model to take a source view and its pose as inputs and generate novel view for target pose. This method has only been demonstrated on synthetic data in the ShapeNet dataset (Chang et al., 2015). Similarly, Zhou & Tulsiani (2023) build view- conditioned novel view diffusion model in the latent space and demonstrate its utility in sparse view 3D reconstruction. Chan et al. (2023) improve the view consistency of the diffusion model by reprojecting the latent features of the input view prior to diffusion denoising. More recently, Liu et al. (2023) propose variant of this technique for performing text-to-3D with the generative ability of novel views enabled by fine-tuning language- guided priors on renderings from 3D datasets.
Generally, models trained in these methods are unable to accurately capture the view specifications, resulting in the generation of multiple views that are only approximately 3D consistent. Although our AGP also takes as input camera specifications to be viewpoint-aware, its purpose is merely to generate coarse geometries that will evolve subsequently into 3D consistent object. Furthermore, aside from the differences in the formulation of these methods and ours on the text-to-3D task, our approach diverges significantly in that it does not capitalize on the appearance information, i.c., synthetic renderings, residing in 3D datasets, which is at the risk of compromising the visual priors learned in pre-trained diffusion models and thus may result in degraded visual quality.
3 METHOD
As aforementioned, the issues of multi-view or 3D inconsistency can be categorized from two per- spectives: geometric inconsistency issues, which pertain to misplaced geometric structures in 3D, and appearance inconsistency issues, which relate to incorrect visual appearance modelling on the 3D geometric structures. Realizing that geometric inconsistency is the main reason for most 3D inconsistent results, our goal is to equip the 2D priors with the capability to produce 3D consistent geometric structures while retaining their generalizability. As result, the generated consistent geo- metric structures play a continuous role in contributing to the modeling of intricate geometric details and visual appearance in pipelines that perform text-to-3D.
4

---

Canonical Coordinates Map (CCM)
Aligning Geometric Priors (AGP)
Integration into Text-to-3D
10.1.3)
11
DMTet
NeRF
Pre-trained SD Weights
M
UNet
Text
Hender CCM
Render
DreamiFusion
Sns
MagicIO
Nolse
MMP
Prolificdreamer
Fantasia30
TEXTura
CCM
AGP
11.00
Camera Extrinsic Param.
00 0)
Figure 2: Method overview. We fine-tune the 2D diffusion model (middle) generate viewpoint- conditioned canonical coordinates maps, which are rendered from canonically oriented 3D assets (left), thereby aligning the geometric priors in the 2D diffusion. The aligned geometric priors can then be seamlessly integrated into existing text-to-3D pipelines to confer 3D consistency (right), while retaining their generalizability obtain high-fidelity and highly varied 3D content.
To this end, we propose ensure the geometric priors in 2D diffusion act in way that aligns with well-defined 3D geometries as depicted in 3D dataset (Section 3.1). Specifically, we assume to have access 3D dataset, which comprises extensive and diverse 3D models that are canonically oriented and normalized. We then render depth maps from random views, and convert them into canonical coordinates maps. Note we only render from rather coarse geometries, as the goal is merely to use 3D data for aligning rather than for generating geometric details. The benefits of using such 3D data are two-fold: all geometries are well-defined in 3D, so there no ambiguity their spatial arrangement; ii) by further injecting the viewpoint into the model, we can confer viewpoint awareness and eventually 3D awareness. Then, we fine-tune the 2D diffusion model to generate the canonical coordinates map under specified view, eventually aligning the geometric priors in 2D diffusion. Finally, the aligned geometric priors can be seamlessly integrated into various text-to-3D pipelines (Section 3.2), significantly mitigating the inconsistency issues, resulting in the generation of high-quality and diverse 3D content. Figure 2 presents an overview.
3.1 ALIGNNING GEOMETRIC PRIORS IN 2D DIFFUSION
Then, we elaborate technical details involved in aligning the geometric priors in 2D diffusion with well-defined 3D geometries during the lifting, while retaining its generalizability.
Canonical Coordinates Map (CCM) To eliminate the distraction caused by the gauge free- dom and thereby ease the modeling, we assume that all objects within the same category adhere to a canonical orientation in the training data, common practice in various publicly accessible datasets (Deitke et al., 2023; Chang al., 2015). Note that, while the object orientation assumed to be canonicalized per category, our objective is not to learn category-speciff data priors. Instead, our aim is to extract general knowledge from variety of objects the 3D datasets, which will aid in aligning the 2D geometric priors. Analogous to (Wang et al., 2019; Shotton et al., 2013), the canonical object space defined as a 3D space contained within a unit cube {z, y, 2} E [0, 1]. Specifically, given object, we normalize its size by uniformly scaling the object such that the max and centered at the origin. While we can render
extent of its tight bounding box has a length coordinates maps at random views from these canonically oriented and uniformly normalized ob- jects for training, we further propose to anisotropically scale the three components in the coordinate maps rendered from an object, such that the value of each component is within the range from 0 to 1. This anisotropic normalization amplifies the discrepancy of spatial coordinates on thin structures at different views, easing the perception of the 3D structures and thereby improving the awaawreessss in the subsequent learning.
Camera Condition Although the canonical coordinates maps contain rough viewpoint informa- tion, we found that the diffusion model has difficulties in exploiting it. Therefore, we inject the cam- information into the model to improve viewpoinntawarrensss, following MVDream (Shi et al.,
5

---

2023). Specifically, explicitly input the corresponding camera extrinsic parameters to the diffu- sion model, which is passed through an MLP before being fed to the middle layer of the diffusion model. Note that, in contrast to other models that rely accurate viewpoint-awareness for gener- ating consistent 3D, the use of camera conditions in our model is only to roughly generate coarse geometries that will evolve subsequently into a 3D consistent object.
Fine-tuning 2D Diffusion for Alignment Given the pairs of the canonical coordinates map and its corresponding camera specification, we keep the architecture of the 2D diffusion model while slightly adapting it to be conditioned on camera extrinsic parameters. This enables us to leverage the pre-trained 2D diffusion model for transfer learning. thereby inheriting their generalizability in terms of highly varied subjects unseen in the 3D dataset. Finally, we fine-tune the diffusion model, originally intended for generating raw RGB or latent images, to generate the canonical coordinates map under a viewpoint condition, eventually aligning the geometric priors in 2D diffusion.
Implementation Details By default, we conduct experiments based on the Stable Diffusion model (we use v2.1), which is commonly used public large pre-trained text-to-image diffusion model.
3D dataset. We use a public 3D dataset- Objaverse (Deitke et al., 2023), which contains around 800k models created by artists, generate the data for fine-tuning.
By having a significant portion of the 3D objects in canonical orientation and only a few remaining misoriented,
able to achieve satisfactory outcomes without the necessity of manually cor- recting those misoriented ones. On the other hand, due to the presence of considerable noise in the textual annotations, we employ 3D captioning model (Luo et al., 2023) to augment the textual description of each 3D asset and randomly switch between the augmented caption and its original textual annotation (typically names and tags) during the training. In addition, to ensure relevance, we apply filtering process based on tags to eliminate 3D assets such as point clouds and low poly models, resulting in approximately 270k objects.
Camera sampling. We render canonical coordinates maps from the 3D objects. The camera is randomly positioned at a distance ranging from 0.9 to 1. units, with a field of view set at degrees. Additionally, the camera's 's elevation is randomly varied between-10 -10 and 45" As we do not rely on visual appearance information, we are able to utilize a fast rasterization renderer for generating the training data, avoiding the computational intensity associated with ray tracing renderers.
Training. We fine-tune our model on the latent space of Stable Diffusion using Diffusers (von Platen et al., 2022). Note that the canonical coordinates map is directly treated as a latent image for the latent diffusion model to produce. This leads to an attractive feature that our aligned geometric priors can be trained fast, without involving the encoding and decoding process of the VAE. We keep the default optimizer setting, as well as the e-prediction. Since we input the camera extrinsic parameters condition to the diffusion model, the training objective is now formulated as follows:
LLDM := B........-N(...) ||c-co(c, (v)),, 2e, t) 1 where c is the camera extrinsic parameters,
the input text prompt and To (y) its embedded feature using tokenizer, and 2, the noisy latent image generated by adding noise clean latent image diffusion timestep f.
3.2 INTEGRATION INTO TEXT-TO-3D
Finally, we elaborate on how to integrate our aligned geometric priors into existing pipelines us- ing different 3D representations, significantly mitigating their inconsistency issues and achieving state-of-the-ard text-to-3D performance. To showcase such compatibility, we provide demonstra- tions of two state-of-the-art text-to-3D methods that utilize different 3D representations, namely, Fantasia3D (Chen et al., 2023), which explicitly disentangles the geometry and appearance model- ing and uses a hybrid representation - DMTet (Shen et al., 2021) - for the underlying 3D geometry, and DreamFusion (Poole et al., 2022), which employs the neural radiance field (NeRF) (Mildenhall et al., 2020) as the 3D representation. Please Figure 3 for the system pipelines of integrating our aligned geometric priors into these two methods. For more technical details, we refer readers to the original papers, we do not elaborate them further in this context.

---

DMTet-based Text-to-3D Pipeline
NeRF-based Text-to-3D Pipeline
Gromety Modelag
Orggnaih sn
Doo0ll
Instant NGP
100000
Nonier
NOTral
Reiderring
FGR
Muokiing
Camers Earrinte Faraon
Crisea Fastinte
Upiiae
Upeate
Figure Seamless integration of our AGP in various text-to-3D pipelines.
DMTet-based Pipeline For the sake of clarity, we refer to the variant obtained by integrating aligned geometric priors into Fantasia3D as the DMTet-based pipeline. All that is required an additional parallel branch, to incorporate our aligned geometric priors for supervising the geometry modeling in the original pipeline. With this seamless integration of our aligned geometric prior, high-quality and view-consisten results can be easily achieved, without the need for carefully designed initialization shapes in the original pipeline.
Optimization. Specifically, we add this additional supervision imposed by our aligned geomet- ric priors in both the coarse and fine geometry modeling stages. Simply, our aligned diffusion model takes as input the canonical coordinates map and produces the SDS loss to update the 3D representation. Then, the final loss function in the geometry modeling can be written as LsDS = X SDS geo + Xalign Colign from the original diffusion model, the latter is the SDS loss derived from our aligned geometric pri- ors. Here, Xori and aliggn are the weights to balance their effects. The revised system pipeline is shown on the left of Figure 3. Note that this integration implemented only in the coarse and fine geometry stages, while the appearance modeling stage is untouched.
-go' where the first term is the geometry SDS loss derived
NeRF-based Pipeline NeRF is another common choice for the 3D representation in text-to-3D, as it more friendly for optimization compared traditional discrete meshes, and can also be combined with volume rendering for great photo-realism. Specifically, we base on a popular imple- mentation (Guo et al., 2023) of the pioneer DreamFusion (Poole et al., 2022), which uses NeRF as the 3D representation, and refer to it as the NeRF-based pipeline. Particularly, the 3D scene is represented by Instant-NGP with an extra MLP for modeling the environment map, allowing the modeling of rich details with low computing cost. Then we can volume-render the 3D object/scene to obtain the RGB images and feed them into the Stable Diffusion to calculate SDS loss.
Optimization. During the lifting optimization, we render the canonical coordinates map and feed it to our aligned geometrie priors to calculate the geometry SDS loss LSDS-geo help update the ge- ometry branch of the NeRF, in addition to the origin SDS loss LSDS calculated with the RGB image. Similar to the previous integration, the final loss is the weighted combination of the original SDS loss and our aligned geometric SDS loss: LSDS = Xori Lsbs + Xalign C where ori and allign are the weights balancing these terms. Note our AGP continues model 3D consistent coarse geometries this pipeline, while again leaving the appearance modeling untouched.
4 TEXT-TO-3D GENERATION
We present the qualitative and quantitative evaluation of the text-to-3D pipelines described in Sec- tion 3.2, as well as comparison results against other text-1o-3D baseline methods. For convenience and clarity, refer to the DMTet-based pipeline and NeRF-based pipeline Ours (DMTet-based) and Ours (NeRF-based) respectively. Furthermore, depending on the different pre-trained diffusion models used in its original pipeline, developed two versions of Ours (NeRF-based) namely, Ours (NeRF-baser IF) using Deepfloyd IF, and Ours (NeRF-based baseed full) using DeeFloyd IF first and then Stable Diffusion. Please refer to the supplementary for more details.
7

---

Dreamfusion-IF
Magic3D-iF
TextMesh--F
S/C
Ours (NeRF-based
Ours (NeRF-based full)
Fantasia3D
Ours (DMTet-based)
A ceramic Lion
A crocodile playing drum set
dragon-cat kybrid
DSLR photo of bear dressed in medieval armor
Figure 4: Visual comparisons. Compared to other competing methods, our text-to-3D pipelines can generate high-fidelity 3D content with high 3D consistency. See 3D inconsistency issues in the baseline results (highlighted by red arrows).
Baselines We extensively compare to baselines follows: i) Fantasia3D, based on which our DMTet-based pipeline implemented. We compare our DMTet-based pipeline against it to show specifically the effectiveness of our AGP; ii) DreamFusion-IFF which replaces the unreleased Ima- gen (Saharia et al., 2022a) with DeepFloyd IF (IF, 2023). We compare our NeRF-based against validate our AGP again; And more other baselines implemented in (Guo et al., 2023), including iii) SJC (Wang et al., 2023a), that applies the chain rule on the learned gradients of a diffusion model and backpropagates the score of diffusion model through the Jacobian of differentiable renderer to optimize 3D world; iv) Magic3D-IF (Lin et al., 2023) is a hybrid pipeline that uses NeRF at the coarse stage and then converts into a mesh for fine details. We also adapt it to use DeepFloyd IF for SDS calculation; v) TextMesh-IF (Tsalicoglou et al., 2023a), which is also a DeepFloyd IF-based implementation, is similar to Magic3D but uses an SDF-based representation at the coarse stage. vi) us. Note that, since their official imple- mentation is unavailable by the time of our submission, we use the same prompts as listed on their website for side-by-side comparisons.
MVDream (Shi et al., 2023), which is concurrent work to
Due to various reasons, we were not able to obtain the original implementation most baselines. Therefore, except for Fantasia3D and MVDream, we use the implementation from Guo et al. (2023) for all baselines. We consider these implementations to be the most reliable and comprehensive open-source option available in the field. By default, use the Stable Diffusion model as the prior, except those with the name suffixed "IF" use DeepFloyd within the pipeline.
Quantitative Evaluation It is important acknowledge that currently, there is lack of well- established metrics that can quantitatively and comprehensively evaluate the text-to-3D results from various perspectives.
this work, our primary focus lies in generating multi-view consistent 3D) content, rather than placing specific emphasis on enhancing the appearance or texture quality of existing pipelines. So we focus on quantitatively evaluating the multi-view consistency of 3D results. Specifically, we randomly select 80 text prompts from the DreamFusion gallery (dre, 2023), and perform text-to-3D synthesis to generate 80 results using each method. We then manually check and count the number of occurrences of 3D inconsistencies (e.g.. multiple heads, hands, legs) and report the success rate, i.c., the number 3D consistent objects divided by the total number of generated results. Note that, while the 3D consistency be vague sometimes describe, the
8

---

human participants involved are researchers with highly relevant backgrounds such as Computer Vision and Computer Graphics, so they roughly follow a general definition of 3D consistency in the evaluation. As shown in Table I, our method outperforms other methods by a large margin. Our success rates are over 85+ % in both pipelines, while the previous methods are only around 30%.
Dreamfusion
Magic3D
TextMesh -IF
Ours (DMTet-based) (NeRF-hased IF)
Ours
SIC
Fantasia3D
-IF
-IF
Cons. Rate
30.0%
35.0%
23.8%
7.5%
32.5%
87.5%
88.8%
Table 1: Quantitative comparison results for the 3D consistency rate.
Qualitative Evaluation As shown in Figure By integrating our AGP into Fantasia3D, i.e., Ours (DMTTe--bsee)), the results have been significantly improved. The original Fantasia3D only pro- duced coarse and inaccurate results without hand-crafted initial geometries. We believe this is due to the domain gap between the rendered normal map and the geometric information extracted in the latent space of the Stable Diffusion, resulting in optimization difficulties in converging able 3D shape. As for Ours (NeRF-basede the generated results clearly have high 3D consistency and possess a more realistic appearance. This is because our aligned geometric priors only con- tribute to the geometry modeling during the lifting, while they do not compromise the appearance modeling guided by the powerful visual priors learned by Stable Diffusion from billions of real im- ages. In general, the 3D results generated by most remaining baselines, even when equipped with the more powerful Deepfloyd IF, suffer from multi-view inconsistency (casier to see from spinning views in the supplementary file). Note that, when evaluating the results, we focus on assessing the 3D consistency, and hence do not heavily penalize blurry images, as they can be caused by the use of DeepFloyd IF with limited computing resources. Last, although the concurrent work, MVDream, can also resolve the multi-view inconsistency problem, we observe that it is prone to overfit the lim- ited 3D data, consequently resulting compromise of the generalizability the original powerful 2D diffusion model. Since they use synthetic multi-view renderings for fine-tuning their multi-view diffusion model, the appearance of the generated results lacks the desired level of photo-realism.
reason-
User Study We also conducted user study on 30 generated 3D results of relevant methods. Each participant was presented with videos rendered from the 3D models obtained by various methods based on one of the 30 text prompts. Then, they were requested to choose a 3D model they favored the most, considering only the 3D consistency. We report the rate of preference for each method in the inset pie chart. As shown, while this evaluation is our method consistently outperforms the competing methods by large margin, showing the robustness of our method in generating results of high 3D consistency.
Farfasiegg
Ours
|NeRF- based)
Magicab
Ours (DMTet based)
noisy, yet
TooMen
5 CONCLUSION
We introduced Aligned Geometric Priors (AGP), which is obtained by fine-tuning a pre-trained 2D diffusion model to generate condpointcooniitioned coarse geometric maps of canonically oriented objects, thereby conferring 3D awareness. AGP is generic and can be seamlessly integrated into various existing pipelines to generate 3D objects of high consistency. Most importantly, AGP im- proves the geometry modeling and does not compromise the appearance modeling guided by strong priors learned from billions of real images. While AGP has shown ofate-of-the-an performance in text-to-3D, we also note few limitations. Our work does not directly consider appearance model- ing, where inconsistency may still arise rarely due to the remaining ambiguity in the mapping from the geometric structure to its associated appearance. Early in our development, we attempted to in- corporate an appearance generator by fine-tuning the 2D diffusion model to generate the appearance image conditioned on given canonical coordinates map. Unfortunately, this approach resulted in overfitting to the renderings derived from 3D data, leading to 3D results that lacked the desired level of photorealism. We leave the study in this direction for future work. All that being said, we believe the work opens up a novel "less is more" direction of utilizing relatively limited 3D data to enhance 2D diffusion priors for text-to-3D synthesis.

---

ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers for their constructive comments, We thank Jianx- iong Pan for his effort in data preparation. Weiyu was partly supported by the Shenzhen Collab- orative Innovation Program (CJGIZDD002044499260000000
REFERENCES
Dreamfusion project webpage, 2023. URL URLAtpp:://rreemtiiiioa....
.i/.
Mohammadreza Armandpour, Huangjie Zheng, Ali Sadeghian, Amir Sadeghian, and Mingyuan Zhou, Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond. arXiv preprint arXiv:2304.04900 2023.
Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Genvs: Generative novel view synthesis with 3d-aware diffusion models, 2023.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rice 3d model repository. arXiv preprint arXiv:1512.03012. 2015.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In International Conference Computer Vision (ICCV), October 2023.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of an- notated 3d objects. In Conference Computer Vision and Pattern Recognition (CVPR), pp-
13142-13153, 2023.
Yuan-Chen Guo, Ying-Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia- Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang. threestudio: A unified framework for 3d content generation. https://gia tttbtttttuuuggggo com/ threestudio- projitt/ threestudio, 2023.
Susung Hong, Donghoon Ahn, and Seungryong Kim. Debiasing scores and prompts of 2d diffusion for robust text-to-3d generation. arXiv preprint arXiv:2303.15411 2023.
Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi, Zheng-Jun Zha, and Lei Zhang. Dream- time: An improved optimization strategy for text-to-3d content creation. arXiv preprint arXiv:2306. 12422, 2023.
DeepFloyd IF. Deepfloyd if, 2023. URL https::/hhigiinggaaat Batetaacccco/DDeeeeeeee co/.eeeeeeooddd
Jan Leike and Ily Sutskever. Introducing superalignment, 2023. URL https:/::// com/
2llgg/nntrode sug.engggglaeeeeiggeeeeeeee
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: Zero-shot one image to 3d object. arXiv preprint arXiv:2303.11328, 2023.
Tiange Luo. Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pre- trained models. arXiv preprint arXiv:2306.072 2023.
B Mildenhall, PP Srinivasan, M Tancik, JT Barron. R Ramamoorthi, and R Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. European Conference on Computer Vision (ECCV). 2020.
10

---

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphies prim- itives with a multiresolution hash encoding. ACM Transactions on Graphics (SIGGRAPH), 14):102:11102:15, July 2022. doi: 10.144535282333331227 htts::::: //ooi//////// 10. 1145/3528223 35301
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference Machine Learning (ICML), pp. 16784-16804. PMLR, 2022.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference on Learning Representations (ICLR), 2022.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning (ICML), pp. 8821 1-331. PMLR, 2021.
Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text- guided texturing of 3d shapes. arXiv preprint arXiv:2302.01221, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Conference Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems (NeurIPS), 35:36479-36494, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems (NeurIPS), 35:36479-36494, 2022b.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023.
Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2930-2937, 2013.
Rakesh Shrestha, Siqi Hu, Minghao Gou, Ziyuan Liu, and Ping Tan. A real- worldwrrd dataset for multi- view 3d reconstruction. In European Conference on Computer Vision (ECCV), pp. 56-73, 2022.
Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. 2023a.
Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. arXiv preprint arXiv:2304.12439. 2023b.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the- diffusion models. https:
//gi thub. cwm/cooo///oooooo hoog//uugii ing face/ di fusers, 2022.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12619- 126299 2023a.

---

He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Conference Computer Vision and Pattern Recognition (CVPR), pp. 2642-2651, 2019.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305.1622 16213, 2023b.
Daniel Watson, William Chan, Ricardo Martin Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. In International Conference on Learning Representations (ICLR), 2022.
Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang. Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, al. Omniobject3d: Large-vocabulary 3d object dataset for realistic percep- tion, reconstruction and generation. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 104 2023.
Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In Kweon. Text-to-image diffusion model in generative ai: A survey. arXiv preprint arXiv:2303.07909 2023.
Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12588- 12597, 2023.
12

---

A APPENDIX
A. ABLATION STUDY
We also validate the effectiveness of key algorithmic design choices using the NeRF-based pipeline. Regarding the geometric representation, we have also derived variants of our method using two types of geometric representations, namely depth and normal maps, and show that, while these two geometric representations can also be incorporated into our pipeline for aligning the geometric priors in 2D diffusion, they indeed fail in producing 3D results as high-quality as CCM.
On one hand, a depth map is a single-channel map storing the per-pixel distance between the sur- face to the camera. Compared to CCM, the depth map alone does not contain essential information about the relative orientation of the 3D object and the camera, the network can solely rely on the camera information for learning 3D-aware priors, consequently leading to degraded robustness of the learned 3D priors. In stark contrast, the CCM alone has actually encoded the information of the orientation to some extent. This is manifested by the distinct color coding results of CCMs from dif- ferent viewpoints. As result, the variant using depth maps is more prone multi-view inconsistent issues, compared to our method using CCM. On the other hand, a normal map stores the local direc- the 3D surface, we normal maps obtained from the canonical coordinate
tions of fragments space for fair comparisons. Our experimental results show that the text-to-3D optimization process has difficulty in generating a 3D world of which the derivative follows faithfully generated normal maps, and may fail to generate meaningful 3D results. Those successfully generated 3D results tend to be slightly smoother regarding the geometric and appearance details of the surface.
Last, we have also conducted a study where the camera information injection is removed from our method. The results demonstrate that removing camera parameters leads to a decrease of the consistency in the generated results, due to the lack of critical camera information. Please see Figure for the visual results and refer to Section A.6 for more.
Guided
Depth Guided
Normal Guided
Embedding
A DSLR photo of a dog made out of salad
Figure 5: Ablation study. Replacing CCM with depth maps results in inconsistent geometry and the use of normal maps as guidance produces smooth geometry. Additionally, removing the camera embeddings led to noisy and inconsistent results.
A.2 MORE IMPLEMENTATION DETAILS
Dataset Notably, while there is no explicit specification of the coordinate system in Objaverse, most 3D objects Objaverse uploaded by artists who usually adhere to convention regarding the orientation when creating 3D assets. Furthermore, for some special categories, such as char- acters, we filter out misoriented data simply by the ratio of its axis-aligned bounding box. After these filterings, we found that approximately 80% (based on statistical random samplings) of the remaining data are orientated canonically, which sufficient for our purpose as evidenced by exten- sive results in our paper (please also refer to the appendix for more discussions on the noise in the dataset).
Fine-tuning We use Diffusers (von Platen et al., 2022) to finetune the Stable Diffusion v2.1 us- ing our rendered CCMs at a resolution of 64 x 64. During the fine-tuning process, we remove the encoder of the VAE and directly concatenate the CCM with corresponding alpha channels as the latent to the UNet, with the background color of the CCM set to random. We use the default param- eters as in Diffusers, including setting the learning rate to le-5 with the constant scheduler, and
13

---

batch size of 96 per GPU with 4 gradient accumulation steps. The entire fine-tuning process takes approximately 2 days using 8 V100 GPUs for 100k steps.
Ours (DMTTe--based) We integrate our Aligned Geometric Priors in the official repository of Fan- tasica3D (Chen et al., 2023) as described in Section 3.2. We follow the same parameters as in the original paper. We also disentangle the learning of geometry and appearance. It takes about 12 and 8 minutes to generate fine geometry and its corresponding Physically-Based Rendering (PBR) materials, respectively, for each object. For the time step range of SDS loss, We adopt uniform sampling strategy of annealing from [0.5, 0.98] to [0.05, 0.5). The whole process takes about 0.5 hours to generate each object using 4 V100 GPUs.
Ours (NeRF-based full) We implement it in the threestudio (Guo et al., 2023), which imple- mented a diverse set of state-of-the-arr text-to-3D generation pipelines. Specifically, we use Instant- NGP (Müller al., 2022) as the 3D representation to optimize, which uses musti-ressoluton hash- grid to predict the RGB and the density of the sampled ray points. The sampled camera views follow the same protocol as the render dataset to fine-tune the UNet. We use DeepFolyd at the coarse stage with 64 x 64 resolution and then switch to Stable Diffusion with 512 512 for detailed optimiza- tion. In addition, we also use time annealing, negative prompts, and CFG rescaling tricks from open source implementation for improved performance. For SDS, the maximum time step is decreased from 0.98 to 0.5 linearly and the maximum time step is kept to 0.02. We use a rescale factor of 0.7 for the CFG rescale. The whole process takes about hour generate each object with 10, 000 steps using 2 V100 GPUs.
A.3 MORE COMPARISON RESULTS USING PROMPTS FROM MVDREAM
Note that, since MVDream's 's official implementation is unavailable by the time of our submission, we use the same prompts as listed on their website for side-by-side comparisons. We present the visual comparisons in Figure 6. Although the concurrent work, MVDream, can also resolve the multi-view inconsistency problem, we observe that it is prone to overfit the limited 3D data, con- sequently resulting in a compromise of the generalizability in the original powerful 2D diffusion model. Specifically, as shown in the results, MVDream misses the "backpack" in its generated re- sult presented with the prompt "an image of a pig carrying a backpack". Additionally, since they use synthetic multi-view renderings for fine-tuning their multi-view diffusion model, the appearance of the generated results lacks the desired level of photorealism.
A.4 GENERALIZABILITY
Our method can effectively address the notorious multi inconsistency problem, and equally im- portantly, retains to the maximum extent the generalizability of the foundation text-to-image model in terms of the highly varied appearance and geometric details. We would like to highlight that the pre-trained text-to-imane toagg diffusion model is powerful, and this preservation of its generalizability is particularly attractive, as it can lead to more diverse and highly realistic 3D generation results. We have achieved this by only aligning the geometric priors in 2D diffusion using the coarse geometric information (CCM) of well-defined geometries, without compromising the original diffusion priors in the text-to-3D pipeline regarding detailed appearances and geometries. This in contrast to other models that hinge on all appearance and geometric details in the 3D dataset for fine-tuning the diffu- sion priors, which is at the risk of compromising the integrity of the original geometric priors learned in the pre- trained eextte--mage foundation model, leading to the degradation of the generalizability in terms of highly diverse and photo-realistic 3D objects. While we have validated this through the comparison results against MVDream (Figure 6), we have also prepared more extensive qualitative results to further showcase such ability of our method to generalize smoothly to 3D worlds with highly realistic and varied appearance and geometric details, that are unseen in the 3D dataset. See Figure for more visual results.
More specifically, the generalization encompasses both geometric and appearance detils. In our work, we only perform fine-tuning using the coarse geometric structures, which enables us to pre- serve the highly realistic appearance details of the original image priors learned from large real- world dataset. Interestingly, we found that in our result of using the prompt "corgi riding on a rocket", the rough geometry obtained with only AGP priors is not yet completely accurate. How-
14

---

Dreamfusion-IF Magic3D-IF
TextMesh-IF
ProlificDreamer MVDream
Ours(DMTettbasee)
Ours(NeRFF-ased)
Corgi riding rocket
A DSLR photo of a peacock on a surfboard
A pig wearing a back pack
DSLR photo of squirrel playing guitar
Figure 6: Side visual comparisons using prompts from MVDream. Note that some key concepts in the prompts missing MVDream results, such as the rocket, backpack, and squirrel missing their results.
Terracotta warriors sitting on the safa
A DLSK photo of the head of Steve Jobs, 4K
dragon-cat hybrid
futuristic city filled with high-tech ...
Figure 7: Our method can produce wide range of 3D results spanning from weird characters fancy architectural models. worth noting that none of these 3D models exists in the training data. Our approach can generalize very well these highly diverse and high-quality results.
ever, after optimizing with SDS of image priors, we are able to fix the slightly incorrect geometry and obtain more detailed geometric and appearance details, shown in Figure 8.
A.5 THE EFFECT OF DIFFERENT SETTINGS OF DATASET
We used a set rules to filter the original dataset and obtained approximately 270k objects, which approximately 80% have consistent orientation (estimated based statistical random samplings). In this section, we aim to investigate the impact of different-sized dataset on the performance of proposed method.
15

---

AGP SDS only
AGP DeepFloyd SDS
AGP DeepFloyd SDS SD2.1
wrong geometry
Figure 8: Visualization of each stage of NeRF-based method of prompt "Corgi riding rocket". Using only AGP SDS as supervision may result in incorrect geometry. However, the image priors can help eliminate these errors in the later stages, thereby improving the generalization performance.
We calculated the distance between the front-view images and their corresponding captions' blip feature and sorted them accordingly. We then manually screened the top 20k data, resulting dataset of 18,488 instances with uniform orientation and high correspondence with their descrip- tions.
From the experimental results obtained by training on such a smaller dataset, we are surprised to find that even with smaller dataset of only 20k instances for fine-tuning, our text-to-3D pipeline still exhibits strong generalization and is capable of generating diverse 3D objects. As shown in Figure 9, the majority the generated results are satisfactory. with some lacking intricate details. We hypothesize that this may be attributed to the inadequate acquisition of 3D prior knowledge from a rather limited dataset, which consequently leads to the occurrence of coarse geometries. This observation necessitates the need for further exploration and more thorough study of the impact of the 3D dataset.
A.6 MORE DETAILS OF ABLATION STUDY
We conducted ablation experiments using manually curated small dataset 20k data) for each object. We rendered CCM, normal, and depth maps for each object and used the same parameters to fine-tune the Stable Diffusion model. We then tested using the same prompt and seed. Extensive visual results are presented in Figure 10.
A.7 QUANTITATIVE EVALUATION OF THE APPEARANCE GENERATED OBIECTS
We also present quantitative results using appearance-related metrics. However, it is worth noting that the image diffusion models used in Dreamfusion and Magic3D are open-sourced, and the performance of the versions used here may differ from the original papers. For faimess, we used the same image generation model, DeepFloyd IF (IF, 2023) for all methods, and NeRF as the 3D representation.
To assess the appearance quality, we rely on DreamFusion's R-Precision metric. When provided with rendered images, R-Precision gauges the top-N accuracy in retrieving the correct caption from a pool of distractions, utilizing CLIP scores derived from averaging the similarity between each of four distinct rendered images and caption. We employ the CLIP ViT-B/32 and ViT-B/16 models to respectively compute the top-1 and top-5 R- R-Precision, using the identical set 80 prompts for evaluation presented in the paper. Please note that since other methods may not have access to the original image diffusion model, the performance of reproduced code may vary. Therefore, the specific numerical results can only serve reference.
16

---

Finetuned using 270k data
Finetuned using 20k data
3D model of mini China town, highly detailed, 8K, HD, blender
Steampunk Clockwork Dragon, mechanical marvel, cogs and gears,
Albert Einstein with grey suit is riding bieycle
Figure 9: Comparison of finetuning results with different amounts of data. Despite reducing the training data to 1/10 of the original amount, the finetuned models still exhibit strong generalization capabilities (right). Note that, the model fine-tuned on fewer data tends produce results of slightly fewer details and probably a bit more bulky geometries.
CLIP R- R-Precision %)
CLIP
B/32
CLIP B/16
Method
R@1
R@5
R@1 1
R@5
Magic3I
60.1
71.5
62.7
77.7
TextMesh
51.7
65.1
55.1
78.4
SJC
40.2
51.2
52.5
62.5
DreamFusion
59.7
70.2
61.6
74.3
Ours
77.5
84.9
88.7
92.3
Table 2: Quantitative results demonstrating the coherence of visual appearance with their corre- sponding prompts, assessed by CLIP retrieval models.
The results highlighted in Table underscore the significant advantages of our proposed approach in terms of appearance. This superiority primarily stems from the fine-tuning process which only learns from coarse geometries and leaves the appearance model of the text-to-3D pipeline untouched.
A.8 MORE DETAILS ABOUT THE USER STUDY
In contrast to the quantitative evaluation above, the human users involved in this study are from various loosely related backgrounds in CS and EE. The interactive interface used in the user study can be found in Figure 11.
A.9 MORE TEXT-TO-31 RESULTS
We present more text-to-3D synthesis results obtained with methods (Figure 12, Figure 13, Figure 14, and Figure 15).
17

---

Albert Einstein with grey suit is riding bicycle Ours CCM Guided (20k data)
Depth Guided
Normal Guided
w/o Camera Embedding
Gandalf wearing a grey cloth wizard hat, white_hair, head
Fire-breathing Phoenix, mythical bird, engulfed in flames
pig wearing a back pack, 4K, HD'
Comparing ablate settings with different guidance methods. Our proposed CCM proves superior to depth and normal guidance, delivering consistent and convincing geometry. Substituting CCM with depth maps leads to round (e.g. the pig) and inconsistent objects (e.g. the Gandalf and phoenix), while normal maps as guidance make the optimization unstable (e.g. the Gandalf, Einstein, and pig). leads smoother (e.g. the phoenix) noisy (e.g. the chihuahua) geometry. is essential to note that removing the camera embedding may result inconsistent results, particularly on complex structures (c.g. the Einstein).
Figure 10:
18

---

Text-to-3D Generation
Text-to-3D Generation
Heese select the resut which has the best cornistency from the video on the left side.
Please select the result which has the best consistency from the video on the leftside. "Meue teren they do main dncrptirr grmmets
er ret dest plue. eeueretyll
2
A baid sagle carved out wood
A boagle in dotection's outfit
desective's
Abkhhon frke wearing scademic...
bichon frise wibklng ocodemc...
lian
Figure 11: User study interface. We have designed two survey forms separately for the NeRF-based method and the DMTet-based method and provide users with multiple rendered videos of generated results. The user needs to select the one they consider to be of the best quality from all results.
19

---

beautiful rainbow fish
Mystical Crystal Garden, enchanted flora, radiant and magical, secret botanical wonders, 3D asset
Interstellar Fortress, space citadel, advanced technology, defensive weaponry, highly detailed, 3D model
delicious chocolate brownie dessert with ice cream on the side
classic Packard car
statue of angel, blender
Army Jacket, 3D scan
Anclent Mayan Calendar, intricabe glypks, astronomical precision, kistorical artifact, 3b model
Acrial view of ruined castle
Steampunk Clockwork Dragon, meckanical marvel, cogs and gears, industrial fantasy, 3D model
Figure 12: More generated results using our proposed DMTet-based model.
20

---

Ancient Roman Colosseum, kistoric arena, architeccural wonder, gladiakors and spectackes, 3D render
mini China town, highly defailed, blender 3d
Enchanted Elven Citadel, ethereal fortress, magical spires, stronghold, asset
crab, Low paly
model of Darth kelmet, highly detailed
A bulldoga wearing a black pirate hot
A 3b model of Flying Dragon, highly detailed, breathing fire
Mystical Bow, ethereal craftsmanship, enchanted arrows, forest protector, asset
Space Explorer's Exosuit, advanced astromaut armor, HUD visor, interstellar adventure, 3b asse!
A bear dressed in medieval armor
Figure 13: More generated results using our proposed DMTet-based model.
21

---

Closerup rendering Woody from Toy Story, capturing geometric shapes comboy hat and
Albert Einstein with grey suit riding bicycle
Ckibi Deadpool, pixar style, modern Disney style, 8k, studio light, cinematic, kyperrealiatic, octane render
Portrait of Harley Suinn, capturing geomekric shapes of her colorful kair, makeup, and playful expression
charming chibi-sbyle renderung of Elsa and Anna from Frozen, featuring geomelric shapes and their heartworming sisterly bond, 8K resolution.
model Simba, the cub from Lion King, standing majestically Pride Rock, character
Photorealistic full-body represeenationn of Captain Sparrow, complete with geometric details
À ab model mint China bown, highly detailed, &K, HD, blender 3d
Picture of Leaning Tower Pisa, featuring structure marble facade
3D model of Doctor Strange, the Sorcerer Supreme, wearing his iconic Cloak of Levitation and holding the Eye of Agamatto amulet
Figure 14; More generated results using our proposed NeRF-based model.
22

---

smashing Ehrough a wall, showeasing muscular physique powerful pose in photorealistic
4K debail
Fisherman House, cute, cartoon, blender, stylized
Parls, highly detailed, 8K,
Image Michael Jackson, showcasing signature dance moves, fedora hat, and stylish wardrobe
Scene of the Temple of Heaven in Beijing, displaying its circular architecture and OrnaGE details
Fire-breatking Phoenix, muthical bird, engulfed flames, rebirth and reuewal, render,
View of Sydneey Opera House, showcasing unique sail-likee design and waterfront location
Detailed headshot Thor, of ThuAder, empkasizikng geomelric shapes of hls majestic intense gaze
Higly detailed, majestic royal ship, realistic painting
Floaking Steampunk City, gears and balloons, Vickorian-era airship metropolis, render,
Figure 15: More generated results using our proposed NeRF-based model.
23


Reviews:
[{"review_id": "7nQm6nXU8g", "reviewer": "Reviewer_Q6rV", "summary": "Text-to-3D is now a very very hot topic, starting from Dreamfushion. Nowadays, there are already many works which demonstrate very good quality of results. However, existing methods undergo a severe issue: the SD lacks view information of the 3D objects, making \"multi-face\" issues are usually caused. This work presented Canonical Coordinates Map as a geometric representation and train a geometric-aligned prior resorting to the recent public 3D dataset-Objaverse. The results are of very high quality.", "questions": "see above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- First, the results are of very high quality. \n- The paper writing is good and the motivation is strong. \n- The proposed method is novel.", "weaknesses": "- My major concern is about the necessity of the proposed CCM. It seems CCM is one of the geometric prior representation. I am curious what will happen if we replace CCM by just using normal map or depth map as a geometric representation? More specifically, an extra experiment is needed: using objaverse to train a normal/depth map (instead of CCM) generative model and also include camera condition.  \n- My another question is: if objaverse is used to finetune SD for CCM generation, will the generalization ability be decreased? As objaverse is lacking diversity especially on complicated scenarios. More discussions are encouraged.\n- In Fig 1, it seems the normal map owns many bumps, what are the details about the normal rendering?\n- It is said in the implementation details \"a significant portion of the 3d objects in canonical orientation and a few misoriented\". The authors mentioned it will not affect the results. I think more detailed explanations are needed, for example, what does it mean about mis-oriented? how many objects are mis-oriented？ if the mis-oriented cases are corrected manually,  will the performance be improved further?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FYxaxqHtla", "reviewer": "Reviewer_nptq", "summary": "This paper proposes aligned geometric priors (AGP) to address the multiview inconsistency problem in diffusion models. In detail, the method finetunes a pre-trained 2D diffusion model to generate viewpoint-specific coarse geometric maps of canonically oriented\n3D objects to ensure 3D awareness. Qualitative experiments and human evaluation results show great geometry consistency of the proposed method.", "questions": "1. The questions in the Weaknesses section.\n\n2. \"Ours (NeRF-based full) using DeeFloyd IF first and then Stable Diffusion\". How is it implemented in detail? What is the method (Ours (NeRF-based)) used in user study (NeRF-based IF or NeRF-based full)?\n\n3. How are $\\lambda^{ori}$ and $\\lambda^{align}$ chosen?\n\n4. The user study interface can be shown in the appendix (How did the interface show users to \"consider only the 3D consistency\"? How are the users chosen? Do they have the concept of 3D consistency?)\n\nI will consider raising the rating if the authors can respond to the weaknesses and questions well in the rebuttal.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed aligned geometric priors (AGP) show effectiveness in improving multiview consistency.\n\n2. The qualitative results show good consistency compared with other methods.\n\n3. The paper is well-written and the core contributions are clear.", "weaknesses": "1. Using Canonical Coordinates Map (CCM) in diffusion model to preserve 3D consistency is one of the core contributions of this paper. However, this coordinate map representation is not something new, and the advantage of using CCM instead of a depth map, normal map, or point cloud is not clear. Experiments to show the superiority of such representation are desired.\n\n2. The experiments have some weaknesses. Only 80 results from 80 text prompts are generated using each method. The results can be noisy based on the small amount of data.\n\n3. The definition of \"3D consistency\" in the quantitative experiment is vague (\"e.g., multiple heads, hands, or legs\"). For example, when the generated object is blurred, or does not have the concept of \"head, hand, leg\" (e.g., building), or some part of the generated object is twisted but not duplicated or missing, how can we determine the 3D consistency is correct or not?\n\n4. It is better to also report the appearance or texture quality measurements, even though this paper mainly focuses on improving the 3D consistency. It can show whether (and if yes, to what extent) the proposed method will influence the appearance or texture quality of the generated objects.\n\n5. The method is based on the assumption that all objects within the same category adhere to a canonical orientation in the training data. However, Objaverse does not align the zero poses of the objects. It is interesting to see whether only using a smaller subset, where objects have a canonical orientation, can lead to better results or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LBLB7hvLdS", "reviewer": "Reviewer_uDHT", "summary": "This paper proposes SweetDreamer, a new solution to the multi-face Janus problem encountered in the text-to-3D generation pipeline. The core strategy revolves around integrating a Canonical Coordinates Map (CCM) SDS loss. SweetDreamer fin-tunes the Stable Diffusion model to predict Canonical Coordinates Maps (CCM) on the Objaverse dataset, offering Aligned Geometric Priors (AGP). Furthermore, the authors incorporated camera information into the diffusion model. Both quantitative and qualitative assessments show that SweetDreamer successfully mitigates the issues of geometry and appearance inconsistency.", "questions": "NA", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of introducing view-conditioned information into 2D foundation models like Stable Diffusion to address the multi-face problem is sound. By fine-tuning the 2D diffusion model to predict the Canonical Coordinates Map, it gains an inherent understanding of the basic viewpoint of an object, thus providing additional aligned geometric priors for the text-to-3D pipeline. Additionally, the inclusion of camera information to speed up convergence is a reasonable choice.\n2. Human evaluation results indeed validate that SweetDreamer alleviates the multi-faced problem. Furthermore, a user study accentuates SweetDreamer's superiority over previous methods.\n3. The presentation is clear and easy to follow. The author gives a thoughtful analysis of the multi-face problem, including “the geometry inconsistency is the key problem compared with appearance inconsistency”.", "weaknesses": "1. There are no ablation studies in this paper. For example, the author claims that the SDS loss from the Canonical Coordinates Map can avoid overfitting the 3D training data. However, there are no experiments to support this idea. Moreover, all the design choices in this paper including how to inject the camera information into the diffusion model are not ablated. \n2. In fact, introducing a view-dependent SDS loss to tackle the multi-face issue has been proven in Zero-123 and Magic-123, which use a view-aware SDS loss to generate a 3D object. SweetDreamer uses a Canonical Coordinates Map SDS loss rather than RGB SDS loss, the novelty and technique contribution is limited.\n3. SweetDreamer uses Objaverse data to fine-tune the Stable Diffusion model to predict the Canonical Coordinates Map. However, defining a canonical coordinate system uniformly across diverse categories is challenging. The authors' choice to whittle down the Objaverse dataset from 800k to 270k, in effect, results in significant data attrition. When compared with techniques that fine-tune based on multi-view images, this could constrain the method's generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2jhKYpeIhe", "reviewer": "Reviewer_UgSj", "summary": "In this paper, the author present SweetDreamer, a text-to-3D generation model. They address the limitations of existing 2D diffusion models by aligning the 2D geometric priors with well-defined 3D shapes from a 3D dataset. To resolve the multi-view inconsistency, SweetDreamer incorporates fine-tuning a pre-trained diffusion model to produce view-specific coordinate maps of canonically oriented 3D objects. Then these aligned geometric priors are integrated into different text-to-image pipelines to generate geometry and appearance.", "questions": "Please see Weaknesses.\n\nOther questions:\n\n1. How long does the fine-tuning for a stable diffusion model take?\n2. How does the efficiency (inference time) of this model compare to other methods?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The problem addressed in this paper is a realistic problem that current diffusion models struggle to handle effectively.\n\n2. The incorporation of 3D geometric priors into the generation process is a compelling and intriguing approach.", "weaknesses": "1. The novelty of this paper is kind of incremental. Most components (such as Stable Diffusion, loss functions) of the proposed method come from previous works, lacking technical novelty.\n\n2. The quantitative results are not satisfying. The evaluation scores reported in Table 1 only show the performance of inconsistency. Did the author compare with other methods by use of other metrics (e.g., R-Precision in DreamFusion)? \n\n3. A simple baseline is missing. Have the authors considered directly replacing the original SDS loss with the aligned geometric SDS loss and calculated the inconsistencies in Table 1?\n\n4. The ablation discussion is absent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)