GeoDream: Disentangling 2D and Geometric Priors for High-Fidelity and Consistent 3D Generation

## Abstract

Abstract Text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models has shown great promise but still suffers from inconsistent 3D geometric structures (Janus problems) and severe artifacts. The aforementioned problems mainly stem from 2D diffusion models lacking 3D awareness during the lifting. In this work, we present GeoDream, a novel method that incorporates explicit generalized 3D priors with 2D diffusion priors to enhance the capability of obtaining unambiguous 3D consistent geometric structures without sacrificing diversity or fidelity. Specifically, we first utilize a multi-view diffusion model to generate posed images and then construct cost volume from the predicted image, which serves as native 3D geometric priors , ensuring spatial consistency in 3D space. Subsequently, we further propose to harness 3D geometric priors to unlock the great potential of 3D awareness in 2D diffusion priors via a disentangled design. Notably, disentangling 2D and 3D priors allows us to refine 3D geometric priors further. We justify that the refined 3D geometric priors aid in the 3D-aware capability of 2D diffusion priors, which in turn provides superior guidance for the refinement of 3D geometric priors. Our numerical and visual comparisons demonstrate that GeoDream generates more 3D consistent textured meshes with high-resolution realistic renderings (i.e., 1024 × \times 1024) and adheres more closely to semantic coherence. Our code and evaluation of 3D metric are available at: GeoDream

## 1 Introduction

Diffusion models [, , ] have significantly advanced text-to-image synthesis. This remarkable achievement has been reached by training scalable generative models on a vast corpus of paired text-image data. Inspired by their success, it is appealing to lift this success from 2D to 3D because this achievement holds significant potential impacts on the modern game and media industry. Template-based generators [] and 3D native generative models [, , , , ] provide a natural and direct approach to the lift.
However, due to the massive and diverse 3D data required to train such generalized models, these methods are usually limited to specific categories with relatively simple topology and texture. Recently, the Score Distillation Sampling (SDS) [] and Variational Score Distillation (VSD) []
have been introduced to optimize 3D representations such that images rendered from any viewpoints maintain a high likelihood, as evaluated by diffusion model conditioned on a given text. This is an exciting direction because it allows for generating 3D assets from any given text prompts, circumventing the need for any 3D data. Despite these methods yielding satisfactory results on a wide range of geometrically symmetrical 3D shapes, empirical observations indicate that SDS and VSD losses still suffer from inconsistent 3D geometric structures (Janus problems) [] and severe artifacts [, ] with asymmetric geometry. This is primarily due to the lack of 3D awareness in 2D diffusion models, which inherently makes the lifting from 2D observations into 3D ambiguous.

Figure: Figure 1: GeoDream alleviates the Janus problems by incorporating explicit 3D priors with 2D diffusion priors. GeoDream generates consistent multi-view rendered images and rich details textured meshes. We remove rendering background to achieve a clearer visualization.
Refer to caption: /html/2311.17971/assets/x1.png

As a remedy, learning 3D priors from 3D datasets seems theoretically reasonable and correct. However, 3D data remains expensive and sparse compared to the plentifully available images. Therefore, the most promising avenue [, , ] presently is to equip 2D diffusion priors with 3D priors learned from relatively limited 3D data, aiming to achieve the best of both worlds. Recently, with the release of large-scale 3D datasets, Objaverse [] and Objaverse-XL [], a few works [, , , ] have attempted to finetune pre-trained 2D diffusion models using multi-view images rendered from 3D dataset. This involves obtaining multi-view images from the fine-tuned diffusion model conditioned on camera parameters and utilizing the clues of predicted multi-view consistency to infer 3D information. Nevertheless, these methods rely heavily on the consistency of content predicted across different
source views. Despite their efforts to employ 3D self-attention to exchange features between different views [], to correlate multi-view features using 3D-aware attention [], or to transform RGB predictions into coarser Canonical Coordinates Map predictions [] to mitigate the negative impact of inconsistencies. Such inconsistencies between the predicted multiple views become particularly noticeable, especially in imaginative and uncommon cases beyond the training data distribution, resulting in over-smoothing and the loss of semantic geometries in the generated 3D assets.

To resolve this issue, we introduce GeoDream, a novel method that incorporates explicit generalized 3D priors with 2D diffusion priors to enhance the capability of obtaining unambiguous 3D consistent geometric structures, while maintaining diversity and high fidelity. Our contributions are listed below. i) In stark contrast to the methods mentioned above that hinge heavily upon the consistency between multi-view priors, we propose to obtain 3D native priors within the 3D world space, which are well-suited to handle the inherent lack of perfect consistency within the multi-view predicted priors, and naturally free from inconsistencies caused by camera viewpoint transition. ii) We justify that disentangling 3D and 2D priors is a potentially exciting direction for maintaining both the generalization of 2D diffusion priors and the consistency of 3D priors. In other words, providing hints through 3D priors to unlock the great potential of 3D awareness in 2D diffusion priors, without the need for invasive finetune 2D diffusion models.

Specifically, we start by reconstructing cost volume as native 3D priors by aggregating the predicted multi-view 2D images into 3D space. Such aggregation operations have been widely used in MVS-based techniques [, , , ], which are known to be robust and generalized to provide valuable cues for geometric reasoning. We find that such operations are well-suited for handling imperfect and inconsistent multi-view predictions. The reason is that they involve multi-view information aggregation, which helps filter out inconsistent content to some extent, rather than dealing with each view individually. Foremost, we conduct extensive experiments to demonstrate that our proposed 3D priors adapt to multiple views predicted by various off-the-shelf multi-view diffusion models, such as Zero123 [], MVDream [] and Zero123++ []. Moreover, we introduce a critical viewpoint sampling strategy to promote the stability of the 3D priors.

We further propose incorporating 3D priors with 2D diffusion priors in a disentangled solution. Existing multi-view diffusion priors are equipped with 2D diffusion priors in a coupled way, including generating multiple views as supervision [, ] or distilling the probability density as a loss [, , , ] to compute gradients for optimizing 3D representations. Instead, we justify that leveraging the geometric clues provided by 3D priors can effectively unleash the great potential 3D awareness capability inherent in 2D diffusion priors, referred to as “disentangled design”.
Very recent works have started to explore how to evoke 3D-aware ability in 2D diffusion by altering score functions [] or negative text prompts []. These efforts have made surprising progress, yet the performance remains unstable regarding 3D consistency.
Our insight is that going through geometric priors to unlock the great potential of 3D awareness in 2D diffusion is a promising direction that is both general and stable.
Moreover, we rely solely on the awakened 3D-aware capability of 2D priors to guide the optimization of Neural Implicit Surfaces (NeuS) [] without the supervision of 3D priors, thereby avoiding compromising the inherent advantages of 2D priors in terms of generalization and creativity.
We show that 3D priors can be further refined to boost rendering quality and geometric accuracy. The 2D diffusion priors benefit from gradually evolved 3D priors, which in turn provide superior guidance for unleashing the 2D priors. Finally, we use DMTet [] to extract textured mesh from optimized NeuS for mesh fine-tuning.
Unlike previous work [, ] attempt to increase the rendering resolution, which typically suffer from over-saturation issues, we successfully increase the rendering resolution from 512 to 1024.
We hypothesize that the improved results are aided and abetted by 3D priors that provide more plausible geometry and realistic texture, making the optimization easier, because the rendered image is closer to diffused distributions.
To comprehensively evaluate semantic coherence, to our knowledge, we are the first to propose $\rm{Uni3D_{score}}$ metric, lifting the measurement from 2D to 3D.

**Table 1: Comparison of design space.**
| Method | Prolific [] | MVDream [] | GSGEN [] | Ours |
| --- | --- | --- | --- | --- |
| Repr. | NeRF+DMTet | NeRF | Gaussian | NeuS+DMTet |
| Resolution | 512 | 512 | 512 | 1024 |
| 3D guidance | - | Multi-Views | Point-E | Cost volume |
| 3D&2D | - | Entangled | Entangled | Disentangled |
| 3D priors | Fixed | Fixed | Fixed | Optimizable |

As summarized in Tab.[1](#S1.T1), we compared the latest methods[, , ] in design space, including 3D representation, rendering resolution, forms of 3D guidance, the disentangling of 3D and 2D priors and the optimizability of 3D priors. As shown in Fig.[1](#S1.F1), GeoDream can yield 1024 $\times$ 1024 high-resolution rendered images and high-fidelity textured meshes while greatly alleviating the notorious Janus problems. In Sec.[4.1](#S4.SS1), we conduct comprehensive evaluations that demonstrate the superiority of the 3D assets generated by GeoDream in terms of plausible geometry and delicate rendering details in visual appearance.

## 2 Related Work

3D Generation Guided by 2D Priors.
Deep generative models have driven the field of 3D generation. Some efforts utilize Variational Auto Encoders (VAEs) [] for texture generation [, ], while Generative Adversarial (GAN) Models [] investigate 3D-aware GAN training [, ]. Thus far, diffusion models have exhibited relatively better generalizability and training stability for diverse object generation compared to GANs and VAEs, and thus have gradually become recent focal points in 3D generation. Specifically, recent endeavors attempt to leverage the potent 2D diffusion priors to aid 3D generation by coupling it with a 3D representation, such as NeRF [], DMTNet [], or NeuS [], among others, bypasses the necessity for extensive text-3D datasets for training 3D generative models. Such methods involve various techniques, including score distillation sampling schedules like SDS [], SJC [], and VSD [] losses, which optimize the 3D representation by enhancing high likelihood evaluated by the 2D diffusion models. A coarse-to-fine training strategy [] strengthens texture representation, decoupling geometric and texture aspects of 3D representation for finer optimization [], improving 3D representation [, ]. Although these methods demonstrate the ability to generate photo-realistic and diverse 3D assets with user-provided textual prompts, they are prone to the notorious 3D inconsistency issues (Janus problems) during the lifting process due to their reliance on 2D diffusion models for training, which lack 3D knowledge. Despite some current methods attempting to address 3D inconsistency by altering score functions [] or negative text prompts [], performance remains instability in terms of 3D consistency. In this work, we aim to explore the distinctive advantages of incorporating explicit 3D priors with 2D priors, enabling the generation of highly detailed 3D objects while remarkably mitigating 3D inconsistency issues.

3D Generation Guided by 3D Priors.
Learning 3D priors from 3D datasets seems theoretically reasonable and correct for enhancing the coherency of 3D generation [, , , , , ]. Therefore, various 3D latent diffusion models trained on 3D data have been recently introduced, including those using Tri-plane [] or feature grid [, , ] encoding 3D representations into the latent space. Additionally, OpenAI has explored models aiming to directly generate 3D formats using several million internal 3D shapes, such as point clouds [] or neural radiance fields []. However, their generalizability to the scope of their 2D counterparts remains unverified, due to the relative sparsity of 3D data compared to the abundance of available 2D images.
Consequently, the most promising avenue currently is to equip 2D diffusion priors with 3D priors learned from relatively limited 3D data, intending to achieve the best of both worlds. Recently, with the release of a large-scale 3D dataset called Objaverse [] and Objaverse-XL [], some work [, , , , , ] has attempted to fine-tune pre-trained 2D diffusion models using multi-view images rendered from 3D data. This aims to generate multi-view images from the fine-tuned diffusion model conditioned on camera parameters and utilize the clues of predicted multi-view consistency to assist in inferring 3D information. Nevertheless, these methods heavily depend on the absolute consistency of content predicted across different views. Nonetheless, their efforts to utilize 3D self-attention [, ] for feature exchange between different views, to correlate multi-view features using 3D-aware attention [], or to transform RGB predictions into coarser Canonical Coordinates Map predictions [] for mitigate the negative impact of inconsistencies. The performance of such methods frequently exacerbates inconsistencies and unrealistic rendering quality in uncommon cases, due to the absence of explicit constraints between different predicted viewpoints within 3D space. In this work, we incorporate explicit generalized 3D priors into 2D diffusion priors. These explicit 3D priors fundamentally ensure consistency in 3D space and avoid the independence of multi-view priors across source views.

Figure: Figure 2: The overview of GeoDream. (a) 3D priors training. (b) Incorporating 3D priors with 2D diffusion priors.
Refer to caption: /html/2311.17971/assets/x2.png

## 3 Method

We focus on generating 3D content with consistently accurate geometry and delicate visual detail, by equipping 2D diffusion priors with the capability to produce 3D consistent geometry while retaining their generalizability. The overview
of GeoDream is shown in Fig.[2](#S2.F2). GeoDream consists of the following two stages. i) During 3D priors training, we build upon the One-2-3-45 [], which encodes geometry into cost volume $V$ and geometry MLP decoder $f_{g}$. In addition, the appearance of the object is modeled to cost volume $V$ and texture MLP decoder $f_{t}$. We refer to the trained geometric decoder $f_{g}$ and appearance decoder $f_{t}$ with cost volume $V$ as native 3D geometric priors and appearance priors, as shown in Fig.[2](#S2.F2) (a). Detials in Sec.[3.1](#S3.SS1). ii) During priors refinement, we show that geometric priors can be further fine-tuned to boost rendering quality and geometric accuracy by combining a 2D diffusion model, as shown in Fig.[2](#S2.F2) (b). Detials in Sec.[3.2](#S3.SS2).

### 3.1 Generalizable 3D Priors Training

We start by reconstructing cost volume $V$ as native 3D priors by aggregating the 2D image features into 3D space, which provides valuable cues for geometric reasoning in the priors refinement stage.

Cost Volume Construction.
Following MVS-based methods [, , , ], given multi-view images $I=\{(I_{i})^{N-1}_{i=0}\}$, we extract 2D feature maps $F=\{(F_{i})^{N-1}_{i=0}\}$ using a 2D feature network $f_{2D}$. Volume reconstruction model takes posed 2D feature maps $F$ as input and outputs cost volume $V$ with per-voxel neural features in voxels. Specifically, for each voxel centered at 3D location $h$, per-voxel neural feature is computed by projecting each location $h$ to $N$ image feature planes and then fetching the variance of the features at the the location of the projection. We use $\mathrm{~{}Var}$ to denote the variance operation and $P$ to denote the projection procedure. We then use a sparse 3D CNN $f_{3D}$ to process the variance features per voxel to regress the cost volume, as formulated by,

$$ $V=f_{3D}(\mathrm{~{}Var}\{P(F_{i},h)\}_{i=0}^{N-1}),$ (1) $$

where the variance operation is invariant to the number $N$ of input images. We find that such an operation is well-suited for handling imperfect and inconsistent multi-view predictions, due to involving information aggregation rather than dealing with each view individually.

Geometry and Texture Decoder.
The cost volume $V$ is directly decoded into signed distance function values (SDF) and color information using the corresponding geometry MLP decoder $f_{g}$ and texture MLP decoder $f_{t}$. For any arbitrary query point $x\in\mathbb{R}^{3}$, we get the SDF $s$ and color $c$ as

$$ $s(x)\,=\,f_{g}(E(x),V(x)),$ (2) $$

$$ $c(x)=f_{t}(\{P(F_{i},x)\}_{i=0}^{N-1},V(x),\{\Delta d_{i}\}_{i=0}^{N-1}),$ (3) $$

where $E$ denotes position encoding, $V(p)$ denotes tri-linearly interpolated feature from cost volume at query point $x$,
$\Delta d_{i}=d-d_{i}$ is the viewing direction of the query ray relative to the viewing direction of the $i{th}$ multi-view image.

The final rendered image $I{{}^{\prime}}$ is achieved via SDF-based differentiable volume rendering $R$. In this work, we get the pre-trained parameters of the $f_{g}$, $f_{t}$, and $f_{3D}$ networks from the One-2-3-45 [], which is trained on ground truth images $I$ rendered from the Objaverse dataset with a loss

$$ $\mathcal{L}_{rgb}=||I-I{{}^{\prime}}||_{2},$ (4) $$

where $I{{}^{\prime}}=R(\{s(x_{j}),c(x_{j}))\}_{j=0}^{M-1})$, $M$ denotes sampling $M$ query points along the ray of viewing direction.

### 3.2 Priors Refinement

We present how to further finetune the geometric priors obtained from 3D priors training stage, i.e., optimizable cost volume $V$ and the fixed pre-trained geometric decoder $f_{g}$, using the 2D diffusion priors, as shown in Fig.[2](#S2.F2) (b). During priors refinement stage, we replace the $N$ ground truth rendered images with multi-view diffusion model predictions. In contrast to One-2-3-45, GeoDream is not limited to the Zero123 [] predictions. We conduct extensive experiments with various multi-view diffusion models, such as MVDream [] and Zero123++ []. We also introduce a critical viewpoint sampling strategy to ensure GeoDream robustly adapts to various multi-view diffusion models, rather than being limited to just one. Overall, we justify that by decoupling 3D and 2D diffusion priors, GeoDream unlocks the immense potential of 3D awareness in the 2D diffusion model, avoiding the tendency to produce canonical views, resulting in 3D assets featuring multiple faces and collapsed geometry. Thanks to the decoupling, GeoDream maintains the generalization and imaginativeness of 2D diffusion priors, while also exploring the significant role that geometric priors play in improving appearance modeling.

Multi-View Images Generation. The rapid advancement of 3D generation has provided a wide range of methods available for generating multi-view images, such as Zero123[], MVDream[], and Zero123++ []. Given a set of predefined camera poses $\{(R_{i},T_{i})^{N-1}_{i=0}\}$ and a user-provided condition $c$, we utilize a fixed multi-view diffusion $f_{mv}$ to predict posed images $I_{p}=\{(I^{p}_{i})^{N-1}_{i=0}\}$ and extract 2D feature maps $F_{p}=\{(F^{p}_{i})^{N-1}_{i=0}\}$,

$$ $F^{p}_{i}=f_{2D}(f_{mv}(c,R_{i},T_{i})),$ (5) $$

where $R\in\mathbb{R}^{3\times 3}$, $T\in\mathbb{R}^{3\times 3}$ respectively denote relative camera rotation and translation of the default viewpoint.

3D Geometric Priors.
By replacing $F_{i}$ in Eq.[1](#S3.E1) into $F^{p}_{i}$, we obtain the value of SDF at an arbitrary query point $x$ defined in Eq.[2](#S3.E2),

$$ $V_{p}=f_{3D}(\mathrm{~{}Var}\{P(F^{p}_{i},h)\}_{i=0}^{N-1}),$ (6) $$

$$ $s_{p}(x)\,=\,f_{g}(E(x),V_{p}(x)),$ (7) $$

where $s_{p}(x)$ is treated as a geometric prior since it encodes the hidden geometric clues in the predicted multiple views.

Texture Decoder.
We propose to drop the pre-trained texture priors $f_{t}$ defined in Eq.[3](#S3.E3) because we empirically find that texture priors tend to generate 3D assets with lighting and texture styles similar to the rendered dataset. We choose Instant NGP [] for efficient high-resolution texture encoding. Specifically, for any arbitrary query point $x\in\mathbb{R}^{3}$, a learnable hash encoding $h_{\Omega}$ is decoded into a color $c$ using initialized texture decoder $f_{t}^{\prime}$, as formulated by,

$$ $c_{p}(x)=f_{t}^{\prime}(h_{\Omega}(x),x),$ (8) $$

where $h_{\Omega}(x)$ denotes the looked-up feature vector from $h_{\Omega}$ at query point $x$.

Texture and Geometry Refinement.
To incorporate 3D geometric priors with 2D diffusion priors, we minimize the VSD loss introduced in ProlificDreamer [] to optimize the parameters of ${\theta}_{1}$ in cost volume $V$, ${\theta}_{2}$ in hash encoding $h_{\Omega}$ and ${\theta}_{3}$ in texture decoder $f_{t}^{\prime}$. At each iteration, we sample a camera pose $o$ from a pre-defined distribution. We render 2D image $\hat{x}$ at pose $o$ by combining Eq.[7](#S3.E7) and Eq.[8](#S3.E8) via differential rendering $R$. Our objective function during priors refinement is to minimize the VSD loss $\mathcal{L}_{VSD}$, the corresponding gradient $\nabla_{{\theta}_{1},{\theta}_{2},{\theta}_{3}}\mathcal{L}_{VSD}$ is

$$ $\mathrm{E}_{t,\epsilon,o}[w(t)(\epsilon_{pretrain}(\hat{x_{t}},t,c)-\epsilon_{l}(\hat{x_{t}},t,c,o))\frac{\partial\hat{x}}{\partial({\theta}_{1},{\theta}_{2},{\theta}_{3})}],$ (9) $$

where $\hat{x_{t}}$ denotes a noisy rendered image in timestep $t$, $w(t)$ denotes a weighting function, $\epsilon_{pretrain}$ is a 2D pretrained diffusion model and $\epsilon_{l}$ is a trainable LoRA [] diffusion model with parameters of $l$. We propose to fix the geometry decoder $f_{g}$ conjointly with a learning rate decay strategy for the cost volume, aiming to maintain geometric priori cues as well as tuning to achieve better details in the early stage of optimization. More details on viewpoint sampling and learning rate decay strategy are provided in Sec.[4.2](#S4.SS2).

Mesh Fine-tuning.
For high-resolution rendering, we use DMTet [] to extract textured 3D mesh representation from optimized NeuS []. By minimizing the loss in Eq.[9](#S3.E9), we follow ProlificDreamer [] first to optimize the geometry using the normal map and then optimize the texture. We empirically find that we can increase the rendering resolution from 512 to 1024. But unlike previous work [, ], attempting to increase the rendering resolution suffers from over-saturation issues.
We successfully increase the rendering resolution from 512 to 1024.
We hypothesize that well-optimized results are aided and abetted by 3D priors that provide more plausible geometry and realistic texture, making the optimization easier, because the rendered image $\hat{x}$ is closer to diffused distributions at each iteration.

## 4 Experiment

### 4.1 Results of GeoDream

Baselines. We report our performance with the latest 3D generation methods, including DreamFusion [], ProlificDreamer [], MVDream [], GSGEN [], Fantasia3D [] and Wonder3D []. Specifically, DreamFusion [], Fantasia3D [] and ProlificDreamer [] adopt a similar approach to optimize 3D representation through the score function of a 2D diffusion model, without intervening in 3D priors. We compare our results with these three methods, highlighting the distinct advantages of inferring 3D-consistent geometry and reducing artifacts by incorporating explicit 3D priors.
Meanwhile, MVDream [] and Wonder3D [] are very recent proposals to use multi-view consistency priors, which derived from finetuned multi-view diffusion models trained on synthetic multi-view rendering image data. GSGEN [], on the other hand, addresses 3D inconsistency by initializing geometry with Point-E [] generated shapes. By comparing these three methods, we demonstrate that our introduced 3D priors offer greater generality in challenging and uncommon cases and effectively prevent the generation of 3D assets with lighting and texture styles similar to the synthetic rendered dataset. For DreamFusion [], ProlificDreamer [] and Fantasia3D [], we utilize their implementations in the ThreeStudio [] library for comparison. For MVDream [], GSGEN [] and Wonder3D [], we use their official implementation.

Figure: Figure 3: Qualitative comparison with baselines. Back views are highlighted with red rectangles for distinct observation of multiple faces.
Refer to caption: /html/2311.17971/assets/x3.png

Experiment Setup.
We collected 35 prompts from various sources, including prompts from previous work [, ] and real user inputs in the wild. To comprehensively assess 3D consistency and semantic coherence, we intentionally selected more prompts indicating asymmetric geometric structures (80% of the collected prompts) and fewer prompts indicating symmetric geometric structures (20%). For a fair comparison, we render 3D assets generated by our method and baselines by circling around the object at a default elevation and camera distance [], resulting in 120 images. We then evaluate the gap between the rendered images and reference images generated by Stable Diffusion [] based on the collected prompts. We sample 10$k$ points on the generated meshes to calculate 3D metric. To demonstrate that our method is trivially adaptable to various multi-view diffusion models, we randomly use either Zero123 [] or MVDream [] and Zero123++ [] for subsequent experiments. For the effect of different diffusions on the results, please refer to supplementary for detail.

2D Metrics.
$\rm{FID_{CLIP}}$ [] for image fidelity measurement, which is calculated by the disparity in distribution between the rendered image and reference image features, both encoded by CLIP ViT-B-32 []. CLIP R-score for semantic coherence measurement is calculated by the probability of rendered images retrieving the correct caption among collected prompts. We average the metric over 120 rendered images for the quantitative comparison.

3D Metric. These metrics mentioned above are for measuring 2D images. Limited by rendering angles and geometric self-occlusion, 2D metrics often struggle to assess 3D objects in 360 degrees fully. To the best of our knowledge, no metrics have yet been introduced in text-to-3D tasks for evaluating the semantic consistency of 3D assets. Therefore, we propose using Uni3D [], the largest 3D presentation model with one billion parameters under text-image-pointcloud alignment learning objective, to lift semantic coherence measurement from 2D to 3D. We adopt a similar strategy to the CLIP R-score, except that we replace the image and text encoders in the CLIP with the point cloud and text encoders from the Uni3D, referred to as “$\rm{Uni3D_{score}}$”.

Subjective Metric.
3D reconstruction tasks are typically evaluated of the error reconstructed shape compared to the ground truth [].
However, these metrics are difficult to apply to text-to-3D tasks, as there is no ground truth.
We further manually check the number of examples with 3D or semantic inconsistency problems, and then report the rate of success as an auxiliary metric, referred to as “Cons. Rate”.

Quantitative Comparison.
In Tab.[2](#S4.T2), we conduct a quantitative comparison over generation quality, text-image consistency and 3D consistency. Overall, the results indicate that our method significantly outperforms the baselines across all metrics, demonstrating that we achieve high-fidelity, text-image and text-3D consistency in the generated quality while ensuring 3D spatial consistency.

**Table 2: Quantitative comparison with baselines.**
| Model | ${\rm{FID}}_{\rm{CLIP}}$ $\downarrow$ | CLIP R-score$\uparrow$ | ${\rm{Uni3D}}_{\rm{score}}$ $\uparrow$ | Cons. Rate$\uparrow$ |  |
| --- | --- | --- | --- | --- | --- |
|  |  | B/16 | L/14 |  |  |
| DreaFusion [] | 59.6 | 0.844 | 0.870 | 0.514 | 0.429 |
| ProlificDreamer [] | 48.8 | 0.866 | 0.892 | 0.629 | 0.257 |
| MVDream [] | 50.6 | 0.852 | 0.886 | 0.771 | 0.829 |
| Ours | 47.9 | 0.935 | 0.962 | 0.800 | 0.914 |

Qualitative Comparison. Fig.[3](#S4.F3) compares our method with the baselines. We present four visual examples: the first three rows depict non-symmetric geometries, while the last row is for symmetric geometry. Notably, we display the front, side, and back views, where the back views are highlighted with red rectangles to enhance the observation of potential multiple faces issues.
We highlight our improvements in visual comparison in Fig.[3](#S4.F3). Dreamfusion and ProlificDreamer produce high-quality frontal views but fail to form a plausible 3D object. In particular, ProlificDreamer delivers photorealistic 3D assets with semantic coherence, where every view resembles canonical views, i.e., the back views that are shown in red rectangles, are mistakenly optimized as front views, resulting in Janus problems. GSGEN mitigates some of the 3D inconsistencies by introducing 3D priors from the pre-trained Point-E. However, the fidelity of the textures it generates is still insufficient for complete satisfaction. Compared to the three methods mentioned above, MVDream stands out as the most effective solution for addressing multi-view inconsistency issues. This is achieved by fine-tuning pre-trained 2D diffusion models using multi-view images rendered from 3D data. Nevertheless, due to the rendering quality and sparsity of 3D training data, the generated results often exhibit cartoon-style textures and semantically lost geometries, particularly when dealing with uncommon and challenging given prompts. For example, it struggles to generate a rocket as required in the second case, a samurai style as required in the third case, and a log as required in the fourth case.
By incorporating explicit 3D priors with a 2D diffusion model that is capable of imagination diversity, GeoDream significantly alleviates the multifaceted nature of generated 3D assets, in terms of both meshes and rendered images exhibiting impressive photorealistic textural details, while maintaining semantic faithfulness, as shown in Fig.[1](#S1.F1) and Fig.[3](#S4.F3).
More analysis and comparisons with other baselines can be found in the supplementary.

Figure: Figure 4: Ablation study of proposed improvements for text-to-3D generation.
Refer to caption: /html/2311.17971/assets/x4.png

### 4.2 Ablation Study

We then conduct ablation studies to justify the effectiveness of each design in GeoDream. We activate all the modules and training strategies mentioned in the Sec.[3](#S3) during ablation studies, except for the modified part described in each ablation experiment below.

The Effect of 3D Priors. We first visualize the initial cost volume obtained through the volume construction model, as shown in Fig.[4](#S4.F4) (a).
Fig. [4](#S4.F4) (a) combined with Fig.[4](#S4.F4) (e) demonstrate that relying solely on rough geometric cues can significantly activate potential of 3D awareness in 2D diffusion, alleviating the character’s tendency to exhibit multifaceted issues. In contrast to fixed priors in Fig.[4](#S4.F4) (b), we propose using optimizable priors that gradually evolve according to the optimization state, thus producing progressively refined results, as shown in Fig.[4](#S4.F4) (e) and Fig.[4](#S4.F4) (j).
To further assess its impact, we also attempt to deactivate the cost volume, i.e., randomly initializing the 3D prior. The 3D inconsistency issue also arises, as shown in Fig.[4](#S4.F4) (f).
To assess the impact of the learning rate decay schedule, an ablation study is conducted, where the learning rate of the cost volume is set to a suitable constant value. The generated 3D assets still suffer severe degeneration, resulting in a completely collapsed geometry in Fig.[4](#S4.F4) (c). The reason is that, during the early stage of optimization, there may be a lot of ambiguity and conflict in the appearance information across different views. Hence, during the early optimization stage, we propose to set the learning rate of the cost volume to a smaller value and gradually increase it for geometric detail optimization. And vice versa for the learning rate of texture, which can prevent content drift in the later stage of optimization, please refer to supplementary for detail.

We further justify whether we should use texture priors. We report a visual result using a pre-trained texture MLP in Sec.[3.1](#S3.SS1), rather than reinitializing the MLP network and hash encoding in Sec.[3.2](#S3.SS2). Fig.[4](#S4.F4) (g) demonstrates that introducing texture priors generally leads to a visual appearance that tends toward non-photorealism and over-smoothing. This observation underlines the necessity of introducing only 3D geometric priors, which only contribute to the geometry modeling during the lifting, avoiding compromising the appearance modeling due to texture priors.

The Effect of Mesh Fine-tuning. We convert NeuS to DMTet to improve geometric and appearance details. We first show the NeuS-based visual results in Fig.[4](#S4.F4) (h).
GeoDream produces better results with finer details, as evidenced in Fig.[4](#S4.F4) (j).
The reason is that the advantages of the 3D assets we generate, which yield improved 3D consistency, lie in the ability to enhance the accuracy of surfaces, thereby reducing the complexity of texture optimization in the DMTet. Fig.[4](#S4.F4) (d) presents an ablation study on SDS and VSD loss. SDS is observed to produce over-saturated textures, as opposed to the VSD loss that we default to using.

The Effect of Rendering Resolution. Through empirical experimentation, we deduce that collapsed geometry often results in textural distortions, thereby increasing the difficulty of optimization. Hence, we conjecture that 3D consistency is one of the main bottlenecks for increasing the rendering resolution in prior work. Instead, by integrating 3D geometric priors, we achieved better results closer to diffused distributions, making the optimization becomes easier. Consequently, we successfully increase the rendering resolution from 512 to 1024, as shown in Fig.[4](#S4.F4) (j). Additionally, Fig.[4](#S4.F4) (i) demonstrates that GeoDream still provides competitive results at 512 $\times$ 512 resolution.

## 5 Conclusion

We significantly improve the rendering fidelity of images and the details of texture meshes, while greatly alleviating the notorious Janus problems by the awakened 3D-aware capability of 2D diffusion priors, which is unleashed by geometric clues provided by 3D priors in a disentangled solution. Additionally, the disentangled design offers a flexible way to optimize 3D priors gradually. The visual and numerical comparisons with the state-of-the-art methods justify our effectiveness
and show our superiority over the latest methods in 3D generation.

Supplementary Material

## 6 Video

Our supplementary material also includes a video, which shows more visualizations, inviting readers to watch for a more intuitive visual experience.

Figure: Figure 5: More visualization comparisons with baselines. For each row from up to down, the given prompts are: (1) *3D render of a statue of an astronaut.* (2) *3D stylized game little building.* (3) *A brightly colored mushroom growing on a log.* (4) *An ice-cream cone*
Refer to caption: /html/2311.17971/assets/x5.png

## 7 Source Code

To facilitate future research, our code and 3D metric are available at: [GeoDream](https://mabaorui.github.io/GeoDream_page)

## 8 Definition of The Janus Problem

We explain in further detail the definition of the Janus problem (3D inconsistency), which refers to a phenomenon that the learned 3D representation, instead of presenting the 3D desired output, shows multiple canonical views of an object in different directions [, ]. For instance, when the given prompt indicates an asymmetric geometric structure, such as a person or an animal, the generated 3D asset has multiple faces but lacks complete and correct back views. In contrast, when the given prompt indicates a symmetric structure, such as a cake or a hamburger, which does not have strictly defined back views, issues of 3D inconsistency typically do not arise. Therefore, when calculating the subjective metric, geometrically symmetric 3D assets do not suffer from 3D inconsistency by default.

## 9 More Visualization Comparisons with Baselines

We report our performance with more 3D generation methods, including Fantasia3D [], Wonder3D [], and Magic123 []. Fantasia3D employs DMTet [] initialized with a handcrafted 3D model or a predefined geometric shape as the 3D representation, which is the same representation used in our mesh fine-tuning phase. We compare our DMT-based results with Fantasia3D to show the gains in rendering appearance from geometry initialization with 3D priors. Wonder3D employs NeuS [] as its 3D representation, which is subsequently processed through the Marching Cube algorithm [] to extract mesh. Magic123 adopts a coupled approach, optimizing the 3D representation by using both 3D and 2D priors as losses. The comparisons with Magic123 justify that the disentangling 3D and 2D priors allows for the simultaneous harnessing of the generalization capabilities of 2D diffusion priors and the 3D consistency of 3D priors. In contrast, Magic123 requires careful design of the balance weights between 3D and 2D loss to avoid compromising between the two types of priors. Visual comparisons in Fig.[5](#S6.F5) reveal that we enhance the fidelity and semantic coherence of the generated 3D assets, accompanied by an absence of geometric and textural distortions, indicating excellent 3D spatial consistency.

Figure: Figure 6: The detailed learning rate schedule.
Refer to caption: /html/2311.17971/assets/x6.png

## 10 Viewpoint Sampling Strategy

Figure: Figure 7: Ablation on the methods for obtaining reference views. We compare the generated 3D assets based on reference views predicted by Stable Diffusion and MVDream, driven by user-provided texts. GeoDream adapt to reference views from various sources. For each row from up to down, the given prompts are: (1) *A majestic giraffe with a long neck.* (2) *Viking axe, fantasy, weapon, blender, 8k, HD.*
Refer to caption: /html/2311.17971/assets/x7.png

Figure: Figure 8: Ablation on the viewpoint sampling strategy. We demonstrate that using our proposed viewpoint sampling strategy contributes to the more robust generation of a consistent cost volume, significantly avoiding the outcomes of geometric collapse. For each row from up to down, the given prompts are: (1) *A dinosaur toy.* (2) *A corgi.*
Refer to caption: /html/2311.17971/assets/x8.png

We propose a critical viewpoint sampling strategy to enhance the stability of constructing cost volumes. Cost volume-based methods [, , , ] rely on the consistency and accuracy of multi-views to find local correspondences and infer geometry. We empirically find that current multi-view diffusion models [, , , , , ] can provide relatively accurate and consistent predictions for small relative pose, when fed with front and side views as reference views. Instead, when a back view is used as the reference view, inconsistencies tend to worsen. Our analysis indicates that these multi-view models are fine-tuned from 2D pre-trained diffusion models, which exhibit weaker performance in predicting non-canonical view information. Additionally, the information implied by back views is quite ambiguous, posing challenges for predicting consistent information. Consequently, we propose a viewpoint sampling strategy to mitigate the aforementioned problems.

Specifically, We obtain reference views driven by a user-provided text in one of two methods: i) Obtaining a front view predicted by Stable Diffusion [], which is trivial as Stable Diffusion often biases towards generating canonical views. ii) Utilizing MVDream [] to output desired views based on our predefined absolute camera positions. In our experiments, following the default settings of MVDream, we set the absolute elevation angle at $15^{\circ}$ and absolute azimuth angles at $0^{\circ}$, $90^{\circ}$, $180^{\circ}$, and $270^{\circ}$. We sample four viewpoints on the sphere surface with a default radius to obtain the front, left, back, and right views as reference views.

When the reference view is predicted by Stable Diffusion, we require either Zero123 [] or Zero123++ [] to randomly sample viewpoints within a range of a relative azimuth angle less than $180^{\circ}$ and a relative elevation angle less than $30^{\circ}$. Subsequently, we sample an image with a relative azimuth angle of $180^{\circ}$ and a relative elevation angle of $0^{\circ}$ to serve as the back view, which is then added to the source views. In the case of reference views are predicted by MVDream, we use Zero123 or Zero123++ to sample viewpoints relative to the front view, left side view, and right side views, within a range of a relative azimuth angle less than $45^{\circ}$ and a relative elevation angle less than $30^{\circ}$. Subsequently, the back view predicted by MVDream is supplemented to the source views. We show the visualized comparison of the impact of reference views generated by Stable Diffusion and MVDream on the generated 3D assets, as shown in Fig.[7](#S10.F7). We report visualized results without viewpoint sampling strategy and the results with viewpoint sampling strategy, as shown in Fig.[8](#S10.F8). The visualized results indicate that our proposed sampling strategy can adapt to reference views predicted by both Stable Diffusion and MVDream, significantly enhancing the quality of the constructed cost volume and the consistency of the generated 3D assets.

Finally, we observe that due to the inherent lack of perfect consistency between source views, the constructed cost volume is quite rough, even with the viewpoint sampling strategy, as shown in Fig.[7](#S10.F7) and Fig.[8](#S10.F8). However, the ultimately generated 3D assets tend to produce rich details and more complete and consistent geometry. This suggests that disentangling 3D and 2D priors is a potentially exciting direction, as it provides a flexible way to further refine 3D priors while maintaining the ability of 3D priors to unleash 2D diffusion priors.

## 11 Learning Rate Decay Schedule

We propose to set the learning rate of the cost volume to a smaller value and gradually increase it for geometric detail optimization, aiming to maintain geometric priori cues in the early stage of optimization. And vice versa for the learning rate of texture, which can prevent content drift in the later stage of optimization. During the early optimization stage, we adopt an initially high learning rate to fight early overfitting [, ]. The detailed learning rate curves are depicted in Fig.[6](#S9.F6).

Figure: Figure 9: Ablation on negative prompting, rendering resolution, and corner case. The given prompts are: (a) and (b) *A 3D printed white bust of a man with curly hair.* (c) *An astronaut riding a horse.* (d) and (e) *A DSLR photo of a squirrel playing guitar.*
Refer to caption: /html/2311.17971/assets/x9.png

## 12 Ablation on negative prompting, rendering resolution, and corner case

Prompting. Perp-Neg [] introduces a negative prompt algorithm that transforms 2D Diffusion into 3D, addressing the Janus problem. We attempt to integrate the negative prompt algorithm into both ProlificDreamer and GeoDream, as shown in Fig.[9](#S11.F9) (a) and Fig.[9](#S11.F9) (b). The result shown in Fig.[9](#S11.F9) (a) demonstrates that the negative prompt algorithm still fails to mitigate the Janus problem stably. Fig.[9](#S11.F9) (b) illustrates that GeoDream is able to yield consistent 3D assets both with and without the negative prompt algorithm. However, since we did not observe a significant improvement in the results, we opt not to use the negative prompt algorithm as a default in our experiments. Instead, we employ view-dependent prompting as in previous works [, ].

Rendering Resolution. We attempt to increase the rendering resolution to 1024 in ProlificDreamer, which typically struggles with over-saturation issues, as demonstrated in Fig.[9](#S11.F9) (c). Our analysis suggests that the absence of 3D priors often leads to collapsed geometry, resulting in textural distortions and thereby increasing the complexity of the optimization.

Corner Case.
We further explore the robustness of GeoDream when faced with failures of multi-view diffusion in predicting multiple views. For instance, when the given prompt is “*A DSLR photo of a squirrel playing guitar*”, multi-view diffusion struggles to accurately predict the correct spatial relationship between the guitar and the squirrel, due to the sparsity of 3D training data. However, GeoDream excels in preserving the generalizability and creativity of 2D diffusion priors, enabling more effective compatibility with imperfect multi-view predictions, and thus generating semantically correct 3D assets, as shown in Fig.[9](#S11.F9) (e).

## 13 Training Stability and Diversity

Stability.
Prior text-to-3D studies are notoriously brittle. The same hyperparameter settings often lead to vastly different results in terms of complete failure or success, depending on the random seed, making them hard to control. To assess the training stability of GeoDream, we conduct several experiments on the same prompt, as shown in Fig.[10](#S13.F10). GeoDream exhibits exceptional training stability. The reason lies in the 3D priors we introduced, which significantly reduce the randomness caused by the random seeds.

Diversity
Additionally, we can generate diverse 3D models by controlling and leveraging the diversity capabilities of Stable Diffusion or MVDream to predict various reference views, as mentioned in Sec.[10](#S10) and Fig.[7](#S10.F7). In summary, GeoDream provides a balanced solution between diversity and stability.

Figure: Figure 10: Ablation on training stability. We conduct several experiments on the same prompt to verify the training stability of GeoDream. The given prompt is: *An astronaut riding a horse.*
Refer to caption: /html/2311.17971/assets/x10.png

## 14 Licenses

We provide the URL, citations, and licenses of the open-sourced assets we used in this work, as shown in Tab.[3](#S17.T3).

## 15 Algorithm

We provide a summarized algorithm of priors refinement in Algorithm [1](#algorithm1).

## 16 Training Details

We construct a cost volume with 150 $\times$ 150 $\times$ 150 voxels in 2 minutes on an NVIDIA-V100-32GB GPU. During the priors refinement stage, we employ a network modified based on ProlificDreamer []. We replace the learnable hash encoding used in ProlificDreamer by cost volume. We choose a single-layer MLP to decode the color from texture hash encoding as Instant-NGP []. Following ProlificDreamer, we set the particle to 1 and utilize v-prediction [] to train the LoRA [] based on Stable Diffusion v2.1 model for VSD loss. Notably, even when the rendering resolution increased from 512 to 1024, the training time does not show a significant difference compared to ProlificDreamer. The reason is that 3D assets generated by GeoDream, which exhibit fewer artifacts and thus enhanced rendering efficiency. Specifically, training the NeuS representation [] with the batch size set to 1 typically requires approximately 3 hours on a single NVIDIA-V100-32GB GPU. Mesh finetuning with a batch size of 2 usually requires around 8 hours on a single NVIDIA-V100-32GB GPU. Utilizing larger batch sizes and parallel multi-GPUs training could potentially reduce training times and we leave this exploration in future work.

## 17 Ablation on Source Views Predicted by Different Multi-View Diffusion Models

To demonstrate that GeoDream is trivially adaptable to various multi-view diffusion models, we conduct the visual comparison with our generated 3D assets based on either Zero123 or Zero123++. Specifically, for a fair comparison, the reference views are generated by MVDream driven by user-provided texts. Then, employing the viewpoint sampling strategy proposed in Sec.[10](#S10), we obtain source views predicted by Zero123 or Zero123++.
Fig.[11](#S17.F11) and Fig.[12](#S17.F12) show the comparison of our generated 3D assets based on source views predicted by Zero123 and Zero123++. Fig.[11](#S17.F11) and Fig.[12](#S17.F12) illustrate that GeoDream can adapt to different multi-view diffusion models, producing 3D assets with plausible geometry and intricate rendering details in visual appearance. The adaptability and seamless integration of GeoDream with various multi-view diffusion models highlight the evolutionary potential of GeoDream, alongside the future advancements of multi-view diffusion models.

**Table 3: URL, citations and licenses of the open-sourced assets we used in this work.**
| URL | Citation | License |
| --- | --- | --- |
| https://github.com/threestudio-project/threestudio | [] | Apache License 2.0 |
| https://github.com/bytedance/MVDream | [] | Apache License 2.0 |
| https://github.com/One-2-3-45/One-2-3-45 | [] | Apache License 2.0 |
| https://github.com/cvlab-columbia/zero123 | [] | MIT License |
| https://github.com/SUDO-AI-3D/zero123plus | [] | Apache License 2.0 |
| https://github.com/huggingface/diffusers | [] | Apache License 2.0 |
| https://github.com/allenai/objaverse-xl | [, ] | Apache License 2.0 |

Figure: Algorithm 1 Priors Refinement

Figure: Figure 11: Ablation on source views predicted by different multi-view diffusion models. We compare our generated 3D assets based on source views predicted by Zero123 and Zero123++. For a fair comparison, the reference views are generated by MVDream driven by user-provided texts. GeoDream adapt to source views predicted by various multi-view diffusion models. For each row from up to down, the given prompts are: (1) *A brightly colored mushroom growing on a log.* (2) *Mech robot with large weapons on top with hexagonal bases.* (3) *A small kitten.*
Refer to caption: /html/2311.17971/assets/x11.png

Figure: Figure 12: Ablation on source views predicted by different multi-view diffusion models. We compare our generated 3D assets based on source views predicted by Zero123 and Zero123++. For a fair comparison, the reference views are generated by MVDream driven by user-provided texts. GeoDream adapt to source views predicted by various multi-view diffusion models. For each row from up to down, the given prompts are: (1) *3D render of a statue of an astronaut.* (2) *A high quality photo of a dragon.* (3) *A cute rabbit in a stunning, detailed Chinese coat.*
Refer to caption: /html/2311.17971/assets/x12.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Table 2 appears to have a formatting issue—could you clarify the template used?
2. Why does Figure 3 present the generated mesh using rendered RGB rather than pure mesh? This choice limits the ability to assess mesh quality directly.

Please see more questions in the weaknesses section.
soundness: 3
strengths: 1. The disentangled approach to handling 2D and 3D priors effectively addresses the Janus problem, resulting in improved 3D consistency.
2. The framework supports rendering up to 1024x1024, surpassing most SDS-based methods in resolution.
3. GeoDream demonstrates robustness across a wide range of prompts and asymmetric structures.
summary: GeoDream presents an innovative approach to 3D asset generation by integrating disentangled 3D priors with 2D diffusion models. This method leverages a multi-view diffusion model to generate posed images, subsequently constructing a cost volume as a 3D prior to ensure spatial consistency. By separating 2D and 3D priors, GeoDream allows iterative refinement of these priors, enhancing the model’s 3D awareness without compromising the diversity or fidelity of generated 3D models.
weaknesses: 1. Lack of comparative analysis with state-of-the-art methods such as LucidDreamer (CVPR 2024), RichDreamer (CVPR 2024), and ImageDream (Arxiv 2023).
2. While primarily compared with SDS-based methods, GeoDream only includes Wonder3D as an exception. Given the limitations of SDS in terms of low-quality mesh and texture, comparisons with direct reconstruction methods such as InstantMesh (Arxiv 2024), One-2-3-45++ (CVPR 2024), and Era3D (NeurIPS 2024) would strengthen the evaluation.
3. The provided video reveals low mesh quality in certain outputs.
4. Additional pure mesh comparisons with other methods are needed to better assess mesh quality.


Reference:

[1] Liang, Yixun, et al. Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching. CVPR. 2024.

[2] Qiu, Lingteng, et al. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. CVPR. 2024.

[3] Wang, Peng, and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv. 2023.

[4] Xu, Jiale, et al. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv. 2024.

[5] Liu, Minghua, et al. "One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. CVPR. 2024.

[6] Li, Peng, et al. Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention. NeurIPS 2024.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: It would be better if the authors can show comparison with baseline methods in the videos.
soundness: 3
strengths: 1. Higher resolution text-to-3D generation
2. Disentangled 2D and 3D representation
3. Results are good
4. They combined Neus and DMTet as 3D representation
summary: This paper disentangles 2D and 3D priors on text-to-3D generation and can generation up to 1024 resolution.
weaknesses: 1. There are still some artifacts on the extracted mesh.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Authors should compare with SOTA baseline methods and demonstrate the advance of method design.
soundness: 2
strengths: 1. The ablation studies are sufficient to validate the effectiveness of the proposed components.
2. The paper is well written and easy to read.
summary: The paper tackles the task of Text-to-3D generation, addressing the technical challenge of the Janus problem that previous methods have encountered. The key insight and motivation is to leverage 2D diffusion priors to enhance the generation quality. The approach involves using a multi-view diffusion model to generate multi-view images, representing the 3D object with a cost volume, and then optimizing the 3D object using differentiable rendering from the generated multi-view images. In the second stage, the geometry decoder is fixed, and the texture decoder is optimized based on the SDS loss, resulting in improved 3D object generation from textual descriptions.
weaknesses: 1. The paper only compares with works from 2023 and does not include comparisons with more recent papers from CVPR 2024 and ECCV 2024. I have observed that some papers have better visualization results than this paper, such as One-2-3-45++. It would be beneficial for the paper to include these comparisons to demonstrate its standing in the current research landscape.
2. I do not understand the rationale behind the second phase, which is the optimization of the texture decoder using the SDS loss. In my view, if the multi-view diffusion model performs well, the SDS loss would be unnecessary, as evidenced by some recent papers. Is it because the multi-view diffusion model trained in this paper does not generate satisfactory results?
3. The use of cost volume to define the 3D representation seems odd. While I understand that cost volume adds 3D regularization, it has several drawbacks, including: (a) The cost volume essentially fuses multi-view features. If the input images have occlusion relationships, the cost volume cannot fuse to obtain the correct feature volume. (b) The spatial complexity of cost volume is quite large, which limits the resolution.
4. Overall, the paper's design shows some technical improvements compared to 2023's work, but it seems somewhat outdated compared to the work presented at CVPR 2024 and ECCV 2024. Although the ablation studies are complete, I still do not believe this paper is suitable for acceptance by ICLR.
5. The paper should present more generation results from the multi-view diffusion model to demonstrate its effectiveness, as this model is the core technology of the paper. It is also necessary to clarify how this diffusion model improves upon MVDream. Personally, I do not believe that improvements in 3D representation or training methods can bring about fundamental enhancements. The essence of 3D generation is to learn a stronger 3D prior, which may be represented by a multi-view diffusion model or a direct 3D generative model.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - Clarification of the novelty.
- clarification of  the sources of artifacts and inaccuracies in the generated geometry? These issues are visible in Fig. 1 and more pronounced in the video examples.
- How does GeoDream compare to RichDreamer (CVPR 2024) in terms of geometric detail and model accuracy?
- Could the authors provide a more thorough user study and ablation study to validate their approach across a broader range of metrics and prompts?
soundness: 3
strengths: - The method constructs a cost volume as native 3D priors by aggregating predicted multi-view 2D images in 3D space, enhancing multi-view consistency in the generated 3D models.
- By refining geometric priors through a 2D diffusion model, this work shows an improvement in rendering quality and geometric accuracy.
- Comparisons with baselines highlight the method's effectiveness in generating 3D-consistent textured outputs.
summary: This paper addresses the challenge of text-to-3D reconstruction with GeoDream, a method that integrates explicit 3D priors with 2D diffusion priors to capture clear, 3D-consistent geometric structures. The approach uses a multi-view diffusion model to generate posed images, then constructs a cost volume from these images as native 3D geometric priors. The 3D priors are integrated with 2D diffusion priors through a disentangled design. Comparisons with existing methods show that GeoDream generates more 3D-consistent textured meshes with strong semantic coherence.
weaknesses: The proposed method design seems incremental compared to the existing work like One-2-3-45 and has several key limitations:

- Generation Quality: The method's quality does not appear to surpass existing approaches. The visual results, even in the teaser of Fig. 1, reveal artifacts around object boundaries and inconsistencies in geometry, especially visible in mismatched normal maps and images. In the supplementary video, examples like the EAGLE, MUSHROOM, and GIRAFFE also exhibit incomplete geometry, likely due to limitations in the generated cost volume. A comparison with RichDreamer (CVPR 2024), which uses a normal-depth diffusion model as a geometry prior, would be valuable.

- Training of 3D Priors: The generalizabilty and quality of the 3D prior rely on the model for multi-view generation and cost-volume construction, raising questions about its generalizability and robustness for diverse prompts. The paper does not describe in detail how the 3D prior model is trained. The tested prompts cover a narrow range, making it difficult to assess performance across broader scenarios.

- Evaluation and Analysis: The evaluation could be stronger. For example, the user study had only 20 to 30 participants, which may be too small for reliable comparisons. Additionally, the ablation study lacks quantitative metrics to substantiate the design choices.

- Optimization Time: The method requires a significant optimization time, taking around 8 hours per instance, which limits its practicality for broader applications.

Meta-review: The submission received negative reviews from the reviewers. The reviewers' main concerns were mostly around the quality of the results and insufficient analysis. The authors did not submit a rebuttal. After reading the paper and the reviewers' comments, the AC agrees with the decision by the reviewers and recommends rejection.

Decision: Reject