DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation
Authors: Jiaxiang Tang
Jiawei Ren
Hang Zhou
Ziwei Liu
Gang Zeng
Venue: ICLR 2024 oral (2024)
Keywords: Text-to-3D
Image-to-3D
3D Generation
Efficiency
Primary area: generative models

Abstract: Recent advances in 3D content creation mostly leverage optimization-based 3D generation via score distillation sampling (SDS).
Though promising results have been exhibited, these methods often suffer from slow per-sample optimization, limiting their practical usage. 
In this paper, we propose DreamGaussian, a novel 3D content generation framework that achieves both efficiency and quality simultaneously. 
Our key insight is to design a generative 3D Gaussian Splatting model with companioned mesh extraction and texture refinement in UV space.
In contrast to the occupancy pruning used in Neural Radiance Fields, we demonstrate that the progressive densification of 3D Gaussians converges significantly faster for 3D generative tasks.
To further enhance the texture quality and facilitate downstream applications, we introduce an efficient algorithm to convert 3D Gaussians into textured meshes and apply a fine-tuning stage to refine the details.
Extensive experiments demonstrate the superior efficiency and competitive generation quality of our proposed approach.
Notably, DreamGaussian produces high-quality textured meshes in just 2 minutes from a single-view image, achieving approximately 10 times acceleration compared to existing methods.

Published as a conference paper at ICLR 2024
DREAMGAUSSIAN: GENERATIVE GAUSSIAN SPLAT- TING FOR EFFICIENT 3D CONTENT CREATION
Jiaxiang Tang'; Jiawei Ren", Hang Zhou², Ziwei Liu", Gang Zeng' National Key Laboratory of General AI, School of IST, Peking University 2 Ls-La, Nanyang Technological University Baidu Inc.
https://breamgaussinn...ttt github. .io/
'A photo of an Ine cream"
photo of hamburger" Input
Generated 3D Model
Mesh
Stage
56
40
60s
Stage
10s
30s
Optimization Progress
Figure 1: DreamGaussian aims at accelerating the optimization process of both image- and text-to- 3D tasks. We are able to generate a high quality textured mesh in several minutes.
ABSTRACT
Recent advances in 3D content creation mostly leverage optimization sinibd 3D generation via score distillation sampling (SDS). Though promising results have been exhibited, these methods often suffer from slow per-sample optimization, limiting their practical usage. In this paper, we propose DreamGaussian, a novel 3D content generation framework that achieves both efficiency and quality simul- taneously. Our key insight is to design a generative 3D Gaussian Splatting model with companioned mesh extraction and texture refinement in UV space. In con- trast to the occupancy pruning used in Neural Radiance Fields, we demonstrate that the progressive densification of 3D Gaussians converges significantly faster for 3D generative tasks. To further enhance the texture quality and facilitate down- stream applications, we introduce an efficient algorithm convert 3D Gaussians into textured meshes and apply a fine-tuning stage to refine the details. Exten- sive experiments demonstrate the superior efficiency and competitive generation quality of our proposed approach. Notably, DreamGaussian produces high-quality textured meshes in just 2 minutes from a single-vie view image, achieving approxi- mately 10 times acceleration compared to existing methods.
"This work was partly done when interning with Baidu Inc. and visiting NTU S-Labb

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Automatic 3D digital content creation finds applications across various domains, including digital games, advertising, films, and the MetaVerse. The core techniques, including image-to-3D and text-to-3D, offer substantial advantages by significantly reducing the need for manual labor among professional artists and empowering non-professional users engage in 3D asset creation. Drawing inspiration from recent breakthroughs in 2D content generation (Rombach et al., 2022), the field of 3D content creation has experienced rapid advancements. Recent studies in 3D creation can be classified into two principal categories: inference-only 3D native methods and optinization- baased 2D lifting methods. Theoretically, 3D native methods (Jun & Nichol, 2023; Nichol et al., 2022; Cupta et al., 2023) exhibit the potential to generate 3D-consistent assets within seconds, albeit at the cost of requiring extensive training on large-scale 3D datasets. The creation of such datasets demand substantial human effort, and even with these efforts, they continue to grapple with issues related to limited diversity and realism (Deitke ct al., 2023b;a; Wu ct al., 2023).
On the other hand, Dreamfusion (Poole et al., 2022) proposes Score Distillation Sampling (SDS) to address the 3D data limitation by distilling 3D geometry and appearance from powerful 2D diffusion models (Saharia et al., 2022), which inspires the development of recent 2D lifting methods (Lin et al., 2023; Wang et al., 2023b; Chen et al., 2023c). In order to cope with the inconsistency and ambiguity caused by the SDS supervision, Neural Radiance Fields (NeRF) (Mildenhall et al., 2020) usually adopted for their capability in modeling rich 3D information. Although the generation quality has been increasingly improved. these approaches are notorious for hours-long optimization time due to the costly NeRF rendering, which restricts them from being deployed to real-world applications at scale. We argue that the occupancy pruning technique used to accelerate NeRF (Müller al., 2022; Sara Fridovich-Ki Keil and Alex Yu et al., 2022) is ineffective in generative settings when supervised by the ambiguous SDS loss as opposed to reconstruction settings.
In this work, we introduce the DreamGaussien framework, which greatly improves the 3D con- tent generation efficiency by refining the design choices an optimization- based pipeline. Photo- realistic 3D assets with explicit mesh and texture maps can be generated from a single-view image within only 2 minutes using our method. Our core design is to adapt 3D Gaussian Splatting (Kerbl et al., 2023) into the generative setting with companioned meshes extraction and texture refinement. Compared to previous methods with the NeRF representation, which find difficulties in effectively pruning empty space, our generative Gaussian splatring significantly simplifies the optimization landscape. Specifically, we demonstrate the progressive densification of Gaussian splatting, which is in accordance with the optimization progress of generative settings, greatly improves the genera- tion efficiency. As illustrated in Figure I, our image-to-3) to-3 pipeline swiftly produces coarse shape within seconds and converges efficiently in around 500 steps on a single GPU.
Due to the ambiguity in SDS supervision and spatial densification, the directly generated results from 3D Gaussians tend to be blurry. To address the issue, we identify that the texture needs to be refined explicitly. which requires delicate textured polygonal mesh extraction from the generated 3D Gaussians. While this task has not been explored before, we design an efficient algorithm for mesh extraction from 3D Gaussians by local density querying. Then a generative UV-space refinement stage is proposed to enhance the texture details. Given the observation that directly applying the latent space SDS loss in the first stage results in ovtr-saturated blocky artifacts on the UV map, we take the inspiration from ceffusiin-bassee image editing methods (Meng et al., 2021) and perform image space supervision. Compared to previous texture refinement approaches, refinement stage achieves better fidelity while keeping high efficiency.
In summary, contributions are:
1. We adapt 3D Gaussian splatting into generative settings for 3D content creation, signifi- cantly reducing the generation time of optimization-baser 2D lifting methods.
2. We design an efficient mesh extraction algorithm from 3D Gaussians and UV-space tex- ture refinement stage to further enhance the generation quality.
3. Extensive experiments both Image-to-3D and Text-to-3D tasks demonstrate that our method effectively balances optimization time and generation fidelity, unlocking new pos- sibilities for real-world deployment of 3D content generation.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
2.1 3D REPRESENTATIONS
Various 3D representations have been proposed for different 3D tasks. Neural Radiance Fields (NeRF) (Mildenhall et al., 2020) employs volumetric rendering and has been popular for enabling 3D optimization with only 2D supervision. Although NeRF has become widely used in both 3D reconstruction (Barron et al., 2022; Li et al., 2023d; Chen et al., 2022; Hedman et al., 2021) and generation (Poole et al., 2022: Lin et al., 2023; Chan et al., 2022), optimizing NeRF can be time- consuming. Various attempts have been made to accelerate the training of NeRF (Müller et al., 2022; Sara Fridovich- Keil and Alex Yu ct al., 2022), but these works mostly focus on the reconstruction setting. The common technique of spatial pruning fails to accelerate the generation setting. Recently, 3D Gaussian splatting (Kerbl et al., 2023) has been proposed as an alternative 3D representation to NeRF, which has demonstrated impressive quality and speed in 3D reconstruction (Luiten et al., 2023). The efficient differentiable rendering implementation and model design enables fast training without relying on spatial pruning. In this work, we for the first time adapt 3D Gaussian splatting into generation tasks to unlock the potential of optimization- based methods.
2.2 TEXT-TO-3D GENERATION
Text-to-3D generation aims at generating 3D assets from a text prompt. Recently, data-driven 2D diffusion models have achieved notable success in text-to-image generation (Ho et al., 2020; Rom- bach et al., 2022; Saharia et al., 2022). However, transferring it to 3D generation is non-trivial due to the challenge of curating large-scale 3D datasets. Existing 3D native diffusion models usually work on single object category and suffer from limited diversity (Jun & Nichol, 2023; Nichol et al., 2022: Gupta al., 2023; Lorraine et al., 2023: Zhang et al., 2023; Zheng al., 2023; Ntavelis al., 2023; Chen et al., 2023b; Cheng et al., 2023; Gao et al., 2022). To achieve open-vocabulary 3D generation, several methods propose to lift 2D image models for 3D generation (Jain et al., 2022; Poole et al., 2022; Wang et al., 2023a; Mohammad Khalid et al., 2022; Michel et al., 2022). Such 2D lifting methods optimize a 3D representation to achieve a high likelihood in pretrained 2D diffusion models when rendered from different viewpoints, such that both 3D consistency and realisticity can be ensured. Following works continue to enhance various aspects such as generation fidelity and training stability (Lin et al., 2023; Tsalicoglou et al., 2023; Zhu & Zhuang, 2023; Yu et al., 2023; Li et al., 2023c; Chen et al., 2023d; Wang et al., 2023b; Huang et al., 2023; Metzer et al., 2022; Chen al., 2023c), and explore further applications (Zhuang et al., 2023; Singer al., 2023; Raj et al., 2023). However, these optimization- based 2D lifting approaches usually suffer from long per-case optimization time. Particularly, employing NeRF as the 3D representation leads expen- sive computations during both forward and backward. In this work, we choose 3D Gaussians as the differentiable 3D representation and empirically show that it has a simpler optimization landscape.
2.3 IMAGE-TO-3D GENERATION
Image-to-3D generation targets generating 3D assets from reference image. The problem can also be formulated as single-view 3D reconstruction (Yu et al., 2021; Trevithick & Yang, 2021; Duggal & Pathak, 2022), but such reconstruction settings usually produce blurry results due the lack of un- certainty modeling. Text-to-3D methods can also be adapted for image-to-3D generation (Xu et al., 2023a; Tang et al., 2023b; Delas-Kyriazi et al., 2023) using image captioning models ct,, 2022; 2023a). Recently, 7ero-1-to-3 (Liu et al., 2023b) explicitly models the camera transformation into 2D diffusion models and enable zero-shot image-conditions
novel view synthesis. It achieves high 3D generation quality when combined with SDS, but still suffers from long optimization time (Tang, 2022; Qian et al., 2023). One-2-3-45 (Liu et al., 2023a) trains a multi-view reconstruction model for acceleration at the cost of the generation quality. With efficiency-optimzze framework, our work shortens the image-to-31 optimization time to 2 minutes with little sacrifice on quality.
3 OUR APPROACH
In this section, we introduce our two-stage framework for efficient 3D content generation for both Image-to-3D and Text-to-3D tasks illustrated in Figure 2. Firstly, we adapt 3DD Gaussian splat-
3

---

Published as a conference paper at ICLR 2024
i) Generative Gaussian Splatting
it) Efficient Mesh Extraction
Text/ Tmae/Image
Densify
Local Density Query
Optimize
Diffusion Prior
Color Back-Projose Projection
Random
View
SDS Loss
3D Gaussians
3D Gaussians
Textured Mesh
ii
UV-Space Texture Refinement
Text/Image
Mapping
Add Noise
Denoising
Diffusion
Prior
Multi-step
Optimize
MSE Loss
UV Map
Random View
Refined Image
DreamGausee Framework. 3D Gaussians are used for efficient initialization of ge- ometry and appearance using single-step SDS loss. We then extract a textured mesh and refine the texture image with a multi-step MSE loss.
Figure 2:
ting (Kerbl et al., 2023) into generation tasks for efficient initialization through SDS (Poole et al., 2022) (Section 3.1). Next, we propose an algorithm to extract textured mesh from 3D Gaussians (Section 3.2). This texture is then fine-tuned by differentiable rendering (Laine et al., 2020) through a UV-space refinement stage (Section 3.3) for final exportation.
3.1 GENERATIVE GAUSSIAN SPLATTING
Gaussian splatting (Kerbl et al., 2023) represents 3D information with set of 3D Gaussians. It has been proven effective reconstruction settings (Kerbl et al., 2023; Luiten et al., 2023) with high inference speed and reconstruction quality under similar modeling time with NeRF. However, its usage in a generative manner has not been explored. We identify that the 3D Gaussians can be efficient for 3D generation tasks too.
Specifically, the location of each Gaussian can be described with a center x E TR3 scaling factor 3 and rotation quaternion q R44. We also store opacity value R and a color feature c E 3 for volumetric rendering. Spherical harmonics are disabled since we only want model simple diffuse color. All the above optimizable parameters presented by O, where O = {xi, S;, qi, Oi,, Cj is the parameter for the i-th Gaussian. To render a set of 3D Gaussians, we need project them onto the image plane as 2D Gaussians. Volumetric rendering is then performed depth order to evaluate the final color and alpha. In this work, we use the highly optimized renderer implementation from Kerbl et al. (2023) to optimize O
for each pixel in front-to-batt
We initialize the 3D Gaussians with random positions sampled inside sphere, with unit scaling and no rotation. These 3D Gaussians are periodically densified during optimization. Different from the reconstruction pipeline, we start from fewer Gaussians but densify it more frequently to align with the generation progress. We follow the recommended practices from previous works (Poole et al., 2022; Huang et al., 2023; Lin et al., 2023) and use SDS to optimize the 3D Gaussians (Please refer to Section A. for more details on SDS loss). At each step, we sample random camera pose p orbiting the object center, and render the RGB image IRGB and transparency I of the current view. Similar to Dreamtime (Huang et al., 2023), we decrease the timestep t. linearly during training, which is used weight the random noise added the rendered RGB image. Then, different 2D diffusion priors 6 can be used to optimize the underlying 3D Gaussians through SDS.
Image-to-3D For the image-to-3D task, an image i and a foreground mask 7 are given as input. Bero-1-to-3 XL. (Liu al., 2023b; Deitke et al., 2023b) is adopted as the 2D diffusion prior. The
4

---

Published as a conference paper at ICLR 2024
SDS loss can be formulated as:
VeLSDS = Beppe (et)( RGB Ap) - -c) a
(1)
de
where ve(t) is weighting function, <o(-) is the predicted noise by the 2D diffusion prior o, and Ap is the relative camera pose change from the reference camera r. Additionally, we optimize the reference view image RRG and transparency I to align with the input:
LR ARGB RGB J 11 + + K
(2)
where ARGB and AA are the weights which are linearly increased during training, The final loss is the weighted sum of the above three losses.
Text-to-3D. The input for text-to-3D is single text prompt. Following previous works, Stable- diffusion (Rombach et al., 2022) is used for the text-to-3D task. The SDS loss can be formulated as:
VeLsns Et, (e(())e (,,) e) 01 d0
where e is the CLIP embeddings of the input text description.
Discussion. We observe that the generated Gaussians often look blurry and lack details even with longer SDS training iterations. This could be explained by the ambiguity of SDS loss. Since each optimization step may provide inconsistent 3D guidance, it's hard for the algorithm to correctly densify the mndestrccontttttii regions prune over-reconstruution regions in reconstruction. This observation leads us to the following mesh extraction and texture refinement designs.
3.2 EFFICIENT MESH EXTRACTION
Polygonal mesh is widely used 3D representation, particularly in industrial applications. Many previous works (Poole et al., 2022; Lin et al., 2023; Tsalicoglou et al., 2023; Tang et al., 2023a) export the NeRF representation into a mesh-based representation for high-resolution fine-tuning. We also seek to convert the generated 3D Gaussians into meshes and further refine the texture.
To the best of our knowledge, the polygonal mesh extraction from 3D Gaussians is still an unex- plored problem. Since the spatial density is described by large number of 3D Gaussians, brute- force querying of a dense 3D density grid can be slow and inefficient. It's also unclear how to extract the appearance in 3D, as the color blending is only defined with projected 2D Gaussians (Kerbl et al., 2023). Here, we propose an efficient algorithm to extract a textured mesh based on block wise local density query and back- color.
Local Density Query. To extract the mesh geometry, a dense density grid needed to apply the Marching Cubes (Lorensen & Cline, 1998) algorithm. An important feature of the Gaussian splat- ting algorithm is that over-sized Gaussians will be split or pruned during optimization. This is the foundation of the tile-based culling technique for efficient rasterization (Kerbl et al., 2023). We also leverage this feature to perform block- wise density queries.
We first divide the 3D space of (-1, into 16 overlapping blocks, then cull the Gaussians whose centers are located outside each local block. This effectively reduces the total number of Gaussians query in each block. We then query 8 dense grid inside each block, which leads a final 128 dense grid. For each query at grid position x, we sum up the weighted opacity of each remained 3D Gaussian:
d(x) = Σ a: expl- exp((> - (x-xxxxxx)))))) x (xxx)))
(4)
where E,, is the covariance matrix built from scaling s; and rotation qr- An empirical threshold then used to extract the mesh surface through Marching Cubes. Decimation and remeshing (Cignoni et al., 2008) are applied to post-process the extracted mesh to make it smoother and more compact.
Color Back-projection. Since we have acquired the mesh geometry, we can back-project the ren- dered RGB image to the mesh surface and bake it as the texture. We first unwrap the mesh's 's UV coordinates (Young, 2021) (detailed in Section A.1) and initialize an empty texture image. Then, we uniformly choose azimuths and 3 elevations, plus the top and bottom views to render the cor- responding RGB image. Each pixel from these RGB images be back-projected to the texture
5

---

Published as a conference paper at ICLR 2024
Stage
Stage (SDS)
Stage (MSE)
Figure 3: Different Texture Fine-tuning Objectives. We show that SDS loss produces artifacts for UV space texture optimization, while the proposed MSE loss avoids this.
image based on its UV coordinate. Following Richardson et al. (2023), exclude the pixels with a small camera space z-direction normal to avoid unstable projection at mesh boundaries. This back back-projected texture image serves initialization for the next texture fine-tuning stage.
3.3 UV-SPACE TEXTURE REFINEMENT
We further use a second stage to refine the extracted coarse texture. Different from texture gen- eration (Richardson et al., 2023; Chen et al., 2023a; Cao et al., 2023), we hope to enhance the details given coarse texture. However, fine-tuning the UV-space directly with SDS loss leads to artifacts as shown in Figure 3, which is also observed in previous works (Liao al., 2023). This is due to the mipmap texture sampling technique used in differentiable rasterization (Laine et al., 2020). With ambiguous guidance like SDS, the gradient propagated to each mipmap level results in over-saturated color blocks. Therefore, seek more definite guidance to fine-tune blurry texture.
We draw inspiration from the image-to-image synthesis of SDEdit (Meng et al., 2021) and the re- construction settings. Since we already have initialization texture, Locarse from an arbitrary camera view p. Then, we perturb the image with random noise and apply multi-step denoising process f-) using the 2D diffusion prior obtaining refined image:
render blurry image
line = Sø (Poourse + <laaan); lwant,
(5)
where e(fstant is a random noise at timestep tstart. is Ap for image-to-3D and e for text-to-3D respectively. The starting timestep start is carefully chosen to limit the noise strength, so the refined image can enhance details without breaking the original content. This refined image is then used to optimize the texture through pixel-wise MSE loss:
LMSE TP
TP Tooars! 2
For image-to-3D tasks, still apply the reference view RGBA loss in Equation 2. We find that only about 50 steps can lead to good details for most cases, while more iterations can further enhance the details of the texture.
4
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We train 500 steps for the first stage and 50 steps for the second stage. The 3D Gaussians are initialized to 0.1 opacity and grey color inside sphere of radius 0.5. The rendering resolution is increased from 64 to 512 for Gaussian splatting, and randomly sampled from 128 to 1024 for mesh. The loss weights for RGB and transperency are linearly increased from to 10 and 10a during training. We sample random camera poses at fixed radius of for image-to-3D and 2.5 for text-to-3D, y-axis FOV of 49 degree, with the azimuth in [-180, 180) degree and elevation in [-30, 30) degree. The background is rendered randomly as white or black for Gaussian splatting. For image-to-3D task, the two stages each take around minute. We preprocess the input image by background removal (Qin et al., 2020) and recentering of the foreground object. The 3D Gaussians are initialized with 5000 random particles and densified for each 100 steps. For text-to-3D task, due to the larger resolution of 512 x 512 used by Stable Diffusion (Rombach al., 2022) model, each stage takes around 2 minutes to finish. We initialize the 3D Gaussians with 1000 random particles and densify them for each 50 steps. For mesh extraction,
empirical threshold of for

---

Published as a conference paper at ICLR 2024
Avg. Time Input
~0 minutes
~45 seconds
22 seconds
~ minutes
Zero-1-to-3
One-2-3-45
Shap-E
Ours
Figure 4: Comparisons on Image-to-3D. Our method achieves better balance between generation speed and mesh quality on various images.
Marching Cubes. All experiments are performed and measured with an NVIDIA V100 (16GB) GPU, while our method requires less than 8 GB GPU memory. Please check the supplementary materials for more details.
4.2 QUALITATIVE COMPARISONS
We first provide qualitative comparisons on image-to-3D in Figure 4. We primarily compare with three baselines from both optimization-basee methods (Liu et al., 2023b) and inference-only meth- ods (Liu et al., 2023a; Jun Nichol, 2023). For all compared methods, we export the generated models as polygonal meshes with vertex color texture images, and render them under ambient

---

Published as a conference paper at ICLR 2024
campfire"
"a small saguaro cactus planted a clay pot"
photo of tulip"
Avg. Time Prompt
~1 hour DreamFusion"
13 seconds Shap-E
5 minutes
Ours
Figure 5: Comparisons on Text-to-3D. For Dreamfusion, we use the implementation from Guo as the 2D prior.
ct al. (2023) which also uses
Type Inference-only Inference-only Inference-only
CLIP CLIP-Similarity
Generation Time +
One-2-3-45 (Liu et at., 2023a)
0.594
45 seconds
Point-E (Nichol et al., 2022)
0.587
78 seconds
Shap-E (Jun & Nichol,
0,591
27 seconds
Zero-I-to-3 (Liu et al., 2023b)
Optimization-Be
0.647
20 minutes
Zero-1-to-3" (Liu et al., 2023b)
Optimization-bans based
0.778
30 minutes
Ours (Stage Only)
Optimization-base
0.678
minute
Ours
aptimization-based
0.738
2 minutes
Table 1: Quantitative Comparisons on generation quality and speed for image-to-3D tasks. For Zero-I-to-3*, mesh fine-tuning stage is used further improve quality (Tang, 2022).
lighting. In terms of generation speed, our approach exhibits a noteworthy acceleration compared to other optimization-basee methods. Regarding the quality generated models, our method out- performs inference-only methods especially with respect to the fidelity of 3D geometry and vi- sual appearance. general, our method achieves a better balance between generation quality and speed, reaching comparable quality optinization-base than inference-only methods. In Figure we compare the results on tcxt-to-3D. Consistent with our findings in image-to-3D tasks, method achieves better quality than inference-based methods and faster speed than other optimization-bl based methods. Furthermore, we highlight the quality of our exported meshes in Figure These meshes exhibit uniform triangulation, smooth surface normals, and clear texture images, rendering them well-suited for seamless integration into downstream ap- plications. For instance, leveraging software such as Blender (Community, 2018), we can readily employ these meshes for rigging and animation purposes.
methods while only marginally slower
4.3 QUANTITATIVE COMPARISONS
In Table I, we report the CLIP-similarity (Radford et al., 2021; Qian et al., 2023; Liu et al., 2023a) and average generation time of different image-to-3D methods on collection of images from pre- vious works (Melas-Ke (Melas-Kyriazi et al., 2023; Liu et al., 2023a; Tang et al., 2023b) and Internet. We also conduct user study the generation quality detailed in Table 2. This study centers the assess- ment of reference view consistency and overall generation quality, which are two critical aspects in the context of image-to-3D tasks. Our two-stage results achieve better view consistency and gener-
8

---

Published as a conference paper at ICLR 2024
Mesh
Wireframe
Normal
Animation
Texture
Figure 6: Mesh Exportation. We export high quality textured mesh from 3D Gaussians, which can be seamlessly used in downstream applications like rigged animation.
Stage Full Setting
w/o Densification
w/o Timestep Annealing
w/o Reference View Loss
Figure 7: Ablation Study.
We ablate the design choices in stage training.
Zero-I-to-3
One-2-3-45
Shap-E
Ours 4.31
Ref. View Consistency Overall Model Quality 1 3.11
3,48
2.34
1.80
1.91
1.57
3.92
Table 2: User Study image-to-3D tasks. The rating is of scale 1-5, the higher the better.
ation quality compared to inference-only methods. Although our mesh quality falls slightly behind that other optimization-bnn based methods, we reach a significant acceleration of over 10 times.
4.4 ABLATION STUDY
We carry out ablation studies on the design our methods in Figure 7. We are mainly interested in the generative Gaussian splatting training, given that mesh fine-tuning has been well explored in previous methods (Tang et al., 2023a; Lin et al., 2023). Specifically, we perform ablation on three aspects of our method: 1) Periodical densification of 3D Gaussians. 2) Linear annealing of timestep I for SDS loss. 3) Effect of the reference view loss LRet- Our findings reveal that omission of any design elements results in degradation the generated model quality. Specifically, the final Gaussians exhibit increased blurriness and inaccuracies, which further affects the second fine-tuning stage.
5 LIMITATIONS AND CONCLUSION
In this work, we present DreamGausssion, a 3D content generation framework that significantly improves the efficiency 3D content creation. We design an efficient generative Gaussian splatting pipeline, and propose mesh extraction algorithm from Gaussians. With our texture fine-tuning stage, we can produce ready-to-use assets with high-quality polygonal meshes from either single image or text description within few minutes.
Limitations. We share common problems with previous works: Multi-face Janus problem, over- saturated texture, and baked lighting. It's promising to address these problems with recent advances in score debiasing (Armandpour et al., 2023; Hong et al., 2023), camera-conditions models (Shi al., 2023; Liu et al., 2023e; Zhao et al., 2023; Li et al., 2023b), and BRDF auto- encoder (Xu et al., 2023b). Besides, the back-view texture generated in our image-to-33 results may look blurry, which be alleviated with longer stage training.
2D diffusion

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
We share common ethical concerns to other 3D generative models. Our optimization-baset 2D lift- ing approach relies on 2D diffusion prior models (Liu et al., 2023b; Rombach et al., 2022), which may introduce unintended biases due to training data. Additionally, our method enhances the au- tomation of 3D asset creation, potentially impacting 3D creative professionals, yet it also enhances workflow efficiency and widens access to 3D creative work.
ACKNOWLEDGMENTS
This work is supported by the Sichuan Science and Technology Program (2023YFSY0008), Na- tional Natural Science Foundation of China (61632003, 61375022, 61403005), Grant SCITLAB- 20017 of Intelligent Terminal Key Laboratory of SiChuan Province, Beijing Advanced Innovation Center for Intelligent Robots and Systems (2018IRSS (8RS11), and PEK Sekk-SeneTTime Joint Laboratory of Machine Vision. This study also supported by the Ministry of Education, Singapore, under its 0012), NTU NAP, and under the RIE2020 Industry Align- ment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
MOE AcRF Tier 2 (MOE-T2EP2222,
REFERENCES
Mohammadreza Armandpour. Huangjie Zheng. Ali Sadeghian, Amir Sadeghian, and Mingyuan Zhou. Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond. arXiv preprint arXivv2304. arXiv:2304.04968 2023.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022. 3
Tianshi Cao, Karsten Kreis, Sanja Fidler, Nicholas Sharp, and Kangxue Yin. Texfusion: Synthe- sizing 3d textures with text-guided image diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 4169-41811 2023.
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. CVPR, 2022.
Dave Zhenyu Chen, Yawar Siddiqui, Hsin- Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303. 11396, 2023a.
Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single- stage diffusion nerf: unified approach to 3d generation and reconstruction. arXiv preprint arXiv:2304.06714, 2023b. 3
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.13877 2023c. 2,
Yiwen Chen, Chi Zhang, Xiaofeng Yang. Zhongang Cai, Gang Yu, Lei Yang, and Guosheng Lin. It3d: Improved text-to-3d generation with explicit view synthesis. arXiv preprint arXiv:2308.11471 11473, 2023d.
Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. Mobilenerf: Exploit- ing the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. arXiv preprint arXiv:2208.002277 2022.
Yen-Ch Cheng, Hsin- Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang- Yan Gui. Sdfu- sion: Multimodal 3d shape completion, reconstruction, and generation. In CVPR, pp. 4456-4465,
2023. 3
10

---

Published as a conference paper at ICLR 2024
Paolo Cignoni, Marco Callieri, Massimiliano Corsini, Matteo Dellepiane, Fabio Ganovelli, and Guido Ranzuglia. MeshLab: am Open-Source Mesh Processing Tool. In Vittorio Scarano, Rosario De Chiara, and Ugo Erra (eds.), Eurographics Italian Chapter Conference. The Eu- rographics Association, 2008. ISBN 978-3-905673-66-5. doi: 0023112
15
Blender Online Community. Blender - a 3D modelling and rendering puckage. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL http: www http://wwenee....... 8
Matt Deitke, Ruoshi Liu, Matthew Wallingford. Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. arXiv preprint arXiv:3307.05663, 2023a. 2
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In CVPR, pp. 13142-13153, 2023b. 2,
Shivam Duggal and Deepak Pathak. Topologically deformation fields for single-view 3d reconstruction. In CVPR, pp. 1536-1546, 2022. 3
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeie, and Sanja Fidler. Get3d: generative model of high quality textured shapes learned from images. Advances In Nearal Information Processing Systems, 35:31 841-31854, 2022. 3
Yuan-Chen Guo, Ying- Ying-Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia- Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song- Soi Zhang. threestudio: A unified framework for 3d content generation. https: / com/ whwrestudio-prrort. threestudio, 2023. 8
Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint trXiv:2303.03377 2023. 2,
Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul Debevee. Baking neural radiance fields for real-time view synthesis. ICCV, 2021. 3
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 33: 6840-6851, 2020. 3, 15
Susung Hong, Donghoon Ahn, and Seungryong Kim. Debiasing scores and prompts of 2d diffusion for robust text-to-3d generation. arXiv preprint UrXiv:2303.15413 2023.
Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi. Zheng-Jun Zha, and Lei Zhang. Dream- time: An improved optimization strategy for text-to-3d content creation. arXiv preprint arXiv:2306. 12422, 2023. 3,4
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In CVPR, pp. 867-876, 2022. 3
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463. 2023. 3.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ToG, 42(4):1-14, 2023. 2, 3, 4, 15
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint
arXiv: 1412.690, 2014. 15
Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high- reg-hperformance differentiable rendering. ToG, 39(6), 2020. 4, 6, 15
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified lasion-language understanding and generation. In ICML, pp. 12888-18 -29000 PMLR, 2022. 3
11

---

Published as a conference paper at ICLR 2024
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXhv:2301.12597, 2023a.
Weiyu Li, Rui Chen, Xuelin Chen, and Ping Tan. Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d. arXiv preprinl arXiv:2310.02596. 2023b. 9
Yuhan Li, Yishun Dou, Yue Shi, Yu Lei, Xuanhong Chen, Yi Zhang, Peng Zhou, and Bing- bing Ni. Focaldreamer: Text-driven 3d editing via focal-fusion assembly. arXiv preprint arXiv:2308. 10608, 2023c. 3
Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming- Yu Liu, and Chen-Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In CVPR, 2023d. 3
Tingting Liao, Hongwei Yi, Yuliang Xiu, Jiaxaing Tang, Yangyi Huang, Justus Thies, and Michael Black. Tada! text to animatable digital avatars. arXiv preprint arXiv:2308. 10899, 2023. 6
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler. Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, pp. 300-309, 2023. 2, 5,
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, al. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. arXiv preprint arXiv:2306. 16928, 2023a. 3, 7, 8, 15,
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 1-to-3: Zero-shot one image to 3d object. arXiv preprint arXiv:2303./1328, 2023b. 3, 4, 7, 8. 10, 16, 17
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consister images from a single-view image. arXiv preprint arXiv:2309.03453, 2023c. 9, 15
William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field, pp. 347-353, 1998. 5, 15
Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung- Yi Lin, Ming- Yu Liu, Sanja Fidler, and James Lucas. Aut3d: Amortized text-to-3d object synthesis. arXiv preprint (tt.:23306073499 202.3.
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint wrXiv:2308.009733 2023. 3, 4
Luke Melas-Kyriazz Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion: 360deg reconstruction of any object from a single image. CVPR, pp. 9446-8455, 2023. 3, 8, 15
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint 1xXi::2008.0103: 2021. 2, 6
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint arXiv:2211.07600 2022, 3
Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In CVPR, pp. 13492-13502, 2022. 3
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
2, 15
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Generat- ing textured meshes from text using pretrained image-text models. In SIGGRAPH Asia, pp. 1-8, 2022.3 3
12

---

Published as a conference paper at ICLR 2024
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphies prim- itives with a multiresolution hash encoding. ACM TOG, 2022. 2,3
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.0875.1 2022. 2, 3,
Evangelos Ntavelis, Aliaksandr Siarohin, Kyle Olszewski, Chaoyang Wang, Luc Van Gool, and Sergey Tulyakov. Autodecoding latent 3d diffusion models. arXiv preprint arXiv:2307.0544 2023. 3
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022. 2, 3, 4, 15
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306. 17843, 2023. 3, 8, 15, 16
Xuebin Qin, Zichen Zhang. Chenyang Huang. Masood Dehghan, Osmar R Zaiane, and Martin Jagersand. U2-net: Going deeper with nested u-structure for salient object detection. Pattern recognition, 106:107404, 2020. 6
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763, PMLR, 2021. 8
Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et Dreambooth3d: Subject-driven text-to-3d generation. arXiv preprint arXivv2233. 2023. 3
Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text- guided texturing of 3d shapes. arXiv preprint arXiv2230..01221, 2023. 6
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022. 2, 3, 5, 6, 10, 16
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 35:36479-36494, 2022. 2,
Sara tridovich-Keii and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022. 2,
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 165 2023. 9, 16
Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, lurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, al. Text-to-4d dynamic scene generation. arXiv preprint arXiv:2301. 11280, 2023. 3
Jiaxiang Tang.
Stable-dreamfussion: http://gthbbb.ooaaaaawwvv/bbbbbd
Text-to-3d 3, 8, 16
with
stable-diffusion, 2022.
Jiaxiang Tang, Hang Zhou, Xiaokang Chen, Tianshu Hu, Errui Ding, Jingdong Wang, and Gang Zeng. Delicate textured mesh recovery from nerf via adaptive surface refinement. arXiv preprint 0rXiv:2303.00009) 2023a. 5.9
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. arXiv preprint arXiv:2303. 14184, 2023b. 3, 8,
13

---

Published as a conference paper at ICLR 2024
Alex Trevithick and Bo Yang. Grf: Learning general radiance field for 3d representation and rendering. In ICCV, pp. 15182-15192, 2021.
Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. arXiv preprint arXi::2304.124335 2023. 3,5
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, pp. 12619-12629, 2023a. 3
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint 2rXiv:2305.162/3 2023b. 2,
Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang. Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, et al. Omniobject3d: Large-vocabulary 3d object dataset for realistic percep- tion, reconstruction and generation. In CVPR, pp. 803-8 814, 2023.
Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Wang. and Zhangyang Wang. Neurallift-360- Lifting in-the-wild 2d photo 3d object with 360deg views. In CVPR, pp. 4479 4489, 2023a.
Xudong Xu, Zhaoyang Lyu, Xingang Pan, and Bo Dai. Matlaher: Material-aware text-to-3d via latent brdf auto-encoder. arXiv preprint arX
2023b. 20236.
Jonathan Young. Xatlas, 2021. URL hitps:tpps...c /tt::///itttt
com/ http:///ccccmccccccccca/aaa/ jpcy: 15
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one few images. In CVPR, pp. 4578-4587, 2021.
Chaohui Yu, Qiang Zhou, Jingliang Li, Zhe Zhang, Zhibin Wang, and Fan Wang. Points-to-3d: Bridging the gap between sparse points and shape-controllable text-to-3d generation. arXiv preprint arXiv:2307.13990 2023. 3
Biao Zhang, Jiapeng Tang. Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape rep- resentation for neural fields and generative diffusion models. arXiv preprint arXiv:2301. 1144,,,,1114,, 2023.
Minda Zhao, Chaoyi Zhao, Xinyue Liang, Lincheng Li, Zeng Zhao, Zhipeng Hu, Changjie Fan, and Xin Yu. Efficientdreamer: High-fidelit and robust 3d creation via orthogonal-viet diffusion prior. arXiv preprint 2tt.:2308.1122233 2023. 9
Xin-Yang Zheng, Hao Pan, Peng-Shuai Wang, Xin Tong, Yang Liu, and Heung- Shum. Locally attentional sdf diffusion for controllable 3d shape generation. arXiv preprint arXiv:2305.04461, 2023. 3
Joseph Zhu and Peiye Zhuang. Hifa: High-fidelity text-to-3d with advanced diffusion guidance. arXiv preprini urXiv:2305. 18776066666..... 2023. 3
Jingyu Zhuang, Chen Wang, Lingjie Liu, Liang Lin, and Guanbin Li. Dreameditor: Text-driven 3d scene editing with neural fields. arXiv preprint ar arXiv:2306.1344 13455, 2023. 3
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 PRELIMINARY
Score Distillation Sampling (SDS). SDS was initially introduced by Dreamfusion (Poole et al., 2022), providing a framework that leverages pretrained 2D diffusion models as priors to optimize a parametric image generator. A representative example involves employing a differentiable 3D representation, such as NeRF (Mildenhall et al., 2020), the image generator:
x=
where x represents the rendered 2D image from the camera pose p, and ge(+) denotes the differen- tiable rendering function with optimizable NeRF parameters O The SDS formulation is expressed as:
0x ae
VeLSDS Et.p.e (()))c((x;;t)))-)
where t ~ (0.022 0.98) is a randomly sampled timestep, p is a randomly sampled camera pose (N, 1) is random Gaussian noise, w(l) of is weighting
orbiting the object center, function from DDPM (Ho et al., 2020), (x(-) ) is the noise predicting function with a pretrained parameters 6, and e is the text embedding. By optimizing this objective, the denoising gradient (x,(x; f, e) - e) that contains the guidance information is back-propagated to the rendered image x, which will be further back paepapaopa to the underlying NeRF parameters Ô through differentiable rendering (Mildenhall et al., 2020). Therefore, the NeRF can be optimized to form a 3D shape corresponding to the text description.
UV Mapping. UV Mapping is used to project 2D texture image onto the surface of 3D polygonal mesh. This requires to map each mesh vertex to a position on the image plane, which stored as the UV coordinates for each vertex. UV unwrapping (Young, 2021) is employed to automatically com- pute these UV coordinates given mesh. Retrieving the texture value at any surface point on a trian- gle involves barycentric interpolation to calculate the UV coordinate. We utilize NVdiffrast (Laine et al., 2020) for texture mapping and differentiable rendering, facilitating the optimization of the texture image through rendered images.
A.2 MORE IMPLEMENTATION DETAILS
Learning Rate. For the learning rate of Gaussian splatting,
different values for different
parameters. The learning rate for position is decayed from 1 x 10-3 to 2 x 10-5 in 500 steps, for feature is 0.01, for opacity is 0.05, for scaling and rotation is 5 10-3, For mesh texture fine-tuning, the learning rate for texture image is set to 0.2. We use the Adam (Kingma & Ba, 2014) optimizer for both stages.
Densification and Pruning. Following Kerbl et al. (2023), the densification in image-to-3D is applied for Gaussians with accumulated gradient larger than 0.5 and max scaling smaller than 0.05. In text-to-3D, we set the gradient threshold to 0.01 to encourage densification. We also prune the Gaussians with an opacity less than 0.01 or max scaling larger than 0.05.
Mesh Extraction. After extracting the mesh using Marching Cubes (Lorensen & Cline, 1998), we apply isotropic remeshing and quadric edge collapse decimation (Cignoni et al., 2008) control the mesh complexity. Specifically, we first remesh the mesh to an average edge length of 0.015, and then decimate the number of faces to 10
Evaluation Settings. We adopt the CLIP-similarity metric KMelas-Kyriazi et al., 2023; Qian et al., 2023; Liu et al., 2023a) to evaluate the image-to-3D quality. A dataset 30 images collected from previous works (Melas-Kyriazi et al., 2023; Liu et al., 2023a; Tang et al., 2023b; Liu et al., 2023c) and Internet covering various objects is used. We then render 8 views with uniformly sampled azimuth angles [0, 45, 90, 135, 180, 225, 270, 315) and zero elevation angle. These rendered images are used to calculate the CLIP similarities with the reference view, and we average the different T T gG- 4- a-llain222
views for the final metric. We use the lai /etooo//I/PP. checkpoint to calculate CLIP similarity. For the user study, we render 360 degree rotating videos
60k!
--V big--bbg----- 1 14- 1aaioonn---- 398-b1
15

---

Published as a conference paper at ICLR 2024
"a red fox"
"an astronaut, full body"
"an eagle"
"a cool bike"
Figure 8: Text-to-3D results with MVDream (Shi al., 2023) the guidance model.
w/ annealing
wo/ annealing
w/ annealing
wo/ annealing
Figure 9: Ablation on timestep annealing for text-to-3D, We use MVDream (Shi al., 2023) as the guidance model.
of 3D models generated from a collection of 15 images. There are in total 60 videos for 4 methods (Zero-I-to) (Liu et al., 2023b), One-2-3-45 Liu al. (2023a), Shap-E Jun Nichol (202), and method) to evaluate. Each volunteer is shown 15 samples containing the input image and rendered video from random method, and ask them rate two aspects: reference view consistency and overall model quality. We collect results from 60 volunteers and get 900 valid scores in total.
A.3 MORE RESULTS
Image-to-3D. In Figure 10, we show more visualization results of our method. Specially, we com- pare the mesh output before and after texture fine-tuning stage. We also compare against SDS-based mesh fine-tuning method for Zero-I-te 1-to-3 (Liu al., 2023b) noted as Zero-I-to-3* (Tang, 2022). Both stages of method are faster than previous two-stage image-to-3D methods, while still reaching comparable generation quality. Our method also support images with non-zero ele- vations. As illustrated in Figure 11, our method can perform image-to-3D correctly with extra estimated elevation angle as input. We make sure the random elevation sampling covers the input elevation and least 30, 30] degree.
tox--t--imag-----3.. In Figure 13, we demostrate the text-to-image-to-3D pipeline (Liu et al., 2023a; Qian et al., 2023). We first apply text-to-image diffusion models (Rombach al., 2022) to synthesize an image given text prompt, then perform image-to-3D using our model. This usu- ally gives better results compared to directly performing text-to-3D pipeline, and takes less time to generate. We show more animation results from our exported meshes in Figure 12.
Text-to-3D with MVDream. Figure 8, we show text-to-3D results using multi-view diffusion model MVDream (Shi et al., 2023) as the guidance. The multi-face Janus problem can be signifi- cantly mitigated by incorporating camera information to the 2D guidance model. However, still suffers from over-saturation and unsmooth geometry. We further perform ablation study the linear timestep annealing in Figure 9. With the timestep annealing, we find the model converges to more reasonable shape with the same amount of trianing iterations.
Limitations. We also illustrate the limitations of our method Figure 14. Our image-to-3ll pipeline may produce blurry back-view image and cannot generate fine details, which looks unmatched to the front reference view. With longer training of stage 2, the blurry problem of back view can be alleviated. For text-to-3D, share common problems with previous methods, including the multi- face Janus problem and baked lighting in texture images.
16

---

Published as a conference paper at ICLR 2024
Input
7em-1-to-3
7em-11tn-3*
Ours (Stage 1)
Ours (Stage 2)
Figure 10: More Qualitative Comparisons. We compare the results from two training stages of our method and Zero-I-to-3 (Liu et al., 2023b).
Input
Generated 3D Model
Mesh
Figure 11: Results on images with different elevations. Our method supports input images with non-zero elevation angle.
Figure 12: Results mesh animation. Our exported meshes ready-to-use for downstream applications like rigged animation.
17

---

Published as a conference paper at ICLR 2024
"a nendoroid of a cute boy"
"a nendoroid of cute girl"
"a penguin"
"a potted cactus plant"
"a 3D model of a fox"
"a 3D model of a soldier"
Prompt
Image
Generated 3D Model
Mesh
Figure 13: Text-to-image-to-3D.. We first synthesize an image given text prompt, then perform image-to-3D generation.
Front View (50 iters)
Back View (50 iters) Image-to-3D
Back View (500 iters)
Multi-face
Baked Lighting
Text-to-3D
Figure 14: Limitations. Visualization of the limitations of our method.
18


Reviews:
[{"review_id": "5JZaD9MX7p", "reviewer": "Reviewer_L9mE", "summary": "The present work presents a new methodology for generative 3d modelling using a 2d lifting approach. Its main underlying hypothesis is that using 3d Gaussian Splatting with its densification results in much faster convergence compared to using traditional neural radiance fields. On the technical side, this work focuses on two main contributions. First it proposes a mesh extraction technique based on the marching cubes algorithm for the setting of environment representations using 3d gaussians. Second, it proposes a UV-space texture refinement stage for further quality enhancement of the resulting textures.", "questions": "* Maybe rephrase the formulation \"to release the potential of optimization-based methods.\" to something like \"to unlock the potential. ...\" or something similar.\n* When writing \"we decrease the timestep t linearly, which is used to weight the random noise ϵ added to the rendered RGB image\", it is not fully clear to me how this is performed?\n* The SDS loss formulation is not clear without knowing the sds loss, e.g. the expectation is taken among other variables over p and t. What is the distribution of p and t? \n* Also, the evaluation section should be more explicit / more structured about the evaluation protocol and datasets used.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "Overall I think this paper to be interesting and to make relevant contributions. While most of the underlying ideas have already been presented in prior works, their combination for the presented use-case is relevant and results in significant performance improvement for the tasks of image-to-3d and text-to-3d.", "weaknesses": "While the work has overall a good contribution, one of my main concern is its writing. First, there are numerous minor language issues such as grammar mistakes and sometimes unnecessarily complicated sentence structure. This can easily be solved by a few rounds of careful proofreading. Second, the work reads more like a paper written for a computer vision conference without providing sufficient background to a broader audience at the ICLR community. While this style is not unprecedented in ML, it makes this work much harder accessible and misses an opportunity. Moreover, for people outside the specific subarea of 3d content creation some of the important implementation details may be missing.\n\nThese concerns range from minor points such as assuming the reader to be familiar with all mentioned vision / graphics concepts such as UV space etc without providing a proper background section. It also involves more complicated points such as the decision to provide some background (e.g. on SDS loss) but only to an extent that is only meaningful for people already familiar with dreamfusion. While in vision, many of these things can be assumed known, it would be useful to the learning community to provide some information here.\n\nAlso, I would rephrase the contribution bullet points to focus stronger on the technical aspects rather than first mentioning the overall framework, then dedicating one point to the actual technical meat and then talking about experimental evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "58OTkz4dfi", "reviewer": "Reviewer_E7Ru", "summary": "The paper addresses the problem of 3D content generation from text- or single view inputs into 3D gaussian splats which can further be transformed into textured meshes. The proposed method is coined DreamGaussian and revisits the recent ToG 2023 3D Gaussian Splatting paradigm with a generative twist to it.\n\nThe proposed contribution is favorably compared to a comprehensive set of state-of-the-art comparative baselines and in particular is able to produced high fidelity mehses of objects within 2min of compute time, which is remarkable.", "questions": "The main questions I would have cover the aforementioned weaknesses that have been pinpointed. In particular regarding the missing bits of informations.\n\nBesides those remaining grey areas, I would be happy to bump my initial rating were they to be addressed accordingly.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "+ ## Readability.\nAs it currently stands, the paper is very well written. The main ideas and concepts are mostly well explained and articulated throuthout.\n\n+ ## Organization of the contents and overall paper structure.\nThe contents are also very well structured and balanced.\n\n+ ## Overall maturity of the submitted package, which makes it very reasonably within camera ready territory.\n\n+ ## The actual performance of the proposed methodological contribution.\nIn particular, it dramatically cuts down the computation time in the space of optimization-based techniques in the field, by an order of magnitude.\n\n+ ## Related work section and discussion. \nIt is very well structued, articulated and populated with very relevant and up to date references.", "weaknesses": "+  ## 1. Missing bits of context information - How much does it cost?\nWhile indicative timings and implementation details (covering experimental setup,  are provided, information regarding the resource usage, model size and complexity are currently underdescribed.\n\nA comparative disclosure of such information covering the main experimental baselines that are considered would help the reader better assess its relative positioning throughout the typical criteria.\n\nMentioning where the computation bottlenecks lie in terms of pipeline components would also be valuable in order to fully assess the practical usefullness of the proposed sequential pipeline, beyond rough timings.\n\n+  ## 2. Challenging the ad hoc meshing post processing.\n\nAs it currently stands, the  mesh extraction technique relies on many subsequent post-processing steps, including mesh decimation and remeshing (end of page 5 in the main paper). I believe the explainations around eq. (4) (before and after) could be further improved and detailed. My current intuition is that the mesh complexity at least could be controled jointly during the density query step. Also, given the lack of statistics given regarding each step (Weakness 1 above), the relative need and ROI to fuse these steps is also hard to assess.\n\nThe current procedure also produces non-manifold and non-watertight meshes with arbitrary complexity.\n\n+  ## 3. Evaluation.\nThe user study (ie, subjective quality analysis) that is presented in the main paper and further detailed in the appendix is a good idea and often overlooked in the field. \n\nNevertheless, its size and statistical informative validity are rather limited as they are based on a \"cohort\" of 40 users and 15 input samples to assess from.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QSZBriKqml", "reviewer": "Reviewer_huut", "summary": "This paper proopsed a way to combine SDS loss and the recently proposed point-based rendering method GS. The pipeline is composed of 3 stages, 1) gaussian splatting optimization; 2) mesh extraction from point clouds; 3) texture refinement. The first stage is similar to other SDS-based methods which require pretrained 2d image diffusion models. However, the rendering method is switched to gaussian splatting. The second stage is done by applying marching cubes to opacities learned in the first stage. In the last stage, the texture is refined using 2D diffusion models. The full pipeline takes several minutes and we can obtain a mesh with textures. The authors show some results of image-conditioned and text-conditioned generation.", "questions": "See weakness section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has many strengths. Thanks to GS, the full pipeline is fast compared to some previous nerf-based methods. The author observed that a longer optimization of SDS does not give better results (sharp and detailed). Thus the focus of the method is not the SDS part. Instead, the authors only optimize the SDS loss with a few iterations (which is also the main reason why it is so fast). After obtaining the blurry 3D object, a refinement step inspired by diffusion-based image editing is applied. In the end, we can have a detailed mesh with textures. Another important aspect of this method, is the mesh extraction algorithm. Extracting a surface from a point cloud is not straightforward. The authors found out a way to use the opacity as the isosurface.\n\nWhen we are generating data, GS seems to be more suitable because of its progressive nature. The paper can be seen as a proof of this claim.", "weaknesses": "1. It seems the focus of the paper is image-conditioned generation. The results of text-to-3d are very limited and the comparison is weak.\n2. I am curious about the setup of the stage 3. If we already have a mesh with coarse texture, we can optimize it with differentiable mesh rendering and SDS loss, as Fantasia3d did in the appearance modeling stage. What is the advantage of the proposed refinement compared to this?\n3. Another concurrent ICLR submission ( https://openreview.net/forum?id=pnwh3JspxT ) optimizes SDS much longer (1 hour and 40 minutes) than this paper. However, this paper claims that longer training does not give better results. Can the authors clarify the differences?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CNmk9yYTjq", "reviewer": "Reviewer_MZY6", "summary": "DreamGaussian is a novel 3D content generation framework designed to efficiently produce high-quality 3D content. The core innovation is a generative 3D Gaussian Splatting model paired with mesh extraction and texture refinement processes in UV space. Unlike the occupancy pruning seen in Neural Radiance Fields, this approach uses progressive densification of 3D Gaussians, which results in faster convergence for 3D generation tasks. The framework also incorporates an algorithm to transform 3D Gaussians into textured meshes and employs a fine-tuning stage for detail refinement. In tests, DreamGaussian was able to produce high-quality textured meshes from a single-view image in just 2 minutes, marking a tenfold speed improvement over existing methods.", "questions": "Please consider the comments in the weaknesses section. I believe the current paper as it presents a great contribution to the field, and by addressing my current comment I am willing to increase my score even more.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The paper's main contribution is changing the NeRF representation to Gaussian Splatting representation in the current text-to-3D or image-to-3D pipeline. This has significant challenges in terms of implementation. Also, the authors' current implementation allows for fast generation of assets, which has significant importance in the 3D field.", "weaknesses": "Regarding the note “Similar to Dreamtime (Huang et al., 2023), we decrease the timestep t linearly”: Did the decrease of t help with the training time as well? Did it bring instability in training to the model? My previous exploration in this direction (for text to 3D in NeRF) showed improvement in the speed of generation for some objects but brought instability in the training. Since some objects needed more training steps than others, having a fixed annealing strategy damaged the performance, specifically for objects (e.g., motorcycles or dogs). It would be great if the authors could explore, in text to 3D, what the effect of the speed of annealing would be for different sets of prompts, (e.g., a corgi vs. a tree). Does the speed of annealing need to be tuned for each prompt?\n\n\nThe authors provided the experiment for time annealing in Fig 7 for image to 3D, but the paper is missing the same figure for text to 3D. Also, the effect of the speed of annealing needs to be explored.\n\nRegarding the loss function, eq. 6, for UV-Space texture refinement: what happens if the object boundary in the Img2Img stage changes? How do the authors prevent the color of the background from leaking into the mesh color?\n\nIt would be great if the authors could compare the proposed texture refinement to other SOTA methods like TexFusion [1] or Text2Tex [2] or any other method of their choice. The reason for this ask is because texture refinement has been applied on top of the mesh. So, if the authors want to consider texture refinement as one of their contributions, they should either compare it with other baselines or reframe the paper and consider this as a side contribution.\n\n\nI also tried experimenting with the code (great codebase!) and observed that, for text to 3D, the texture in most cases was very saturated. It would be great if the authors could comment on this phenomenon in the paper as a shortcoming and provide some insight into which parameter tuning might help.\n\n\nRegarding the Janus problem, the authors provided a list of papers that address the Janus problem mostly using 3D data. However, it would be great if the authors could also reference methods like Perp-Neg [3] or Prompt-Debiasing [4] that address the Janus problem without the need for 3D assets. This has significant importance because they don't introduce bias from 3D data into the pipeline. For instance, they allow for the generation of a dog without the strict square position for standing that comes from the 3D asset.\nIt would be great if the authors could also comment on the guidance of the SDS loss and the effect of increasing those values.\n\nThere are many typos in the paper, and it would be great if the authors could fix them. Examples are: on page 2, \"Gaussian splitting\" appears to be a typo; it should likely be \"Gaussian splatting\". “severl methods” on page 3 and “Dissusion” on page 5 is it supposed to be discussion?\n\nIt would also be great if the authors could comment on how to get the model to consider both the input image and the text prompt simultaneously in both the initial stage and the later mesh/texture optimization stage. At the moment, it seems to ignore the optional text prompt. Was this intentional?\n\nThe paper contrary to DreamFusion does not learn an MLP for the background, it would be great if authors could comment on why they made that choice and what are their findings. \n\n[1] https://openaccess.thecvf.com//content/ICCV2023/papers/Cao_TexFusion_Synthesizing_3D_Textures_with_Text-Guided_Image_Diffusion_Models_ICCV_2023_paper.pdf\n\n[2]https://openaccess.thecvf.com/content/ICCV2023/papers/Chen_Text2Tex_Text-driven_Texture_Synthesis_via_Diffusion_Models_ICCV_2023_paper.pdf\n\n[3] https://arxiv.org/abs/2304.04968\n\n[4] https://arxiv.org/abs/2303.15413", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)