DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation
Authors: Chenguo Lin
Panwang Pan
Bangbang Yang
Zeming Li
Yadong MU
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Generation
Diffusion models
3D Gaussian Splatting
Primary area: generative models
TL;DR: We introduce a novel 3D generative framework that directly generate 3D Gaussian splats by taming large text-to-image diffusion models, effectively utilizing 2D priors and maintaining 3D consistency in a unified model.

Abstract: Recent advancements in 3D content generation from text or a single image struggle with limited high-quality 3D datasets and inconsistency from 2D multi-view generation. We introduce DiffSplat, a novel 3D generative framework that natively generates 3D Gaussian splats by taming large-scale text-to-image diffusion models. It differs from previous 3D generative models by effectively utilizing web-scale 2D priors while maintaining 3D consistency in a unified model. To bootstrap the training, a lightweight reconstruction model is proposed to instantly produce multi-view Gaussian splat grids for scalable dataset curation. In conjunction with the regular diffusion loss on these grids, a 3D rendering loss is introduced to facilitate 3D coherence across arbitrary views. The compatibility with image diffusion models enables seamless adaptions of numerous techniques for image generation to the 3D realm. Extensive experiments reveal the superiority of DiffSplat in text- and image-conditioned generation tasks and downstream applications. Thorough ablation studies validate the efficacy of each critical design choice and provide insights into the underlying mechanism.

Published as a conference paper at ICLR 2025
DIFESPLAT: REPURPOSING IMAGE DIFFUSION MOD- ELS FOR SCALABLE GAUSSIAN SPLAT GENERATION
Chenguo Lin', Panwang Pan Bangbang Yang", Zeming Li,, Yadong Mu!!
PPeking University, "ByteDance https: http:///whhgggnnncc/c// github. io/prrojects/DDfffSlatt
ABSTRACT
Recent advancements in 3D content generation from text or a single image strug- gle with limited high-quality 3D datasets and inconsistency from 2D multi-view generation. We introduce SPAAT, a novel 3D generative framework that na- tively generates 3D Gaussian splats by taming large-scale text-to-image diffusion models. It differs from previous 3D generative models by effectively utilizing web-scale 2D priors while maintaining 3D consistency in unified model. To bootstrap the training, a lightweight reconstruction model is proposed to instantly produce multi-view Gaussian splat grids for scalable dataset curation. In conjunc- tion with the regular diffusion loss on these grids, a 3D rendering loss is intro- duced to facilitate 3D coherence across arbitrary views. The compatibility with image diffusion models enables seamless adaptions of numerous techniques for image generation to the 3D realm. Extensive experiments reveal the superiority of DIFFSPLAT in text- and image-conditioned generation tasks and downstream ap- plications. Thorough ablation studies validate the efficacy each critical design choice and provide insights into the underlying mechanism.
INTRODUCTION
Generating 3D content from a single image text long-standing challenge with wide range of applications, such as game design, digital arts, human avatars, and virtual reality. It is highly ill-posed problem that requires reasoning the unseen parts of any object in the 3D space only from single view or textual descriptions, posing great challenge to both fidelity and generalizability.
With the development of diffusion generative models (Sohl-Dickstein et al., 2015; Ho et al., 2020), recent works train conditional 3D generative networks directly on datasets of various 3D represen- tations (Nichol et al., 2022: Jun & Nichol, 2023; Cao et al., 2024; He et al., 2024; Zhang et al., 2024b), demonstrated in Figure 1 (1), or only using 2D supervision with the help of differentiable rendering techniques (Anciukevičius et al., 2023; Karnewar al., 2023b; Szymanowicz al., 2023; Xu et al., 2024d) as in Figure 1 (2). Despite 3D consistency, they limited by the supervision scale and can't 't utilize 2D priors from abundant pre- pre-trained models. Current advanced generalizable 3D content creation methods (Li al., 2024a; Wang et al., 2024; Tang al., 2024) reconstruct implicit 3D representations from generated multi-view images using pretrained 2D diffusion models (Wang & Shi, 2023; Shi et al., 2023; Voleti al., 2024), illustrated in Figure 1 (3). Although these two- stage methods can reconstruct high-quality 3D content from multi-view posed images, the synthesis pipeline collapses and fails produce faithful results when generated images from upstreamed 2D multi-view diffusion models are of poor quality or inconsistency.
To overcome the drawbacks of previous works, we present DIFFSPLAT, novel 3D generative frame- work that exhibits multi-view consistency and effectively leverages generative priors from large- scale image datasets. We adopt 3D Gaussian Splatting (3DGS) (Kerbl al., 2023) as the 3D content representation for its efficient rendering and quality balance. Instead of relying on time-consuming per-instance optimization to obtain 3D datasets for training (He et al., 2024; Zhang et al., 2024b), we represent 3D object by set of well-structured splat 2D grids. During the training stage, these grids can be instantly regressed from multi-view images in less than 0.1 seconds, facilitating scalable and high-quality 3D dataset curation. Each Gaussian splat in 2D grids holds properties that imply object texture and structure. Noting that image diffusion models trained on web-scale datasets are capable
t: Project lead; t: Corresponding author.

---

Published as a conference paper at ICLR 2025
Rendering Loes
Diffusion Loss
3D Network
3D Network
3D Content
(I) Native
(2) Rendering-basee
Diffasion Rendering Loss
Diffasion
Rendering Loss
2D
Network
Nerwerk
Network
MuRt Images
3D Cuntent
(3) R Reconstruction-bssee
(4) DiftSplat (Ours)
Figure 1: Comparison with Previous 3D Diffusion Generative Models. (1) Native 3D methods and (2) rendering- based methods encounter challenges training 3D diffusion models from scratch with limited 3D data. (3) Reconstruction-basee methods struggle with inconsistencies in generated multi-view images. In contrast, (4) DIFESPLAT leverages pretrained image diffusion models for the direct 3DGS generation, effectively utilizing 2D diffusion priors and maintaining 3D consistency. "GT" refers to ground-truth samples in a 3D representation used for diffusion loss computation.
of 3D geometry estimation (Li et al., 2024b; Ke et al., 2024; Fu et al., 2024), we find that by treating reconstructed Gaussian splat 2D grids as images in special style, we can harness the power of pretrained 2D image diffusion models for direct 3DGS generation.
Specifically, open-sourced latent diffusion models (Rombach et al., 2022; Podell et al., 2024; Chen et al. 2024c;b; Esser et al., 2024) trained web-scale image datasets are repurposed to directly gen- erate Gaussian splat properties for 3D content creation. To ensure that reconstructed splat grids have the same shape as the input latents of image diffusion models, we also fine-tune their VAEs to com- press Gaussian splat properties into a similar latent space, which are coined as splat latents. During shown in Figure 1 (4), in addition
the training of SIFASPLAT, latents, which resembles regular image diffusion models, we propose to incorporate a rendering loss enable the generative model to function in 3D space and facilitate 3D consistency, since Gaus- sian splat properties are processed in the network and can be differentially rendered in arbitrary views. Furthermore, thanks to the minimal modifications on 2D denoising network architectures, various pretrained text-to-image diffusion models can serve as the base model for DIFFSPLAT, and techniques proposed for them can be seamlessly adapted the realm of generation within our framework, establishing a bridge between 3D content creation and the image generation community.
standard diffusion loss on splat
Our contributions can be summarized as follows:
A novel 3D generative framework that directly generates 3D Gaussian splats by fine-tuning image diffusion models, effectively utilizing 2D priors and maintaining 3D consistency.
Abundant pretrained image diffusion models and associated techniques can be adapted for the proposed model, seamlessly connecting 3D generation and the image community.
Extensive experiments demonstrate the superior performance of the proposed method, and ablation studies are conducted to analyze the effectiveness of each design choice.
2 RELATED WORK
Native 3D Generative Models Diffusion-based models (Ho et al., 2020; Liu et al., 2023) have emerged as the de facto approach for generative models. The most straightforward solution for 3D content creation is to train a 3D denoising network on 3D datasets as shown in Figure 1 (1), termed as "native 3D generative models". Numerous native 3D methods are proposed for explicit 3D representations, such as voxel (Sanghi et al., 2023; Ren et al., 2024) and point cloud (Nichol et al., 2022; Mo et al., 2023), which are readily available but often result in poor visual quality. On the other hand, r representations like implicit functions (Jun & Nichol, 2023; Liu et al., 2024a; Zhang et al., 2024d; Li et al., 2024c; Chen et al., 2024d), triplanes (Cao et al., 2024; Liu et al., 2024b; Zhang et al., 2024a; Wu et al., 2024) and 3DGS (Henderson et al., 2024; He al., 2024; Zhang
2

---

Published as a conference paper at ICLR 2025
et al., 2024b) offer more faithful appearances but are usually inaccessible and require extra time- intensive preprocessing, making it impractical to maintain a large-scale dataset. Moreover, native 3D generative models face limitations in leveraging pretrained 2D models, posing great challenges to the quality and scale of 3D datasets, as well as the efficiency of 3D network training from scratch.
Rendering-basee based 3D Generative Models Instead of relying on time-consuming 3D dataset cura- tion, with differentiable rendering techniques (Mildenhall et al., 2020), some works propose to train 3D generative models with only 2D supervision, which are called "rendering- gandering-based generative mod- els" and shown in Figure 1 (2). These methods aim to denoise images rendered from corrupted implicit 3D representation the absence of ground-truth clean 3D data. HoloDiffusion (Karnewar et al., 2023b) and HoloFusion (Karnewar et al., 2023a) propose bootstrapped latent diffusion strat- egy, in which feature volumes are denoised twice to solve the problem that there is no ground-truth rendering volume for radiance fields. Compared to RenderDiffusion (Anciukevičius et al., 2023) that reconstructs clean triplane features from single-view noised images, Viewset Diffusion (Szymanow- icz et al., 2023) and GIBR (Anciukevicius et al., 2024) denoise multi-view images simultaneously, ensuring coherent and plausible appearance and geometry for the intermedia rendering volume from sufficient viewpoints. DMV3D (Xu et al., 2024d) further scales up to highly diverse datasets con- taining nearly 1 million objects. However, it is extremely computationally intensive and costs 128 A100 GPUs for 7 days to train a unified model for 2D denoising and 3D reconstruction without any 2D generative prior, suffering from the same drawback of native 3D methods.
Reconstruction-aasel 3D Generative Models By scaling up network parameters and training datasets, Large Reconstruction Model (LRM) (Hong et al., 2024b; Tochilkin et al., 2024), a feed- forward Transformer encoder (Vaswani et al., 2017), can reconstruct a triplane field (Chan et al., 2022) from single image. It only needs multi- view images for supervision akin rendering- based methods, but operates deterninistically rather than for denoising. As presented in Figure (3), reconstruction-basen methods leverage 2D priors by utilizing frozen image diffusion models to generate multiview images (Shi et al., 2024; Wang & Shi, 2023; Shi et al., 2023; Voleti et al., 2024; Han et al., 2024) with generalizable 3D reconstruction models. Instant3D (Li et al., 2024a) follows LRM and produces triplane features from posed images in 2x2 grids generated by fine-tuned large image diffusion model (Podell et al., 2024). Some methods (Wang et al., 2024; Wei et al., 2024; Xu et al., 2024b; Siddiqui et al., 2024; Boss et al., 2024) replace the rendering representation from triplane to FlexiCubes (Shen et al., 2021; 2023) or VoISDF (Yariv et al., 2021) to directly extract meshes. Other methods produce Gaussian attributes from pixels (Tang et al., 2024; Xu et al., 2024c; Zhang et al., Icaraaae Iearnable tokens (Chen et al., 2024a) or latent features (Pan ct al., 2024). However, all these methods regard the multi-view diffusion model as independent plug-and-play module, so 3D generation is conducted as a two-stage proceeding, which requires extra network parameters and may collapse due to the 3D inconsistency in generated images.
3 METHOD
Motivated by the effectiveness of web-scale pretrained image diffusion models in estimating 3D geometry attributes, such as depth (Stan et al., 2023; Ke et al., 2024), coordinates (Li al., 2024b; Xu et al., 2024a) and normal (Long et al., 2024; Fu et al., 2024), our goal in this work is taming image diffusion models to directly generate 3D content. As illustrated in Figure 2, the proposed method consists of three parts: (1) scalable 3D data curation by structured splat reconstruction (Sec. 3.1), (2) splat latents (Sec. 3.2), and (3) the generative model SPLAS (Sec. 3.3).
3.1 DATA CURATION: STRUCTURED SPLAT RECONSTRUCTION
Inspired by generalizable 3DGS reconstruction techniques (Szymanowicz et al., 2024; Charatan et al., 2024), we utilize a set of structured multi-view Gaussian splat grids to represent a 3D object. Specifically. given Vn posed images in R3xH* small network Fo can regress per-pixel splat from these contextualized images in under 0.1 seconds, and is trained by the rendering loss Grender:
Crender =
(CMSE (1u:1 LGT) + Xp LLPIPS (L. 1 + Ao CMSE(MMM))) MCT)), (I)
where I, and M,, are respectively RGB images and silhouette masks differentially rendered from predicted 3DD Gaussian primitives g == (g: N. via efficient rasterization technique (Kerbl et al.,
3

---

Published as a conference paper at ICLR 2025
Image Conditin
Todt Condition
rosket skeheboard
Loicat Excoder
Luncet Desadar
Aepensentatin
(2) Splat Latents
dr (c
Prstrained Tust tat-iteirenge Daftaate Model ?
Resonstration ModelFF,
Bandettggoo
Ganocin Proviitonn
Hulth-stew w/ Gecmearric Griddnnce
Diffuctinn
(1) Data Curation:
(3) DifiSplat Generative Model
Structured Splat Reconstruction
Figure 2: Method Overview. (1) lightweight reconstruction model provides high-quality struc- tured representation for "pseudo" dataset curation. (2) Image VAE is fine-tuned to encode Gaussian splat properties into a shared latent space. (3) DIFFSPLAT is natively capable of generating 3D contents by image and text conditions utilizzig 2D priors from text-to-image diffusion models.
2023) across random V viewpoints, and N V x H x W due to the pixel-aligned prediction. CMSE and LLPIPS stands for mean squared error loss and VGG-based perceptual loss (Zhang al., 2018). "GT" denotes ground-truth data for supervision, and Ap, Xa E IR are hyper-parameters to adjust the relative importance of three different losses.
Each Gaussian primitive g, E R is parameterized by its RGB color R², location x E R3 scale R3 rotation quaternion r E R and opacity R. To simplify the parameterization and regulate the distribution of primitives, location x is determined by depth d E R, camera intrinsic K e R3x3 and extrinsic matrices (rotation R. SO(3) and translation t R$), x = R R [u|d] is homogeneous pixel coordinates u E R2 with depth. For numerical values of Gaussian splat properties to confined within [0, 1] in preparation of diffusion- generation, outputs P's all activated by the sigmoid function o-)), except for which is Ln-nomaalized to yield unit quaternions. RGB color с and opacity o are already supposed to be in [0, 1]. Raw scale é linearly interpolated with predefined values Smin and Smax (Xu et al., 2024c):
t.
S
= Smin (s) + Smax (1 oS)).
(2)
Raw depth is defined to relative to the image plane, and objects our datasets can be normalized into a [-1, 1]s cube, allowing its value to be restricted in [0, 1] by the sigmoid function:
2 a
||t||2-
Different from previous reconstruction-bit based methods (Tang et al., 2024; Xu et al., 2024c; Zhang et al., 2024c), besides multi-view posed RGB images, we also incorporate corresponding coordinate maps (Shotton et al., 2013) and normal maps additional inputs for auxiliary geometric guidance. Note that these extra inputs are only employed to enhance the reconstruction quality of Gaussian splat grids and the efficacy of geometric guidance alongside RGB images in Sec. 4.5.1.
required during the generation stage. Ablation study is conducted assess
3.2 SPLAT LATENTS
Aforementioned designs allow for high-quality representation 3D objects using multi-view splat grids g = { Y which are structured for image-like processing, obtained efficiently, and con- fined within a suitable numerical range. To encode Gaussian splat grids image diffusion latent space, the most straightforward idea is to split them into multiple feature groups with 3 channels to analog RGB images, and process them separately. However, this results in poor auto-encoding quality, as encoded latents display notable deviations from natural images, as shown in Sec. 4.5.1.
Instead, duplicate the columns and rows of pretrained input and output convolution weights times respectively to match the feature dimensions of Gaussian splat grids G, E 122xXXX VAEs for latent image diffusion models (Rombach et al., 2022; Podell et al., 2024; Esser al., 2024)
4

---

Published as a conference paper at ICLR 2025
then fine-tuned to auto auto-encode each Gaussian splat grid independently with both reconstruction loss and rendering loss:
Vin
DVVE == V Σ
1
(CMSE(Gi, G.)) Ar Crender (9);
is auto-encoded G; by the VAE's encoder Es, and decoder DoL- and Crender computed across V random views as presented in Equation 1. X, is the weighting term for the rendering loss. Encoded Gaussian splat grids from Ví input views
where G. == : ( (G:))
(G.)) Vnn are referred to as splat latents, which undergoes diffusion and denoising process in DIFFSPLAT. Rendering loss Lrender is significant for high-quality splat latent auto-encoding, and quantitative evaluations are provided in Sec. 4.5.1.
3.3
DIFFSPLAT GENERATIVE MODEL
3.3.1 MODEL ARCHITECTURE
Given a set of multi-view splat latents z {z,} )= structured as 2D grids, two widely adopted manners for multi-view image generation are explored in this work to simultaneously generate z as a whole from text prompts or single-view images, coined as view-concat and spatial-concal.
In the view-concat manner, Vn splat latents of an objects, shaped as Rdxhxw, are treated like video frames and concatenated along the view dimension into R R. Xdxxxxxxx and processed individually by the denoising network, except the self-attention module, where they are reshaped into R R(Vi---wex and treated as integral sequence (Long et al., 2024; Shi et al., 2024; Kant et al., 2024; Gao et al., 2024b). While for the spatial-concal manner, splat latents are organized into an r x e grid along the height and width dimensions, resulting in a shape of 2023; Li et al., 2024a; Gao et al., 2024a). In both manners, Plücker embeddings (Sitzmann et al., 2021) are concatenated along the feature dimension with respective splat latents, enabling dense encoding of relative camera poses. It facilitates better flexibility in viewpoint selection and reduces requirements for multi-view datasets. The only new parameters introduced to pretrained models are zero-initialized new columns in the input convolution weights for Plücker embeddings. These model designs result in minimal modifications and generalizability across various text-to- text-to-image diffusion models (Rombach et al., 2022; Podell et al., 2024; Chen al., 2024c;b; Esser et al., 2024).
where Vn = r x (Shi al.,
Unlike multi-view image diffusion models (Li ct al., 2024a; Kant et al., 2024), it's not feasible for text-conditioned DUFFSPLAT to simply denoise other views except for the input image view for image-conditioned generation, as the input condition (pixels) and generated outputs (splat proper- ties) are in different domains. For view-concat, the original VAE-encoded input image is concate- nated along the view dimension, with additional dense binary masks concatenated along the feature dimension to distinguish between image and splat latents. For spatial-concat, the input image is padded with blank background to form an r X grid, and then concatenated along the feature dimension after the image VAE encoding. Ablation study on the multi-view manners for both text- and mnage-conditioned 3DGS generation tasks is conducted in Sec. 4.5.2.
3.3.2 TRAINING OBJECTIVES
DIFFSPLAT Fe can be trained with the regular diffusion loss Lair, which aims to denoise corrupted splat latents á = AddNo se t) from a randomly sampled noise level t:
Can = w(): ( 1 2/12-
Here, iddNoi se(:) corrupts the sample z with random Gaussian noises ~ AO, 1) at the noise level é (Ho et al., 2020; Nichol & Dhariwal, 2021; Karras et al., 2022; Liu et al., 2023), and w(t) is the weighting term of diffusion loss at different noise levels (Song et al., 2021; Salimans & Ho, 2022; Hang al., 2023; Kingma & Gao, 2023). Text and image conditions omitted for concision.
However, optimizing solely with Lans similar to multi-view image diffusion models (Shi et al., 2024; 2023; Li et al., 2024a), does not guarantee 3D consistency. This limitation stems from the model essentially operating in the 2D space of Gaussian splat grids, and the stereo correspondences among Vnn viewpoints are learned implicitly during the fine-tuning of image diffusion models, which
5

---

Published as a conference paper at ICLR 2025
originally trained single-view natural images. Therefore, it makes the fine-tuning process less straightforward for achieving consistent learning in a 3D context. Moreover, treating splat latents ground-trume samples, which are compressed after lightweight reconstruction model, also limits the upper bound the generative model, as real multi-view datasets are not involved in the training process, relying completely the results from reconstruction and auto-encoding models.
Recognizing that splat latents are processed during the diffusion process, not as pixels but as a natural 3D representation that can be efficiently rendered from arbitrary views, we propose incorporate an additional rendering loss Lrender, as defined in Equation 1, alongside Laitr. where denoised splat latents are decoded back into Gaussian splat properties and rendered from random V viewpoints, with supervision from ground-trun -ouud multi-view images. The final training objective is:
DIISSMLAT = Adift Lain + Avender (1) frender Deo (F (2, ()),
(6)
where wi, (1) is the weighting term of the rendering loss at different noise levels, and Adir, Arender are used adjust the importance of two types of losses.
Notably, by setting Adir = O, DIFFSPLAT effectively becomes a rendering-based model, but denois- ing splat latents instead of pixels. On the other hand, by setting >render = 0, DIFFSPLAT transforms into a "pseudo" native 3D model by treating splat latents as a pscudo ground-truth 3D representation. The effectiveness of the two types of losses is evaluated in Sec. 4.5.2.
4
EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
All our models in this work are trained on G-Objaverse (Qiu et al., 2024), high-quality subset of Objaverse (Deitke et al., 2023) and comprising images from 38 different views of around 265K 3D objects. Captions of these 3D objects are provided by Cap3D (Luo et al., 2023; 2024). To quantita- tively evaluate the performance of text-conditioned generation, 300 text prompts from T3Bench (He et al., 2023), describing single object, single object with surroundings and multiple objects, are employed as conditions. CLIP similarity score (Radford et al., 2021) and CLIP R- R-Precision (Park et al., 2021) based on vi V-B//2 are used to measure the alignment of input prompts and rendered images, and ImageReward (Xu et al., 2023) is used to reflect human aesthetic preference. For both reconstruction and mongeccondiiioee generation task, 300 objects from the unseen GSO (Downs et al., 2022) dataset are randomly selected and rendered to serve as ground-truth images, which are then compared with rendered images from reconstructed generated 3D contents in terms of PSNR, SSIM and LPIPS (Zhang et al., 2018) metrics. All metrics are averaged across different viewpoints for 3D-aware evaluation. Implementation details are provided in Appendix A.
4.2 TEXT-CONDITIONED GENERATION
Baselines Four state-of-the-an open-sourced methods that support native text-to-3D generation are evaluated, where GVGEN (He et al., 2024) uses Gaussian volume to represent 3D objects, while triplane-based NeRF is used in LN3Diff (Lan et al., 2024), DIRECT-3D (Liu et al., 2024b) and 3DTopia (Hong et al., 2024a). Two reconstruction- based methods LGM (Tang et al., 2024) and GRM (Xu et al., 2024c) can support text-to-3D generation by associating with open-source text- conditioned multi-view diffusion model (Shi et al., 2024).
Results and Comparisions As demonstratee in Table and Figure 3, DIFFSPLAT exhibits the best prompt alignment and visual quality among cutting-egge text-conditioned 3D generation methods, especially for complex prompts. In contrast, 3D native methods struggle to match text prompts due to limited text-3D pairs for training from scratch, while reconstruction-base methods suffer from milti-view diffusion inconsistency, particularly for objects with surroundings or other objects. More visualization results are provided in Appendix Figure 9, 10 and 11.
4.3 IMAGE-CONDITIONED GENERATION
Baselines Two up-to-date native 3D models that support image-conditioned generation are com- pared here: the concurrent work 3DTopia-XL (Chen et al., 2024d) and LN3Diff (Lan et al., 2024). Six advanced reconstruction- based methods for single image-conditioned generation are also eval- uated, including three Gaussian Splatting-based (Kerbl et al., 2023) methods: LGM (Tang et al.,

---

Published as a conference paper at ICLR 2025
Text Prompt
DifISplat (Ours)
GVGEN
LN3DaI
3DTopia
LGM
coctus pink nowers
ceramic teapot with Rorrl patherns
classic leatherette radio with
A while tranquil lake
Figure 3: Qualitative Results and Comparisons Text-conditioned 3D Generation. More visualizations of DIFFSPLAT results are provided in Appendix Figure 9, 10 and 11.
Table 1: Quantitive evaluations on T3Bench prompts for text-conditioned generation. indicates reconstruction- based methods that require additional text-conditioned multi-vien generative models.
DILA
GVGEN
LN3Diff
DIRECT-3D
3DTopia
LGM
GRM
CLIP Sim.z.
30.95
23.66
24.36
24.80
25.55
29.96
28.19
Single Object
t CLIP R- R-Prec.gg
81.00
23.25
27.25
30.75
34.50
78.00
64.75
ImageReward
-0.491
-2.156
-2.008
-2.005
-1.998
-0.720
-1.337
Single Object
1 CLIP Sim.s.
30.20
22.65
22.75
23.05
24.31
27.79
26.24
CLIP R-Prec.o
80.75
26.75
22.00
25.75
39.00
55.00
51.25
w/ Sur.
ImageReward
-0.674
-2.251
-2.244
-2,191
-2.230
-1.772
-1.869
t CLIP Cim.g
29.46
21.48
21.65
21.89
22.88
27.07
24.33
Multiple Objects
t CLIP R-Prec.g
69.50
8.00
8.75
16.50
51.00
26.50
ImageReward
-0.849
-2.272
-2.267
-2,249
-2.225
-1.731
-2.116
2024), GRM (Xu et al., 2024c) and LaRa (Chen et al., 2024a), and two FlexiCube-based (Shen et al., 2023) methods: CRM (Wang et al., 2024) and InstantMesh (Xu et al., 2024b). Image genera- tive models for these reconstruction methods are selected following their original implementations.
Results and Comparisions Single image-conditioned generation performance on the GSO dataset assessed in Table and qualitative results on in-the-wild images are presented in Figure 4 and Appendix Figure 12, 13 and 14. DIFASPLAT delivers accurate 3D content aligned with input images while maintaining strong geometric fidelity compared to other state-of-the methods.
4.4 APPLICATION: CONTROLLABLE GENERATION
Thanks to the compatibility of DIFFSPLAT with image diffusion models, numerous techniques ini- tially developed for image generation can be easily adapted for 3D applications. Here, we explore ControlNet (Zhang et al., 2023) to facilitate controllable generation guided by flexible formats along- side text prompts in Figure 5. DIFTSPLAT can generate diverse and high-quality 3D assets that ac- curately respond to different control inputs, such as normal and depth maps, and Canny edges, while faithfully reflecting the text conditions. Moreover, while most previous reconstruction methods not incorporate text understanding. the flexible conditioning design allows DIFFSPLAT to perform text-guided reconstruction from single-view ambiguous images, shown in Appendix Figure 8.
4.5 ABLATION AND ANALYSIS
We carefully investigate each design choice for splat latent reconstruction and DIFFSPLAT 3D generation in this subsection. Ablation studies are conducted based on Stable Diffusion V1.5 (SD1.5) (Rombach et al., 2022) unless otherwise specified.

---

Published as a conference paper at ICLR 2025
Input Image
DiftSplat (Ours)
3DTopie-X1
LN3Diff
LGM
InstantMesh
Figure 4: Qualitative Results and Comparisons Image-conditioned 3D Generation. More visualizations of DIFFSPLAAT results are provided in Appendix Figure 12, 13 and 14.
Table 2:
Quantitative evaluations on GSO for image-conditioned generation. t indicates reconstruc- tion methods that require additional image generation models for single image-to-3D generation.
DIFFSPLAT
3TTopia-XI.
LN3Diff
I.GM
GRM
LaRa
CRM
InstantMesh
PSNR
22.91
17.27
16.67
18.25
19.65
18.87
18.56
19.14
SSIM
0.892
0.840
0.831
0.841
0.869
0.852
0.855
0.876
L.PPPS
0.107
0.175
0.177
0.166
0.141
0.202
0.149
0.128
4.5.1 SPLAT LATENT RECONSTRUCTION
Reconstruction Inputs Effectiveness of geometric guidance for the reconstruction model is val- idated Table 3. Gaussian Splatting-basee large reconstruction model LGM (Tang et al., 2024) and GRM (Xu al., 2024c) are also evaluated with sparse-view RGB images for comparison. Al- though with much fewer parameters, with the help of coordination and normal maps, the proposed lightweight reconstruction model can instantly provide sigl-tuaitty Gaussian splat grids as "pseudo" ground-trun representation for 3D generation. Coordination maps explicitly indicate the positions of Gaussian primitives, thus providing more effective geometric guidance than normal maps.
Auto-encoding Stragegies We investigate different training strategies for Gaussian splat property auto-encoding in Table 4. Freezing the original image VAE or its encoder results in poor perfor- mance, as Gaussian splat properties differ significantly from natural images. Rendering loss plays crucial role in auto-encoding by ensuring that the VAE is supervised by real datasets rather than being limited by the lightweight reconstruction model, thus enabling the auto-encoded splats to per- form slightly better than the reconstructed ones. VAFs from SD1.5 and SDXL. (Podell et al., 2024) have a similar performance with the same dimension (d = of latent space, while SD3 (Esser al., 2024) shows improved performance with d = 16.
4.5.2 DIFFSPLAT 3D GENERATION
Multi-view Manners Two popular multi-view manners explored in this work, yielding sim- ilar results in text-conditioned 3D generation, as shown in Table 5. View-concat manner performs better in single image-conditioned generation due to the dense attention among conditional image latents and splat latents. Although the spatial-concat method can also achieve dense attention by concatenating image latents along the spatial dimension, it requires additional padding, leading to increased computational costs. Thus, we prefer the view-concat manner this work for its flexibility in accommodating varying numbers of viewpoints and conditioning.
Training Objectives DIFTSPLAT can perform well with merely the regular diffusion loss by set- ting Avender = given high-quality splat latents. However, as shown in Table 5 and Figure 6, the proposed 3D rendering loss Frender can further boost both the aesthetic quality and geometric struc- ture of generated content. Aesthetic appeal and textured details may contributed by the perceptual loss, and fewer translucent artifacts achieved through the mask loss described in Equation 1.
8

---

Published as a conference paper at ICLR 2025
Original Object
Input Control
A steampunk robot with
A cute cartoon robot with oversized eyes
bross gears steam pipes
A Santa festive plush bear toy
An adorable baby panda
An ancient, leather-bound magic book with shimmering gold leaf pages
A slice of freshly baked bread with
golden crusty exterior soft interior
Figure 5: Controllable Generation. ControlNet seamlessly adapt to DIFFSPLAT for control- lable text-to-3D generation in various formats, such as normal and depth maps, and Canny edges.
Table 3: Ablation study of inputs for structured Table 4: erty auto-encoding strategies.
Ablation study for Gaussian splat prop-
splat reconstruction.
t PSNR
SSIM
LPIPS
#Param.
PSNR
SSIM
LPIPS
LGM
26.48
0.892
0.077
415M
Frozen VAB
8.64
0.833
0.566
GRM
28.04
0.959
0.031
179M
Frozen Ess.
24.73
0.927
0.065
RGB
27.43
0.956
0.041
42M
w/o Lrender
26.07
0.937
0.080
Normal
28.89
0.957
0.033
42M
Ours (SD1.5)
30.33
0.966
0.033
Coord.
29.87
0.961
0.028
42M
Ours (SDXL)
30.30
0.964
0.031
Ours
30.09
0.963
0.027
42M
Ours (SD3)
31.08
0.978
0.025
DIFFSPLAT also produces reasonable results only with the rendering loss by setting Aui 0. How- ever, the training process becomes unstable and slow to converge, and gets over-saturated results.
Base Text-to-image Diffusion Models Various popular open-source large text-to-image diffusion models are investigated in this work, including SD1.5 (Rombach et al., 2022). SDXL (Podell et al., 2024), PixArt-a (Chen al., 2024c), PixArt-> (Chen al., 2024b) and SD3 (Esser et al., 2024), fea- turing different model sizes, backbone networks, noise scheduling, and sampling strategies. With the advancements base models, DIFASP consistently benefits in both text- and image-conditioned tasks, indicating that the proposed method effectively leverages priors from pretrained models and stands as promising approach for scaling 3D generation within the thriving image community.
4.5.3 INTERPRETING SPLAT LATENTS
To understand the feasibility of generating Gaussian splat properties through fine-tuning image diffusion models, we visualize splat latents and their corresponding proper- ties to interpret the mechanisms behind DIFFSPLAT. As shown in Figure 7, auto-encoded Gaussian splat proper- ties are presented
CSCALE
CS0cd
moosees
055et 2-6
GARK
RGB or grayscale images. For ro- tations r E IR,, the first three channels and the last one are visualized individually. Splat latents encoded by a fine-tuned VAE are decoded by the original image VAE. Gaussian splat property grids are analogous to natural im- ages, reflecting the hue and edge properties of 3D objects. Decoded splat latents from the image VAE can be inter- preted as the original objects in "special style" or illu- minated in a "special environment light", featuring a cyan light at distance and rufous light nearby. Image diffu- sion models are essentially fine- tuned to learn this special style, indicating that the input latents are splat latents, which enables the repurposing of image diffusion models to generate 3DGS.
Figure 7: Splat Latents Visualization. 3DGS properties are structured in grids. "Decoded GS" shows the splat latents decoded by an image diffusion VAE.

---

Published as a conference paper at ICLR 2025
Table 5; Ablation study of DPLAT design choices.
T3Bench-300
GSO-300
CLIP Sim-g
CLIP R-Prec.
IeageReward
PSNR
SSIM
J. L.PPPSS
Mulli-view Mavner
Spatial-conen
28.741008
55.92
-102241a0m3
21.61 16.36
0.875
0,121 1009
View-concat
28.66*014
58,92+2.0
=aus1
22.58
0.885
+0017
Training Objective
Lan
27.721009
50.83
-144361003
17.162320
0.794
0.267
w/o Crender
28. 28.54
56.42+227
-1.219
22.30
0.883
4.125
Cueaar Lai
28.66:0014
58,92
-1.201 1:0031
22.58
0.885 0.,022
0.117
Base Model (#Parom.)
SD1.5 (0.86B)
28.66+014
58,92=2.30
11.201 #cuns1
22.58
0.885
0.117
SDXL (2.6B)
29.701022
13.00
-0 10.04
22. 22.65
0.88710008
1.0.00
PixArt-a (0.61B)
29. 19.10
#9.01
=1.052
22.8 #5500
0.884
PixArt-E (0.61B)
29.75:+000
64.83+
-0.834=a007
22.85 45.75
0.877
0.105
+0000
SD3-medium (2.0B)
30.15
68.08:2072
-0.6855
22.91
0.892
0.107
A plush dragon toy
an old car overgrown by vines and weeds
wi Ereader
wio Evender
w/ Egender
w/o Eyender
wi feenddrr
wio Erendder
wi Lrender
w/o Eyendur
Figure 6: Ablation of Lrender. Both text- (1st row) and image-conditioned (2nd row) DIFFSPLAT with Frender produces more aesthetic and textured 3D content with fewer translucent floaters.
CONCLUSION
In this work, we present a novel diffusion-based 3D generation framework, DIFFSPLAT. It distin- guishes from previous 3D generative methods by effectively leveraging web-scale 2D priors while maintaining 3D coherence in a unified model. Various large text-to-image diffusion models are fine- tuned to directly generate 3D Gaussian splat properties with both diffusion loss and 3D rendering loss. Thus, DIFFSPLAT benefits from the rapid developments in the image community, facilitat- ing the integration of advanced techniques for image generation into the 3D realm. It positions DIFFSPLAT a promising 3D generative method utilizing abundant 2D priors and merely multi-view supervision. We hope this work can provide new solution for 3D content creation.
Limitations and Future Work Although DIFFSPLAT delivers decent results, the conversion of its 3DGS representation to high-quality mesh remains an unsolved problem. Simultancously generating splat latents from more viewpoints, increasing the supervision resolution, and integrating physical- based material properties could further enhance the quality of generated 3D Gaussian primitives. Numerous advanced techniques developed for image diffusion models can be adopted for 3D gener- ation within the proposed framework, which remains underexplored, such as personalization. few- step distillation, and feedback learning to align with human preferences. Moreover, we only utilize rendered multi-view datasets in this work, which does not fully exploit the scalability potential of the proposed method. By leveraging its image compatibility and 2D supervision-only characteristic, massive real-world video datasets could further unlock the capability of DIFFSPLAT.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work is supported by grant from ByteDance (No. C120240607105799) and an internal grant of Peking University (2024JK28).
REFERENCES
Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy Mitra, and Paul Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and genera- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Titas Anciukevicius, Fabian Manhardt, Federico Tombari, and Paul Henderson. Denoising diffusion via image-based rendering. In International Conference on Learning Representations (ICLR), 2024.
Mark Boss, Zixuan Huang, Aaryaman Vasishta, and Varun Jampani. S13d: Stable fast 3d mesh reconstruction with uv-unwrapping and illumination disentanglement. arXiv preprint arXiv:2408. 3,05:, 2024.
Ziang Cao, Fangzhou Hong, Tong Wu, Liang Pan, and Ziwei Liu. Large-vocabulary 3d diffusion model with transformer. In International Conference on Learning Representations (ICLR), 2024.
Eric R Chan, Connor 7. Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmano, pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
Anpei Chen, Haofei Xu, Stefano Esposito, Siyu Tang, and Andreas Geiger. Lara: Efficient large- baseline radiance fields. In European Conference on Computer Vision (ECCV), 2024a.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision (ECCV), 2024b.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-a: Fast training of diffusion transformer for photore- alistic text-to-image synthesis. In International Conference on Learning Representations (ICLR), 2024c.
Zhaoxi Chen, Jiaxiang Tang. Yuhao Dong, Ziang Cao, Fangzhou Hong, Yushi Lan, Tengfei Wang, Haozhe Xie. Tong Wu, Shunsuke Saito, Liang Pan, Dahua Lin, and Ziwei Liu. 3dtopia-xl. xl: High- quality 3d pbr asset generation via primitive diffusion. arXiv preprint arXiv:2409. 10044099.29990 /2957, 2024d.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In International Conference on Robotics and Automation (ICRA), 2022.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning (ICML), 2024.
11

---

Published as a conference paper at ICLR 2025
Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In European Conference on Computer Vision (ECCV), 2024.
Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-12x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint MeXiv:2445.05944 2024a.
Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo BertinnBruuall,, Pratul Srinivasan, Jonathan Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2024b.
Junlin Han, Filippos Kokkinos, and Philip Torr Vfusion3d: Learning scalable 3d generative models from video diffusion models. In European Conference on Computer Vision (ECCV), 2024.
Tiankai Hang, Shuyang Gu, Chen Li, Jiammin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), 2023.
Xianglong He, Junyi Chen, Sida Peng, Di Huang, Yangguang Li, Xiaoshui Huang, Chun Yuan, Wanli Ouyang, and Tong He. Gvgen: Text-to-3d generation with volumetric representation. In European Conference on Computer Vision (ECCV), 2024.
Yuze He, Yushi Bai, Matthieu Lin, Wang Zhao, Yubin Hu, Jenny Sheng, Ran Yi, Juanzi Li, and Yong-Jin Liu. bench: Benchmarking current progress in text-to-3d generation. arXiv preprint arXiv:2310.02977, 2023.
Paul Henderson, Melonie de Almeida, Daniela Ivanova, and Titas Anciukevičius. Sampling 3d gaussian scenes in seconds with latent diffusion models. arXiv preprint arXiv:2406. 20024406.3999 13099, 2024.
Jonathan Ho and Tim Salimans. Classifier-free diffuusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Fangzhou Hong, Jiaxiang Tang, Ziang Cao, Min Shi, Tong Wu, Zhaoxi Chen, Tengfei Wang, Liang Pan, Dahua Lin, and Ziwei Liu. 3dtopia: Large text-to-3d generation model with hybrid diffusion priors. arXiv preprint 2403::2000.023:4 2024a.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. In International Conference on Learning Representations (ICLR), 2024b.
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint 023:::3305.02443,, 2023.
Yash Kant, Aliaksandr Siarohin, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Guler, Bernard Ghanem, Sergey Tulyakov, and Igor Gilitschenski. Spad: Spatially aware multi- view diffusers. Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Animesh Karnewar, Niloy J Mitra, Andrea Vedaldi, and David Novotny. Holofusion: Towards photo-realistic 3d generative modeling. In Proceedings of the IEEF/CVF International Confer- ence on Computer Vision (ICCV), 2023a.
Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy Mitra. Holodiffusion: Training a 3d diffusion model using 2d images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
12

---

Published as a conference paper at ICLR 2025
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-basee image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
Bernhard Kerbl. Georgios Kopanas, Thomas Leimkuchler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 2023.
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Yushi Lan, Fangzhou Hong, Shuai Yang, Shangchen Zhou, Xuyi Meng. Bo Dai, Xingang Pan, and Chen Change Loy. Ln3diff: Scalable latent neural fields diffusion for speedy 3d generation. In European Conference Computer Vision (ECCV), 2024.
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-vien gen- eration and large reconstruction model. In International Conference on Learning Representations (ICLR), 2024a.
Weiyu Li, Rui Chen, Xuelin Chen, and Ping Tan. Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d. In International Conference Learning Representations (ICLR), 2024b.
Weiyu Li, Jiarui Liu, Rui Chen, Yixun Liang, Xuelin Chen, Ping Tan, and Xiaoxiao Long. Crafts- man: High-fidelity mesh generation with 3d native generation and interactive geometry refiner. arXiv preprint arXiv:2405. 14979, 2024c.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference Learning Representations (ICLR), 2023.
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. 2ne-2-3-45++: Fast single image 3d objects with consistent multi-view generation and 3d diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024a.
Qihao Liu, Yi Zhang, Song Bai, Adam Kortylewski, and Alan Yuille. Direct-3d: Learning direct text-to-3d generation massive noisy 3d data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024b.
Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In International Conference on Learning Representations (ICLR), 2023.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Iyaa Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Interna- tional Conference on Learning Representations (ICLR), 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations (ICLR), 2018
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems (NeurIPS), 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX0v:221..01095, 2022b.
13

---

Published as a conference paper at ICLR 2025
Tiange Luo. Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pre- trained models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Tiange Luo, Justin Johnson, and Honglak Lee. View selection for 3d captioning via diffusion rank- ing. arXiv preprint wrXiv:2404.0798. 2024.
Mildenhall, PP Srinivasan, M. Tancik, JT Barron, R Ramamoorthi, and R Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
Shentong Mo, Enze Xie, Ruihang Chu, Lanqing Hong, Matthias Niessner, and Zhenguo Li. Dit-3d: Exploring plain diffusion transformers for 3d shape generation. In Advances in Neural Informa- tion Processing Systems (NeurIPS), 2023.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning (ICML), pp. 8162-8171, 2021.
Panwang Pan, Zhuo Su, Chenguo Lin, Zhen Fan, Yongjie Zhang, Zeming Li, Tingting Shen. Yadong Mu, and Yebin Liu. Humansplat: Generalizable single-imag human gaussian splatting with structure priors. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann. Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations (ICLR), 2024.
Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning (ICML), 2021.
Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/VVF Con- ference on Computer Vision and Pattern Recognition (CVPR), 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference Learning Representations (ICLR), 2022.
Aditya Sanghi, Rao Fu, Vivian Liu, Karl DD Willis, Hooman Shayani, Amir H Khasahmadi, Srinath Sridhar, and Daniel Ritchie. Clip-sculptor: Zero-shot generation of high-fidelity and diverse shapes from natural language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hyyrid representation for righuuesolution 3d shape synthesis. In Advances Neural Information Processing Systems (NeurIPS). 2021.
14

---

Published as a conference paper at ICLR 2025
Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flexible isosurface extraction for gradient- based mesh optimization. ACM Transactions Graphics (TOG), 2023.
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol23++: a single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310.15110 2023.
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. In International Conference on Learning Representations (ICLR), 2024.
Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
Yawar Siddiqui, Filippos Kokkinos, Tomand Monnier, Mahendra Kariya, Yanir Kleiman, Emilien Garreau, Oran Gafni, Natalia Neverova, Andrea Vedaldi, David Novotny, and Roman Shapo- valov. Emu3d: Text-to-mesh generation with high-quality geometry, texture, and pbr materials. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning (ICML), 2015.
Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations (ICLR), 2021.
Gabriela Ben Melech Stan, Diana Wofk, Scottie Fox, Alex Redden, Will Saxton, Jean Yu, Estelle Aflalo, Shao- Yen Tseng, Fabio Nonato, Matthias Muller, et al. Ldm3d: Latent diffusion model for 3d. arXiv preprint arXiv:2305. 10853, 2023.
Stanislaw Szymanowicz. Christian Rupprecht, and Andrea Vedaldi. Viewset diffusion:(0-)) image- conditioned 3d generative models from 2d data. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), 2024.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang. Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision (ECCV), 2024.
Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from single image. arXiv preprini 2xXiv:2403.0215:: 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems (NeurIPS), 2017.
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Chris- tian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. In European Conference on Com- puler Vision (ECCV), 2024.
15

---

Published as a conference paper at ICLR 2025
Peng Wang and Yichun Shi. Imagedream: Image-prone propppt multi-view diffusion for 3d generation. arXiv preprint arXiv:2112.0220,, 2023.
Zhengyi Wang, Yikai Wang. Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image 3d textured mesh with convolutional reconstruction model. In European Conference on Computer Vision (ECCV), 2024.
Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh. arXiv preprint arXiv:2404. 125.::2544..2885, 2024.
Shuang Wu, Youtian Lin, Feihu Zhang, Yifei Zeng, Jingxi Xu, Philip Tomr Xun Cao, and Yao Yao. Direct3d: Scalable image-to-3e -dd generation via 3d latent diffusion transformer. In Advances in Neural Information Processing Systems (NeurIPS). 2024.
Chao Xu, Ang Li, Linghao Chen, Yulin Liu, Ruoxi Shi, Hao Su, and Minghua Liu. Sparp: Fast 3d object reconstruction and pose estimation from sparse views. In European Conference on Computer Vision (ECCV), 2024a.
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from single image with sparse-view large reconstruction models. arXiv preprint XXXv:2404.07791,, 2024b.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Yinghao Xu, Zifan Shi. Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. In European Conference on Computer Vision (ECCV), 2024c.
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi- view diffusion using 3d large reconstruction model. In International Conference on Learning Representations (ICLR), 2024d.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Bowen Zhang, Yiji Cheng, Chunyu Wang, Ting Zhang, Jiaolong Yang. Yansong Tang, Feng Zhao, Dong Chen, and Baining Guo. Rodinhd: High-fidelity 3d avatar generation with diffusion models. In European Conference Computer Vision (ECCV), 2024a.
Bowen Zhang, Yiji Cheng, Jiaolong Yang, Chunyu Wang, Feng Zhao, Yansong Tang, Dong Chen, and Baining Guo. Gaussiancube: A structured and explicit radiance representation for 3d gener- ative modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2024b.
Kai Zhang. Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. In European Conference on Computer Vision (ECCV), 2024c.
Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high-quality 3d assets. ACM Transactions on Graphics (TOG), 2024d.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
16

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
Reproducibility We provide comprchensive implementation details in this section to facili- tate the reproducibility of our work. Our code and models are publicly available at https: / //chenguolin wwrpoliinnnntttte www..b..hhu...o. http////oooottto/osse/soee/////e
Training For Gaussian splat grid reconstruction. we train lightweight 12 12-layer and B-head Transformer encoder (Vaswani et al., 2017) with 512 attention dimensions and a patch size of 8, whose parameter size is only 42M and 9. 9% of previous methods (Tang al., 2024; Xu et al., 2024c; Zhang et al., 2024c). Smin and 8max are set to 5e-4 and 2e-2 respectively to repre- sent fine-grain details. The input views Vn =4 are evenly distributed and rendering views V =8 include 4 other random viewpoints. All weighting terms are set to 1. The probability of applying the rendering loss starts at and is set to later for training efficiency. All experiments are conducted at the 256 256 resolution in this work. Training batch size for reconstruction and auto-encoding is 64 in total across up to 16 A100 GPUs with gradient accumulation and the peak learning rate of 4e-4. For diffusion models, the batch size and peak learning rate are 128 and 1e-4 respec- tively. AdamW optimizer (Loshchilov & Hutter, 2018) 8) with weight decay and cosine learning rate scheduler (Loshchilov & Hutter, 2016) with linear warm-up are adopted for parameter optimization.
Inference For diffusion- models (SD1.5 (Rombach et al., 2022), SDXL (Podell et al., 2024), PixArt-a (Chen et al., 2024c) and PixArt-E (Chen et al., 2024b)), the DPM-Solver++ (Lu et al., 2022a;b) ODE solver with 20 inference steps is adopted following Chen et al. (2024c;b). The flow- based model, i.c., SD3 (Esser et al., 2024) uses the original flow matching Euler ODE solver (Lipman et al., 2023) with 28 steps. consistent with its original configuration. In the text-conditioned gener- ation, classifier free guidance (Ho & Salimans, 2021) scales for each model are the same with their default values: 7.5 for SDI.5, 5 for SDXL., 4.5 for PixArt-a and Pix and 7 for SD3. In the image-conditioned generation, all models are fine-tund fie-ttued to predict velocity (Salimans & Ho, 2022; Shi et al., 2023), and their guidance scales are all set to 2. Runtime for SPLAT to generate single 3D object on an A100 GPU is only about 1~2 seconds with half precision.
Cost Notably, with 2D generative priors, DIFFSPLAT only takes about 3 days on 8 A100 GPUs to generate decent results with fp1 6 mixed precision, which is much more training-efficient than pre- vious 3D generative models, such as DMV3D (Xu et al., 2024d) (128 A100 x 7 days), CLAY (Zhang et al., 2024d) (256 A800 x 15 days) and 3DTopia-XL (Hong et al., 2024a) (128 A100 x 14 days).
B MORE VISUALIZATION RESULTS
Input Image
A sculpture
mask
A headwear
flat
Input Image
A pencil box
A small purse
A ball
flat badge
Figure 8: Controllable Generation with Multi-modal Conditions. DIFFSPLAT can effectively utilize both text and image conditions for single-view reconstruction with text understanding.
17

---

Published as a conference paper at ICLR 2025
Text Prompt
Generated 3DGS
A silver teapot on dining table
A candle burns beside an ancient, leather- bound book
A red cardinal on a snowy branch
A dog made out of salad
A pigeon reading a book
A beautiful rainbow fish
An ice cream sundae
Michelangelo style statue of dog reading news cellphone
Figure 9; More results of text-conditioned PPLA..
18

---

Published as a conference paper at ICLR 2025
Text Prompt
Generated 3DGS
A white seashell sandy beach
A silver spoon in bowl of soup
A bluebird perched on tree branch
A red rose in a crystal vase
A colorful parrot on a jungle tree
A blue butterfly on a pink flower
A white cotton t-shirt hanging on wooden hanger
A delicate china teacup resting on a lace doily
Figure 10: More results of text-conditioned DIFFSPLAT.
19

---

Published as a conference paper at ICLR 2025
Text Prompt
Generated 3DGS
A golden retriever with a blue bowtie
A vibrant sunflower with green leaves
A pair of worn-in blue jeans
A gold glittery carnival mask
A fuzzy pink flamingo lawn ornament
A fragrant pine Christmas wreath
A small porcelain white rabbit figurine
A silver mirror with ornate detailing
Figure 11: More results of text-conditioned DIFFSPLAT.
20

---

Published as a conference paper at ICLR 2025
Input Image
Generated 3DGS
Figure 12: More results of image-conditioned PLAT.PLAT.
21

---

Published as a conference paper at ICLR 2025
Input Image
Generated 3DGS
Figure 13: More results of image-conditioned DIFFSPLAT.
22

---

Published as a conference paper at ICLR 2025
Input Image
Generated 3DGS
Figure 14: More results of image-conditioned PLAT.
23


Reviews:
[{"review_id": "SSuI5hDXIC", "reviewer": "Reviewer_Xhe6", "summary": "This paper presents DiffGS, a new method to directly generate 3D Gaussians in a feed-forward manner. DiffGS is fine-tuned from image diffusion models with the 3D Gaussian latent. The experiments on both text and image condition 3D generation show that DiffGS can quickly generate reasonable 3D Gaussians with high-quality rendering quality.", "questions": "I think some video demos are recommended for a 3D generation paper. Video demos are important to visually understand the 3D consistency and some geometric artifacts.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. DiffGS encodes the 3D Gaussian Splatting to latents and trains a diffusion model to generate the latents directly, which is efficient compared with the optimization-based methods. The 3D Gaussian Splatting also offers better rendering quality.\n2. The design of fine-tuning from the image diffusion model is reasonable and effective.\n3. The authors provide various experiments, including different conditions (text, image normal, and edge).\n4. The presentation is clear and easy to follow.", "weaknesses": "1. The overall idea is not very interesting. DiffGS is another latent diffusion by changing the image vae to a 3D Gaussian vae. Only substituting the reconstruction module in LGM to a diffusion model lacks novelty. In fact, the 3D native diffusion model [1] has proved that we can generate very high-quality 3D objects by jointly representing the 3D shapes with geometry latents and PBR materials while the generated 3D Gaussians in DiffGS can only provide good renderings. The advantage of Gaussian Splatting representation is that it may contain higher-resolution information compared with implicit tokens in previous work. I am wondering if the authors could prove that the Gaussian-based latent diffusion can achieve more detailed generation. Besides, the poor geometry of 3D Gaussians is also a problem for applying this representation to 3D generation given that the 3D native diffusion has achieved such great progress.\n2. It would be better if the authors could provide videos of the generated objects. It is hard to evaluate some 3D artifacts just by the current metrics (e.g., PSNR). For example, the Gaussians often show good rendering results on some views while suffering from obvious 3D inconsistency.  Some noisy Gaussians and artifacts can be easily observed in videos.\n3. Applying a rendering loss combined with the diffusion loss is a common trick.\n\n[1] CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tyGTUkkl71", "reviewer": "Reviewer_i4mY", "summary": "The authors propose a new diffusion-based multi-view generative models using 3D Gaussian Splatting rather than directly on image space. They consist of latent space which compress the gaussian representation to apply Latent Diffusion Model (LDM). Both diffusion loss and rendering loss are combined for better multi-view consistent plausible outputs.", "questions": "1. Please compare the proposed methods with more 3D multi-view generative models such as Att3D [1] or Latte3D [2].\n2. Please report rendering time of both single 3D model and multi-view 2D images with the rendered 3D model.\n3. It would be good if the authors compare the 3D model quality of the proposed method with per-single 3D optimization models like GaussianDreamer [3].\n\n[1] Lorraine, Jonathan, et al. \"Att3d: Amortized text-to-3d object synthesis.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Xie, Kevin, et al. \"Latte3d: Large-scale amortized text-to-enhanced3d synthesis.\" European Conference on Computer Vision. 2024.\n\n[3] Yi, Taoran, et al. \"Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose a way to combine 3DGS with latent diffusion model for generating better 3D models.\n2. The proposed 3D generation pipeline seems to be both plausible and novel in that it parametrizes Gaussians and adopts specific VAE architectures to apply LDM on multiple Gaussian sets for 3DGS.\n3. The authors well demonstrate both quantitative and qualitative results compared to competitive methods in image-to-3D and text-to-3D tasks.\n4. Ablation study shows how each component contributes to the performance including omitting 3D rendering loss and 2D diffusion loss.", "weaknesses": "1. The contribution of the paper is on synthesizing conditional 3D models (text, single image) without per-object optimization using SDS such as DreamFusion, MVDream, and Wonder3D. I think the paper lacks some recent competitive methods including ATT3D [1] and Latte3D [2] which also aims to synthesize 3D models without per-object optimization.\n\n2. There is no comparison for synthesizing time during inference. As the required time to get a single object is important, it can be also a weakness for the paper.\n\n[1] Lorraine, Jonathan, et al. \"Att3d: Amortized text-to-3d object synthesis.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Xie, Kevin, et al. \"Latte3d: Large-scale amortized text-to-enhanced3d synthesis.\" European Conference on Computer Vision. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sqdi3s32IY", "reviewer": "Reviewer_2Rvk", "summary": "This paper, DIFFGS, proposes a new 3D generative framework designed to improve 3D content generation, addressing challenges like limited high-quality datasets and inconsistencies from 2D multi-view generation. Unlike other models, DIFFGS harnesses large-scale text-to-image diffusion models, leveraging 2D priors while ensuring 3D consistency. It introduces a lightweight reconstruction model to create multi-view Gaussian grids, enabling scalable data curation. Alongside traditional diffusion loss, DIFFGS uses a 3D rendering loss to enhance 3D coherence across views. This framework is compatible with image diffusion models, allowing seamless integration of image-generation techniques into 3D applications. Experiments show DIFFGS excels in text- and image-conditioned tasks, with ablations confirming the importance of each component. Code and models will be publicly available.", "questions": "1. When fine-tuning the 2D VAEs, have you fixed some parameters or you fine-tune the whole VAE model? Since if fine-tuning the whole 2D VAE, how to stay connected with the pre-trained diffusion latent space?\n2. Does your stage-1 reconstruction model for data curation itself serve as a LGM/GRM-like model, since it consumes multi-view RGB-N-XYZ renderings into the pixel-aligned Gaussians.\n3. Comparison with amortised SDS-based model (e.g., LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis). They also provide a way to leverage pre-trained 2D diffusion models for 3D generation.\n4. One disadvantage of the proposed solution is a lack of 3D latent space, and the generation of interior regions of the 3D object. Is these issues solvable by augmenting the proposed framework?\n5. Compared to training diffusion on the 3D latent space direction (e.g., 3DTopia and LN3Diff), what are the limitations of the proposed method?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper presents a novel paradigm of 3D generation, leveraging strong 2D pretrained T2I models. The proposed method first encodes the given object into multi-view splatter images, fine-tunes the 2D VAEs to the 12xHxW domain, then train the multi-view splatter images diffusion models with both diffusion loss and rendering loss. Both view-concat  and spatial-concat versions perform reasonably well.\n2. The comparison experiment is sound, which included the most recent methods (LN3Diff, 3DTopia-XL) as the strong baselines, which greatly demonstrates the effectiveness of the proposed method.\n3. The proposed method can be well integrated into existing 2D diffusion frameworks, e.g., introducing ControlNet into the 3D generation domain.\n4. Overall, I like this paper very much and strongly lean to accept it as the ICLR main conference paper.", "weaknesses": "Whether representing the given 3D object with multi-view splatter images leads to inconsistency, like in LGM? Since the Gaussians have overlap between different regions, and some “ghosting artifacts” are very common especially in very thin objects (such as axes).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nfERgMebMz", "reviewer": "Reviewer_L5NX", "summary": "This paper proposes DiffGS, a novel 3D generative framework that leverages large-scale text-to-image diffusion models to directly generate 3D Gaussian splats.  It first introduces a lightweight, generalizable 3DGS network that regresses per-pixel Gaussians. Then, it fine-tunes existing VAEs to encode these per-pixel Gaussians into Gaussian latents. Finally, the large-scale text-to-image diffusion models are fine-tuned to predict the noise injected in Gaussian latents. Extensive experiments show that the proposed method achieves high-quality novel view synthesis results compared to baselines.", "questions": "See weakness", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed method uses fewer parameters and achieves better novel view quality compared to baselines.\n\n* A framework that generates 3DGS from noise is interesting.\n\n* The paper is structured well, making it easy to understand and follow.", "weaknesses": "I think there is no critical weakness but there are some unclear components in this study:\n\n1. The authors claim that DiffGS leverages large 2D diffusion priors and maintains 3D consistency through the 3D rendering loss. However, the reported results focus only on novel view synthesis and text similarity scores. It would be better to include evaluations of 3D consistency, similar to the COLMAP-based evaluation used in SyncDreamer [1] or the optical flow-based evaluation used in HarmonyView [2].\n\n2. At the inference stage, how is the input camera pose obtained? Is it derived from the training dataset?\n\n3. Although the Gaussian latents are already aligned in the 3D representation, is there still a need for Plücker ray embeddings? What would happen if ray embeddings were not used?\n\n[1] Liu et al., Syncdreamer: Generating multiview-consistent images from a single-view image, ICLR 2024.\n\n[2] Woo et al., Harmonyview: Harmonizing consistency and diversity in one-image-to-3d, CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)