DiffGS: Functional Gaussian Splatting Diffusion
Authors: Junsheng Zhou
Weiqi Zhang
Yu-Shen Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Gaussian splatting
3D generation
Latent diffusion models
Primary area: machine_vision
TL;DR: Introducing a powerful 3D generative model that generates Gaussian primitives in arbitrary numbers by functionally disentangling Gaussian Splatting.

Abstract: 3D Gaussian Splatting (3DGS) has shown convincing performance in rendering speed and fidelity, yet the generation of Gaussian Splatting remains a challenge due to its discreteness and unstructured nature. In this work, we propose DiffGS, a general Gaussian generator based on latent diffusion models. DiffGS is a powerful and efficient 3D generative model which is capable of generating Gaussian primitives at arbitrary numbers for high-fidelity rendering with rasterization. The key insight is to represent Gaussian Splatting in a disentangled manner via three novel functions to model Gaussian probabilities, colors and transforms. Through the novel disentanglement of 3DGS, we represent the discrete and unstructured 3DGS with continuous Gaussian Splatting functions, where we then train a latent diffusion model with the target of generating these Gaussian Splatting functions both unconditionally and conditionally. Meanwhile, we introduce a discretization algorithm to extract Gaussians at arbitrary numbers from the generated functions via octree-guided sampling and optimization. We explore DiffGS for various tasks, including unconditional generation, conditional generation from text, image, and partial 3DGS, as well as Point-to-Gaussian generation. We believe that DiffGS provides a new direction for flexibly modeling and generating Gaussian Splatting. Project page: https://junshengzhou.github.io/DiffGS.

DiffGS: Functional Gaussian Splatting Diffusion
Junsheng Zhou"
Weiqi Zhang"
Yu-Shen Liu
School of Software, Tsinghua University, Beijing, China
(zhou-js24, ( axq23)mmall......mmmmmmmmmmmm Smails. tsinghua edu. Iiuyushen@tsinghua. edu.
Abstract
3D Gaussian Splatting (3DGS) has shown convincing performance in rendering speed and fidelity, yet the generation of Gaussian Splatting remains a challenge due its discreteness and unstructured nature. In this work, we propose DifGGS, a general Gaussian generator based on latent diffusion models. DiffGS is pow- erful and efficient 3D generative model which is capable of generating Gaussian primitives at arbitrary numbers for high-fidelity rendering with rasterization. The key insight represent Gaussian Splatting in a disentangled manner via three novel functions to model Gaussian probabilities, colors and transforms. Through the novel disentanglement of 3DGS, we represent the discrete and unstructured 3DGS with continuous Gaussian Splatting functions, where then train a latent diffusion model with the target of generating these Gaussian Splatting functions both unconditionally and conditionally. Meanwhile, introduce discretization algorithm to extract Gaussians at arbitrary numbers from the generated functions via setrre-guided sampling and optimization. We explore DifiGS for various tasks, including unconditional generation, conditional generation from text, image, and partial 3DGS, well as Point-to-Gaussian generation. We believe that DiffGS provides new direction for flexibly modeling and generating Gaussian Splatting. Project page: https: :/ junshengzhou. github. io/DiffCS.
Introduction
3D content creation is a vital task computer graphics and 3D computer vision, which shows great potential in real-world applications such virtual reality, game design, film production, and robotics. Previous 3D generative models usually take Neural Radiance Field (NeRF) [41, 2, 62] as the representation. However, the volumetric rendering for NeRF requires considerable computational cost, leading to sluggish rendering speeds and significant memory burden. Recent advances of 3D Gaussian Splatting (3DGS) [28, 68, 23] have demonstrated its potential to serve as the next- generation 3D representation by enabling both real-time rendering and high-fidelity appearance modeling. Designing 3D generative models for 3DGS provides scheme for real-time interaction with 3D creations.
The core challenge in generative 3DGS modeling lies in its discreteness and unstructured nature, which prevents the well-studied frameworks in structural image/voxel/videe generation from trans- ferring directly generate 3DGS. Coneurrent works [72, 19] alternatively transport Gaussians into structural voxel grids with volume generation models [11] for generating Gaussians. However, these methods lead 1) abundant computational cost for high-resolution voxels, and 2) limited number of generated Gaussians constrained by the voxel resolutions. Certain voxelization schemes [19] also introduce information loss, making it challenging maintain high-quality Gaussian reconstructions.
"Equal contribution. Yu-Shen Liu the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Gaussion Probability
Neural Functioning
Function (GauPF)
Gaussian Color
Function (GauCF)
Gaussian Tramsform
Goussian Extraction
Function (GauTT)
VAF+LDM
Multi-View Images
3DGS
Gaussian Splatting Functions
Figure 1: The illustration of DiffGS. We fit 3DGS from multi-view images and then disentangle into three Gaussian Splatting Functions. We train Gaussian VAE with a latent diffusion model for generating these functions, followed by a Gaussian extraction algorithm to obtain the final generated Gaussians.
To address these challenges, we present DifGGS, a novel diffusion-based generative model for general 3D Gaussian Splatting, which is capable of efficiently generating high-quality Gaussian primitives at arbitrary numbers. The key insight of DiffGS is to represent Gaussian Splatting in a disentangled manner via three novel functions: Gaussian Probability Function (GauPF), Gaussian Color Function (GauCF) and Gaussian Transform Function (GauTF). Especially, GauPF indicates the geometry of 3DGS by modeling the probabilities of each sampled 3D location be a Gaussian location. GauCF and GauTF predict the Gaussian attributes of appearances and transformations given 3D location as input, respectively. Through the novel disentanglement of 3DGS, we represent the discrete and unstructured 3DGS with three continuous Gaussian Splatting functions.
With the disentangled and powerful representation, the next step is design a generative model with the target of generating these Gaussian Splatting functions. We propose Gaussian VAE model for creating compressed representation for the Gaussian Splatting functions. The Gaussian VAE learns a regularized latent space which maps the Gaussians Splatting functions of each shape into one latent vector. A latent diffusion model (LDM) is simultancously trained the latent space for generating novel 3DGS shapes. With the powerful LDM, we explore DiffGS to generate diverse 3DGS both conditionally and unconditionally. Finally, we introduce a discretization algorithm to extract Gaussians at arbitrary numbers from the generated functions via octree octree-guided sampling and optimization. The key idea is to first extract 3D Gaussian geometry from GauPF by sampling 3D locations at the 3D spaces with the highest Gaussian probabilities, and then predict the Gaussian attributes with GauCF and GauTF. We illustrate the overview of DiffGS in Fig.
We systematically summarize the superiority of DiffGS in terms of: I) Efficiency, we design DifIGS based on Gaussian Splatting and Latent Diffusion Models, which shows significant efficiency in model training, inference and shape rendering. 2).3) Generality and quality, we generate native 3DGS without processes like voxelization, leading to unimpaired quality and generality in applying to downstream 3DGS applications. 4) Scalability, we scalably generate Gaussian primitives at arbitrary numbers. We conduct comprehensive experiments on both synthetic ShapeNet dataset and real- world DeepFashion3D dataset, which demonstrate our non-trivial improvements over the state-of-the-art methods. summary, our contributions are given as follows.
We propose DiffGS, a novel diffusion-basse generative model for general 3D Gaussian Splatting, which is capable of efficiently generating high-quality Gaussian primitives at arbitrary numbers.
We introduce novel schema to represent Gaussian Splatting disentangled manner via three functions to model Gaussian probabilities, Gaussian colors and Gaussian transforms, respectively. We simultaneously propose a discretization algorithm to extract Gaussians from these functions via octree-guided sampling and optimization.
DiffGS achieves remarkable performances under various tasks including unconditional generation, conditional generation from text, image, and partial 3DGS, as well as Point-to- Gaussian generation.

---

2 Related Work
2.1 Rendering-Guided 3D Representation
Neural implicit representations which learn signed [46, 79, 37] (unsigned [77, 78, 75]) distance functions occupancy functions [39] have largely advanced the field of 3D generation [84, 76, 66, 36], reconstruction [80, 26, 24, 25, 45] and perception [82, 83, 81, 32, 31J. Remarkable progress have been achieved in the field of novel view synthesis (NVS) [41, 47, 62, 2, 43], with the proposal of Neural Radiance Field (NeRF) [41]. NeRF implicitly represents scene appearance and geometries using MLP-based neural networks, optimized through volume rendering to achieve outstanding NVS quality. Some subsequent variants [1, 15, 49] have shown promising performance by advancing NeRF in terms of rendering quality, scalability and view-consistency. Additionally, more recent methods [43, 7, 14, 64] explore the training and rendering efficiency of NeRF by introducing feature- grids based 3D representations. Instant-NGP [43] highly accelerates NeRF learning by introducing multi-resolltion feature grids based on hash table with fully-fused CUDA kernel implementations. However, the NeRF representations which require expensive neural network inferences during volume rendering, still struggles in the applications where real-time rendering is required.
Recently, the emergence of 3D Gaussian Splatting (3DGS) [28, 59, 30, 68, 71, 18, 74] has showcased impressive real-time results in novel view synthesis (NVS). 3DGS [28] has led to revolutions in the NVS field by demonstrating superior performances in multiple domains. However, the generation of Gaussian Splatting remains a challenge due to its discreteness and unstructured nature. In this paper, we introduce novel schema represent the discrete and unstructured 3DGS with three continuous Gaussian Splatting Functions, thus ingeniously tackle the challenge by designing generative models for the functions.
2.2 3D Generative Models
The field of creating 3D contents with generative models has emerged as a particularly captivating research direction. A series of studies [48, 33, 65, 52, 40, 36, 9, 56, 69, 60] focus on optimization- based frameworks based on Score Distillation Sampling (SDS), which achieve convineing generation performances by distilling 3D geometry and appearance of the radiance fields with pretrained 2D diffusion models [44, 21] as the prior. However, these studies entail significant computational costs due to time-consuming per-scene optimization. Going beyond optimization- based 3D generation, recent methods [42, 61, 63, 27] explore 3D generative methods based on diffusion models to directly learn priors from 3D datasets for generative radiance fields modeling, which typically represent radiance fields as structural triplanes [63, 55, 17] or voxels [61, 42, 11]. DiffRF [42] leverage a voxel based NeRF representation with 3D U-Nets as the backbone to train diffusion model.
With the recent advances in 3DGS [28], designing a powerful 3D generative model for generating 3DGS is expected to be a popular research topic. This also brings significant challenges due to the discreteness and unstructured nature of 3DGS, which prevents the well-studied frameworks in structural image/voxel/video generation from transferring to directly generate 3DGS. A series of studies [70, 86, 22, 73] follow the schema of image-based reconstruction without generative modeling, which lack the ability to generate diverse shapes. Concurrent studies GaussianCube [72] and GVGEN [19] follow the voxel-based representations to transport Gaussians into structural voxel grids with volume generation models for generating Gaussians. However, these methods come with several drawbacks, including abundant computational costs for high-resolution voxels and a restricted number of generated Gaussians constrained by voxel resolutions. Some voxelization strategies [19] may introduce information loss, leading to difficulties in preserving high-quality Gaussian reconstructions. In contrast, our proposed DifIGS explores a new perspective to directly represent the discrete and unstructured 3DGS with three continuous Gaussian Splatting Functions. Though the insight, design latent diffusion model for efficiently generating high-quality Gaussian primitives by learning to generate the Gaussian Splatting Functions. DiffGS generates general Gaussians at arbitrary numbers with specially designed octree-based extraction algorithm.
3 Method
We introduce DiffGS, a novel diffusion-based generative model for general 3D Gaussian Splatting, which is capable of efficiently generating high-quality Gaussian primitives at arbitrary numbers.
3

---

(a) Training
Color Probabilliyy
CanPF
GS Encoder
Triplane Decoder
GauCT
C
Gaussian VAE
Transform
Text, Image, Partial os...
Gan'T
Generator
Z
Slep K
Steg
Zo
(0, 1)
Add Noise
Add Nosic
Gauasiun LDM
(b) Gaussian Generation
GauCF
Triplane Decoder 4
GauPF
LDM
N (0.
Geometry
GauTF
GS Locations
Extraction
The overview of DiffGS. (a) We disentangle the fitted 3DGS into three Gaussian Splatting Functions to model the Gaussian probability, colors and transforms, respectively. We then train Gaussian VAE with conditional latent diffusion model for generating these functions. (b) During generation, we first extract Gaussian geometry from the generated GauPF, followed by the GauCF and GauTF to obtain the Gaussian attributes.
Figure 2:
The overview of DifGGS is shown Fig. 2. We first preview Gaussian Splatting in Sec. 3.1 and present the novel functional schema for representing Gaussian Splatting with three disentangled Gaussian Splatting Functions in Sec. 3.2. We then introduce the Gaussian Variational Auto-encoder and the Latent Diffusion Model for compressing and generative modeling Gaussian Splatting Functions, as shown Sec. 3.3. A novel discretization algorithm is further developed in Sec. 3.4 to extract Gaussians at arbitrary numbers from the generated functions via octree-guided sampling and optimization.
3.1 Preview Gaussian Splatting
3D Gaussian Splatting (3DGS) [28] represents a 3D shape or scene set of Gaussians with attributes to model the geometries and view-dependent appearances. For 3DGS G [= containing N Gaussians, the geometry of i-th Gaussian is explicitly parameterized via 3D covariance matrix Σg and its center o, E R3, fomulated as:
(x(x) exp (-1 (=;) (o-:o:)T-- -- (o));,)
where the covariance matrix Σ, r.s;s? r! factorized into rotation matrix r; R and scale matrix s, IR. The appearance of the Gaussian gs is controlled by an opacity value 0, E R and color value 6 E T Note that the color is represented as series sphere harmonies coefficients in practice of 3DGS, yet still keep its definition as three-dimension color c, in our paper for clear understanding on our method. To this end, the 3DGS defined as {g. = (or, Tú, St, 0,, c e g 1 where K is dimension of the combined attributes in each Gaussian.
3.2 Functional Gaussian Splatting Representation
The key challenge in generative 3DGS modeling lies in its discreteness and unstructured nature, which prevents the well-studied generative frameworks from transferring to directly generate 3DGS. address this challenge by introducing to represent Gaussian Splatting in a disentangled manner via three novel functions: Gaussian Probability Function (GauPF), Gaussian Color Function (GauCF) and Gaussian Transform Function (GauTF), respectively. Through the novel disentangling of 3DGS, represent the discrete and unstructured 3DGS with three continuous Gaussian Splatting Functions.

---

Gaussian Probability Function. Gaussian Probability Function (GauPF) indicates the geometry of 3DGS by modeling the probabilities of each sampled 3D location to be Gaussian location. Given a set of 3D query location Q = { E R J==1 sampled in 3D space around fitted 3DGS G= {g: E R the GauPF of G predicts the probabilities of queries {gz ) to be a Gaussian location in G, fomulated as:
1==1+
Pj GauPF(q,) E [0.
(2)
The idea of Gaussian probability modeling comes from the observation that the further a 3D location Oj is from all Gaussians, the lower the probability that any Gaussian occupies the space at yj- Therefore the ground truth Gaussian probability of gj is defined as:
GauPF(g)) =A(( min |واا orll2)),
[E[L.N]
where min 10 a indicates the distance from qj the nearest Gaussian center in {or <E[1.N]
is a truncation function which filters the extremely large values and is a continuous function which maps the query-to-Gaussian distances to probabilities the range [0,1].
A learned GauPF implicitly models the locations of 3D Gaussian centers, which is the key factor for generating high-quality 3DGS. The extraction of 3DGS centers from GauPF is then achieved with our designed Gaussian extraction algorithm which will be introduced in Sec. 3.4.
Gaussian Color and Transform Modeling, Gaussian Color Function (GauCF) and Gaussian Transform Function (Gau'lF) predict the Gaussian attributes of appearances and transformations from Gaussian geometries. Specifically, given the center a, of Gaussian yi in G' input, GauCF predicts the color attribute o and GauTF predicts the rotation Tú, scale 8, and opacity o,, formulated as:
{c} = GauCF(a;); (ri, 5;, J GauTF(or))
Note that GauCF and GauTF mainly focus predicting the Gaussian colors and transforms from 3D Gaussian centers. This is different from the GauPF which models the probabilities of query samples in the 3D space. The reason is that GauPF focuses exploring the geometry of 3DGS from the 3D space, while GauCF and GauTF learn to predict the Gaussian attributes from the known geometries.
Through the novel disentanglement of 3DGS, we represent the discrete and unstructured 3DGS with three continuous Gaussian Splatting Functions. The functional representation a general and flexible term for 3DGS which has no restrictions on the Gaussian numbers, densities, geometries, etc.
3.3 Gaussian Variational Auto-encoder and Latent Diffusson
With the disentangled and powerful representation, the next step is design a generative model with the target of generating these Gaussian Splatting Functions. We follow the common schema to design Gaussian Variational Auto-encoder (VAE) [29] with Latent Diffusion Model (LDM) [44] as the generative model. The detailed framework and the training pipeline of DiffGS are illustrated in Fig. ((a).
Gaussian VAE. The Gaussian VAE compresses the Gaussian Splatting Functions into a regularized latent space by mapping the Gaussian Splatting Functions of each 3DGS shape into latent vector, from which we can also recover the Gaussian Splatting Functions. Specifically, the Gaussian VAE consists of 1) a GS encoder den. learn representations from 3DGS and encodes each 3DGS into latent vector z, 2) a triplane decoder dide which decodes the latent z into a feature triplane, and 3) three neural predictors 2/ps, Pef and the which serve as the implementation of GauPF, GauCF and GauTF to predict Gaussian probabilities, colors and transforms, respectively.
Given fitted 3DGS G
input, the GS encoder den extracts global latent feature from G. which is then decoded into a feature triplane E R xxxxxxx XXx3 with the decoder Ødes formulated as:
{9:} y
z Den ( G)); I = (x)
(5)
The triplane consists of three orthogonal feature planes {!xY, 1xz,/yzi which are aligned to the
axex. For a 3D location qj. we obtain its corresponding feature Jn = inderp(t,, 93) from the triplane t by projecting 9 onto the orthogonal feature planes and concatenating the tri-linear interpolated
5

---

@gtimiiiing
Sampling
(4) Octree depth 1
(b) Octree depth
(c) Octree depth I.
(c) Proxy Caussian centers
(e) Final Caussean cectiers
Figure 3: Gaussian geometry extractions from generated GauPF. The yellow and green regions indicate the high probability area and the low probability area judged by GauPF. (a),(b) and (c) show the progressively octree build process depth 1,2 and L. (d) We sample proxy Gaussian centers from the octree at final depth L. (c) We optimize proxy centers to the exact geometry indicated GauPF.
features the three planes. We then predict the Gaussian probability, color and transform of q; with the neural Gaussian Splatting Function predictors as:
( oof((s: (4) = vor(fr)
(6)
The Gaussian VAE is trained with the target of accurately predicting Gaussian attributes and robustly regularizing the latent space. practice, the training objective is formulated as:
LVAE = (0,C,,,,00) (p.cr,,,,))) + (Dxz. (2<(=()))IP(3)))
The first loss indicates the Li loss between the predicted Gaussian attributes in Eq. (6) and the target ones defined by the ground truth Gaussian Splatting Functions in Eq. (2) and Eq. (4). The second term Eq. (7) is the KL-divergence loss with factor of 3, which constrains on the regularization of the Iearned latent space of z. Specifically, we define the inferred posterior of as the distribution 2s (z)G), which is regularized to align with the Gaussian distribution prior P()) = NO, T), where 7 is the standard deviation.
Gaussian LDM. With the trained Gaussian VAE in place, into a compact ID latent vector 2. We then train a latent diffusion model (LDM) [44] efficiently on the latent space. A diffusion model is trained to generate samples from target distribution by reversing a process that incrementally introduces noise. We define {zo, ž1, =x as the forward process ((=o.) which gradually transforms real data zo into Gaussian noise (2T) by adding noises. The backward process μu( <o:K)) leverages neural generator p to denoise into real data sample.
now able encode any 3DGS
To achieve controllable generation of 3DGS, introduce conditioning mechanism [54] into the diffusion process with cross-attention. Given an input condition y (c.g. text, image, partial 3DGS), leverage custom encoder d to project y into the condition embedding 5(y). The embedding is then fused into the generator with cross attention modules. Following DDPM, we simply adopt the optimizing objective to train the generator for predicting noises c,, formulated as:
LIDM = E,,,,t [|ee en (z,, 8(g), 0)1
(8)
where t is a time step and is noise latent sampled from the Gaussian distribution (o 1). respectively. We adopt the well-studied architecture DALLE-2 [53] as the LDM implementation.
3.4 Gaussian Extraction Algorithm
The final step for the generation process of DifiGS is to extract 3DGS from the generated Gaussian Splatting Functions, similar to the effect of Marching Cubes algorithm [34] which extracts meshes from Signed Distance Functions. The key factor is to extract the geometries of 3DGS, i.e., Gaussian locations and the appearances 3DGS, i.e., colors and transforms. The full generation pipeline is shown in Fig. 2(b).
Octree- Guided Geometry Sampling. The locations of 3D Gaussian centers indicate geometry of the represented 3DGS. We aim to design a discretization algorithm obtain the discrete 3D locations from the learned continuous Gaussian Probability Function parameterized with the neural network dip which models the probability of each query sampled in the 3D space to be 3D Gaussian

---

Table I: Comparisons of unconditional generation under ShapeNet [6] dataset.
Airplane
Chair
Method
FID-50K
KID-50K (%o) +
FID-50K
KID-50K (%ce) +
GET3D [16] DifTTF [4]
59.51
2.414
110.8 47.03
9.173 3,436
93.02
6.708
Ours
35.28
2.148
GET3D
DiffTF
Ours
Figure 4: Visual comparisons with state-of-the-arts on unconditional generation of ShapeNet Chairs.
location. To achieve this, we design octree-based sampling and optimization algorithm which generates accurate center locations of 3D Gaussians at arbitrary numbers.
We show the 2D illustration of the algorithm in Fig. 3. Assume that the 3D space is divided into the high probability area (the yellow region) and the low probability area (the green region) by the generated GauPF. We aim to extract the geometry the locations with high probabilities. A naive implementation is to densely sample queries the 3D space and keep the ones with large probabilities as outputs. However, it will lead high computational cost for inferencing and the discrete sampling also struggles to accurately reach the locations with largest probabilities in the continuous GauPF. We get inspiration from octree [38, 67] design progressive strategy which only explores the 3D regions with large probabilities in current octree depth for further subdivision in the next octree depth. After layers of octree subdivision, we reach the local regions with largest probabilities, from where we uniformly sample N 3D points as the proxy points {p: Ja representing coarse locations of Gaussian centers.
Optimizing Geometry with GauPE. To further refine the proxy points to the exact locations of Gaussian centers with largest probabilities in GauPF, we propose to further optimize the proxy points with the supervision from learned GauPF pps- Specifically, we set the position of proxy points
{p: = {px, Piiis pz; } to be learnable and optimize them to reach the positions L.V with largest probabilities of upf. The optimization target is formulated
{
LGeo
Σ (pr(
Note that set N to arbitrary numbers, enabling DiffGS generate 3DGS with limits the density and resolution.
Extracting Gaussian Attributes. We now obtain the estimated geometry indicating the pre- dicted Gaussian centers {or).v We then extract the appearances and transforms from the generated triplane f, Gaussian Color Function def and Gaussian Transform Function the as

---

"a chair with brown handles and grey back."
"a grey chair two type legs."
(b)
Figure 5: Visualization of conditional 3DGS generation results ShapeNet. (a) Text conditional generation. (b) Image conditional generation. (c) Gaussian Splatting completion.
{&} (interrppt d.)) and (f,, 6.} (interrepp d,)). Finally, the general 3DGS is now generated as G { G, fu &, 6,
Experiment
4.1 Unconditional Generation
Dataset and Metrics. For unconditional generation of 3D Gaussian Splatting, conduct experi- ments under the airplane and chair classes of ShapeNet [6] dataset. Following previous works [42, 4J. we report two widely-used image generation metrics Fréchet Inception Distance (FID) [20] and Kernel Inception Distance (KID) [3] for evaluating the rendering quality of our proposed DifIGS and previous state-of-a the-art works. The metrics evaluated between 50K renderings of the generated shapes and 50K renderings of the ground turth ones, both at the resolution of 1024 1024.
Comparisons. We compare DifGGS with the state-of-the-art methods in terms of the rendering quality of generated shapes, including the GAN-based methods GET3D [16] and the diffusion-basee method DifTTF [4]. The quantitative comparison is shown in Tab. 1, where DiffGS achieves the best performance over all the baselines. We further show the visual comparison on the renderings some generated shapes in Fig. 4, where the GAN-based GET3D struggles in generating complex shapes and the generations of DiffTF is blurry with poor textures. contrast, DifiGS produces significantly more aisual-appealing and figh-iideiity generations in terms of rendering and geometry qualities.
4.2 Conditional Generation.
We explore the conditional generation ability of DiffGS given texts, images and partial 3DGS as the input conditions. All the experiments conducted under the chair class of ShapeNet [6] dataset with commonly used data splits in previous methods [10, 35].
mext//nage-oondiioonal Gaussian Splatting Generation. For introducing texts/images as the conditions for controllable Gaussian Splatting generation, we leverage the frozen text and image encoder from the pretrained CLIP [51] model as the implementation of custom text encoder "tewt and Timage for achieving text/image embeddings. We then train DiffGS with the conditional optimization objective in (8). We show the visualization of some text/image conditional generations produced by DiffGS in Fig. 5(a) and Fig. 5(b). The results show that DiffGS accurately recovers the semantics and geometries described in the text prompts and the images, demonstrating the powerful capability of DiffGS in generating high-fidelity 3DGS from text descriptions or vision signals.
Gaussian Splatting Completion. Additionally, we explore interesting task of Gaussian Splatting completion. To the best of our knowledge, we are the first to focus and introduce solutions for this task. Specifically, the Gaussian Splatting completion task is recover the complete 3DGS from

---

ShapeNet
Figure 6: Visualization of Point-to- -aaass-an fittings on Deepfashion3D and generations on ShapeNet.
partial 3DGS which contains large occlusions. In real-world applications, having only sparse views with limited viewpoint movement available for optimizing 3DGS often results in a partial 3DGS. Solving Gaussian Splatting completion task enables to infer the complete and dense 3DGS from the partial ones for improving the rendering quality at invisible viewpoints.
We introduce DifGGS with partial 3DGS the conditions for solving this task. Specifically, we simply leverage modified PointNet [50] as the custom encoder "spartial for partial 3DGS. Fig. 5(c) presents the visualization of Gaussian Splatting completion results produced by DifIGS. The results show that DifiGS capable of recovering complex geometries and detailed appearances from highly occluded 3DGS. Please refer to the appendix for implementation details on Gaussian Splatting completion.
4.3 Point-to-Gaussian Generation
We further introduce DiffGS for another challenging and vital task of --aas--a generation. This task aims to generate the Gaussian attributes given a 3D point cloud as input. The task serves as the bridge between the easily accessible point clouds and the powerful 3DGS representation which efficiently models high-quality 3D appearances.
Dataset and Implementation. We conduct experiments under the chair and airplane classes of ShapeNet and also the widely-used garment dataset DeepFashion3D [85]. The DeepFashion3D dataset is a real-captured 3D dataset containing complex textures. For implementing Point-to- Gaussian, we simply train the Gaussain VAE with the three-dimension point clouds as inputs, instead of the 3DGS with attributes. Please refer to the Appendix for more details on data preparation and implementation.
Performances. We provide the visualization of some -aa----i fitting and generation results in Fig. 6. We shown the fitting results for Deepfashion3D [85] dataset and the generation results for the test set of airplane and chair classes in ShapeNet [6]. DiffGS produces visual-appealing 3DGS generations given only 3D point cloud geometries as inputs. The results demonstrate that DifiGS can accurately predict Gaussian attributes for 3D) point clouds. We believe DiffGS provides new direction for 3DGS content generation by connecting 3DGS with point clouds.
4.4 Ablation Study
To evaluate some major designs and important hyper- parameters in DifiGS, we conduct ablation studies under the chair class of ShapeNet dataset. We report the performance in terms of PSNR, SSIM and LPIPS of the reconstructed 3DGS with Gaussian VAE.
Framework Designs. We first evaluate some major
Table 2: Ablations on framework designs.
designs of our framework Tab. We justify the effectiveness of introducing the truncation function A
Method
PSNR
SSIM
LPIPS
when modeling GauPF and report the results without as 'w/o truncation'. We then explore implementing the projection function either as T(x) = e- (as
w/o Trunction
29.39
0.9792
0.0173
Exponent
29.74
0.9765
0.0188
Wo Optimization
30.34
0.9875
0.0152
Ours
34.01
0.9879
0.0149
shown in 'Exponent') or a linear projection (as

---

shown in 'Ours'). We also show the results without the optimization process during Gaussian extraction 'w/o Optimization', which demonstrates the effectiveness of optimizing Gaussians to the exact locations.
Gaussian Numbers. One significant advantage of DiffGS lies in the ability of generating high-quality Gaussians at
Table 3: Ablations on Gaussian number.
Num
PSNR
SSIM
LPPPS
arbitrary numbers. To explore how the number of Gaus-
sians affects the rendering quality, we conduct ablations on the Gaussian numbers as shown in Fig. 3. The results demonstrate that denser Gaussians lead to better quality.
50K
28.61
0.9787
0.0251
100K
30.35
0.9838
0.0151
350K
34.01
0.9879
0.0149
5 Conclusion
In this paper, we introduce DiffGS for generative modeling of 3DGS. DiffGS disentangled represent 3DGS via three novel functions to model Gaussian probabilities, colors and transforms. We then train a latent diffusion model with the target of generating these functions both conditionally and unconditionally. DifGGS generates 3DGS with arbitrary numbers by an octree-guided extraction algorithm. The experimental results on various tasks demonstrate the superiority of DiffGS.
6
Acknowledgement
This work supported by National Key R&D Program of China (2022YFC3800600) the National Natural Science Foundation of China (62272263, 62072268), and in part by Tsinghua-Kuatn Kuaishou Institute of Future Media Data.
References
[1] Jonathan Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin -rruala, and Pratul P Srinivasan. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
5855-5864, 2021.
[2] Jonathan Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-
NeRF 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[3] Mikotaj Binkkwwki,, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying
mmd gans. arXiv preprint arXiv: 7.01401, 2018.
[4] Ziang Cao, Fangzhou Hong, Tong Wu, Liang Pan, and Ziwei Liu. Large-vocabulary 3D diffusion model with transformer. arXiv preprint arXiv:2309.07990 2023.
[5] Eric R Chan, Connor Z. Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Lconidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry- aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16123-16133 2022.
[6] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. ShapeNet: An information-rich 2015.
3D model repository. arXiv preprint 1rXiv:1512.0300
[7] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial radiance
fields. In European Conference on Computer Vision pages 333-350. Springer. 2022.
[8] Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pages 2416-2425, 2023.
[9] Zilong Chen, Feng Wang, and Huaping Liu. Text-to-3d using gaussian splatting. arXiv preprint arXiv:2309.1653 16585, 2023.
[10] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939-5948, 2019.
[11] Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander Schwing, and Liang- Yan Gui. Multimodal 3D shape completion, reconstruction, and generation. In Proceedings
SDFusion:
10

---

of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4456-4465. 2023.
[12] Gene Chou, Yuval Bahat, and Felix Heide. Diffusion-SDD SDF: Conditional generative modeling of signed distance functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2262-2272, 2023.
[13] Matt Deitke. Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142-1311 13153,, 2023.
[14] Sara Seadovich-Keill Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pages 5501-55100 2022.
[15] Qiancheng Fu, Qingshan Xu, Yew-Soon Ong, and Wenbing Tao. Geo-Neus: Geometry- consistent neural implicit surfaces learning for multi- reconstruction. Advances Neural Information Processing Systems (NeurIPS), 2022.
[16] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems, 35:31841-31854 2022.
[17] Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3DGen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.055: 053711, 2023.
[18] Liang Han, Junsheng Zhou, Yu-Shen Liu, and Zhizhong Han. Binocular aanoulla-guudee 3d gaussian splatting with view consistency for sparse view synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[19] Xianglong He. Junyi Chen, Sida Peng, Di Huang. Yangguang Li, Xiaoshui Huang, Chun Yuan, Wanli Ouyang, and Tong He. GVGEN: Text-to-3D generation with volumetric representation. arXiv preprin! arXiv:2403. /2957, 2024.
[20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 66000,68111200 2020.
[22] Yicong Hong, Kai Zhang, Jiuxiang Gu Sai Bi. Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3D. arXiv preprint Xr 1.04400, 2023.
[23] Binbin Huang. Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. arXiv preprint arXiv:2403. 2200..7888, 2024.
[24] Han Huang, Yulun Wu, Junsheng Zhou, Ge Gao. Ming Gu, and Yu-Shen Liu. NeuSurf: On- surface priors for neural surface reconstruction from sparse input views. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
[25] Chuan Jin, Tieru Wu, Yu-Shen Liu, and Junsheng Zhou. Music-udf: Learning multi-scale dy- namic grid representation for high-fidelity surface reconstruction from point clouds. Computers Graphics, page 104081, 2024.
[26] Chuan Jin, Tieru Wu, and Junsheng Zhou. Multi-grit representation with field regularization for self-supervised surface reconstruction from point clouds. Computers & Graphics, 2023.
[27] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv
preprint arXivv2305.0mm 02463, 2023.
[28] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 1-14 2023.
[29] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.611 14, 2013.
[30] Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Dngaussian: Optimizing -ierse-view 3d gaussian radiance fields with global-local depth normalization. arXiv
preprint arXiv:2403.06911 06912, 2024.
[31] Shujuan Li, Junsheng Zhou, Baorui Ma, Yu-Shen Liu, and Zhizhong Han. NeAF: Learning neural angle fields for point normal estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
[32] Shujuan Li, Junsheng Zhou, Baorui Ma. Yu-Shen Liu, and Zhizhong Han. Learning continuous implicit field with local distance indicator for arbitrary scale point cloud upsampling. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
11

---

[33] Chen- Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
[34] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3D surface
construction algorithm. ACM Siggraph Computer Graphies, 21(4))163-169, 1987.
[35] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2837-2845, 2021.
[36] Baorui Ma, Haoge Deng. Junsheng Zhou, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Geo- Dream: Disentangling 2D and geometric priors for high-fidelity and consistent 3D generation. arXiv preprint 1. 1.79711, 2023.
[37] Baorui Ma, Junsheng Zhou, Yu-Shen Liu, and Zhizhong Han. Towards better gradient con- sistency for neural signed distance functions via level set alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17734-17734, 2023. Donald Meagher. Geometric modeling using octree encoding. Computer graphics and image processing. 19(2):129-144 147-147. 1982.
[38]
[39] Lars Mescheder, Michael Oechsle. Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3D reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2019. [40] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen- Or. Latent-nert for shape-guided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12663-12613 12673, 2023.
[41] Mildenhall, PP Srinivasan, M. Tancik, JT Barron, R Ramamoorthi, and R Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision 2020.
[42] Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner. Diffrf: sonderinggguidees 3d radiance field diffusion. In Proceedings of the JEFE/VV Conference on Computer Vision and Pattern Recognition, pages 4328-4338, 2023. [43] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), - 15, 2022.
[44] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021. [45] Takeshi Noda, Chao Chen, Xinhai Liu Weiqi Zhang and, Yu-Shen Liu, and Zhizhong Han. Multipull: Detailing signed distance functions by pulling multi-level queries at multi-step. In Advances in Neural Information Processing Systems, 2024.
[46] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 165- 174,
2019.
[47] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Brutin-Brualla. Nerfies: Deformable neural radiance fields. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3865-5874, 2021. Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d 3d using 2d diffusion. arxiv preprint arXiv:2209.1 14988, 2022.
[48]
[49] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguee D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318-10322 2021.
[50] Charles R Qi, Hao Su. Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 2021.
pages
[52] Amit Raj. Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfrr Aberman, Michael Rubinstein, Jonathan Barron, et al. Dreambooth3d: Subject- driven text-to-3d generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2349-2359, 2023.
12

---

[53]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. 2022.
[54] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[55] J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20875-20888 2023.
[56] Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. arXiv preprint 23 168/8, 2023.
[57] Stanislaw Szymanowicz. Chrisitian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11200-10217, 2024.
[58] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, pages 1-18. Springer, 2025.
[59] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. /6059, 2023. [60] Junshu Tang. Tengfei Wang. Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from single image with diffusion prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22819-22829, 2023. [61] Zhicong Tang, Shuyang Gu, Chunyu Wang. Ting Zhang, Jianmin Bao, Dong Chen, and Baining Guo. Volumediffusiin: Flexible text-to-3d tdd generation with efficient volumetric encoder. arXiv preprint 22. 23..119991. 2023.
[62] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems, 34:2717 1127771-221111 17113121,, 2021.
[63] Tengfei Wang. Bo Zhang, Ting Zhang. Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen. Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4563-4573, 2023.
[64] Yiming Wang, Qin Han, Mare Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjic Liu. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 3295-3300 -306, 2023.
[65] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelit and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
[66] Xin Wen, Junsheng Zhou, Yu-Shen Liu, Hua Su, Zhen Dong, and Zhizhong Han. 3D shape reconstruction from 2D images with disentangled attribute flow. In Proceedings of the IEEF/CVF
Conference on Computer Vision and Pattern Recognition, pages 3803-3813, 2022.
[67] Jane Wilhelms and Allen Van Gelder. Octrees for faster isosurface generation. ACM Transac-
tions on Graphics (TOG). 11(3):201-22)
1992.
[68] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08522
2023.
[69] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffu- sion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11908820991 2023.
[70] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng. Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint arXiv:2403. /4621, 2024.
[71] Taoran Yi, Jiemin Fang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamerr: Fast generation from text to 3d gaussian splatting with point cloud priors. arXiv preprint arXiv:23 10.00229, 2023.
[72] Bowen Zhang, Yiji Cheng, Jiaolong Yang, Chunyu Wang, Feng Zhao, Yansong Tang, Dong Chen, and Baining Guo. Gaussiancube: Structuring gaussian splatting using optimal transport for 3d generative modeling. arXiv preprint arXiv:2403. 19655, 2024.
13

---

[73] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao. Kalyan Sunkavalli, and Zex- iang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. arXiv preprint arXiv:2404. /9702, 2024.
[74] Wenyuan Zhang, Yu-Shen Liu, and Zhizhong Han. Neural signed distance function inference through splatting 3d gaussians pulled on zero-level set. In Advances in Neural Information Processing Systems, 2024.
[75] Wenyuan Zhang, Kanle Shi, Yu-Shen Liu, and Zhizhong Han. Learning unsigned distance functions from multi-view images with volume rendering priors. European Conference on Computer Vision, 2024.
[76] Junsheng Zhou, Yu-Shen Liu, and Zhizhong Han. Zero-shot scene reconstruction from single images with deep prior assembly. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[77] Junsheng Zhou, Baorui Ma, Shujuan Li, Yu-Shen Liu, Yi Fang, and Zhizhong Han. Cap-udf: Learning unsigned distance functions progressively from raw point clouds with consistency- aware field optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[78] Junsheng Zhou, Baorui Ma, Shujuan Li, Yu-Shen Liu, and Zhizhong Han. Learning a more continuous zero level set in unsigned distance fields through level set projection. In Proceedings of the IEEF/CVF International Conference on Computer Vision, 2023.
[79]
Junsheng Zhou, Baorui Ma, and Yu-Shen Liu. Fast learning of signed distance functions from noisy point clouds via noise to noise mapping. IEEE transactions on pattern analysis and machine intelligence, 2024.
[80] Junsheng Zhou, Baorui Ma, Liu Yu-Shen, Fang Yi, and Han Zhizhong. Learning consistency- aware unsigned distance functions progressively from raw point clouds. In Advances in Neural Information Processing Systems (NeurlPS), 2022.
[81] Junsheng Zhou, Baorui Ma, Wenyuan Zhang, Yi Fang, Yu-Shen Liu, and Zhizhong Han. Differentiable registration of images and lidar point clouds with voxelpoint-to-pixll matching. In Advances Neural Information Processing Systems (NeurIPS), 2023.
[82] Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3D: Exploring Unified 3D Representation Scale. In International Conference Learning Representations (ICLR), 2024.
[83] Junsheng Zhou, Xin Wen, Baorui Ma, Yu-Shen Liu, Yue Gao, Yi Fang, and Zhizhong Han. 3d- oae: Occlusion auto-encoders for self-super learning on point clouds. IEEE International Conference Robotics and Automation (ICRA), 2024.
[84] Junsheng Zhou, Weiqi Zhang. Baorui Ma, Kanle Shi, Yu-Shen Liu, and Zhizhong Han. Udiff: Generating conditional unsigned distance fields with optimal wavelet diffusion. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[85] Heming Zhu, Yu Cao, Hang Jin, Weikai Chen, Dong Du, Zhangye Wang. Shuguang Cui, and Xiaoguang Han. Deep fashion3d: A dataset and benchmark for 3d garment reconstruction from single images. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Pari I 16, pages 512-530. Springer, 2020.
[86] Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprini 27X19:2312.09117, 2023.
14

---

Appendix
A More Experimental Details
A.1 Gaussian Splatting Data Preparing
DifiGS takes the fitted 3DGS input for learning generative modeling, To prepare the 3DGS dataset of ShapeNet, we uniformly render 100 views from the ground truth meshes with blender to first obtain the dense multi-view images for each 3D shape in the chair and airplane classes of ShapeNet dataset. After that, we leverage the vanilla 3D Gaussian Splatring [28] method for fitting the 3DGS for each shape with the rendered multi-view images.
For achieving more stable and regularized 3DGS data for better generative modeling, we design some strategies for better initialization and optimization of 3DGS. (1) Since we fit 3DGS from existing 3D datasets with known geometries, we can simply sample dense point clouds uniformly from the surfaces as the perfect initialization for 3DGS optimization, instead of initializing with COLMAP points. The sampled point number is set to 100K. (2) We observe that optimizing 3DGS freely may often lead to some extremly large Gaussians. This will lead to unstable training of the Gaussian VAE and latent diffusion models, further affecting the generative modeling results. Therefore, we clip the scales at a maximum size of 0.01 to avoid the abnormal Gaussians.
A.2 Gaussian Splatting Completion
We explore the task of Gaussian Splatting completion. Specifically, the Gaussian Splatting completion task is to recover the complete 3DGS from a partial 3DGS which contains large occlusions. In real-world applications, having only sparse views with limited viewpoint movement available for optimizing 3DGS often results in partial 3DGS. Solving Gaussian Splatting completion task enables us infer the complete and dense 3DGS from the partial ones for improving the rendering quality at invisible viewpoints.
Data preparation. We generate partial 3D Gaussian Splatting data from the complete datasets in a straightforward manner. First, we randomly divide each 3DGS into chunks. Then, we occlude 7 chunks, leaving the remaining chunk the partial 3DGS. This method allows us to prepare partial-complete 3DGS pairs for training and testing.
Implementation. To leverage DifiGS with partial 3DGS as the conditions for Gaussian Splatting completion, we leverage a modified PointNet [50] as the eustom encoder "partial for partial 3DGS, which projects the partial 3DGS with K channels into a global partial 3DGS embedding. The DiffGS for Gaussian Splatting completion is trained with the target of Eq.(8) by introducing partial 3DGS embeddings through the cross-altention module.
A.3 Point-to-Gaussian Generation
Data preparation. For the task of Point-to-Gaussian generation, we first prepare the training/testinng data as point cloud-3DGS pairs obtained through the Gaussian Splatting Fitting process described in Sec. A.I. Specifically, the point clouds are generated by densely sampling 100K points on the ground truth meshes, while the paired 3DGS are obtained by optimizing with multi-view images rendered around the ground truth meshes. The data preparation for the DeepFashion3D [85] dataset follows the same process for the ShapeNet [6] dataset.
DiffGS models for fitting the DeepFashion3D dataset, which contains only 563 garment instances. In contrast, we split the airplane and chair classes of the ShapeNet dataset into train/test sets to learn generalizable representations that enables DifIGS to predict novel appearances for unknown point cloud geometries. The results shown in Fig. 6 of the main paper include both the fitting results on the DeepFashion3D dataset and the generation results from the point clouds in the test set of the airplane and chair classes in the ShapeNet dataset.
Note that we train the Point-to-Gausee
we train the Gaussian VAE using three- dimensional point clouds as inputs instead of 3DGS with attributes. Specifically, we replace the GS encoder in the Gaussian VAE with a PointNet-basee network to learn representations from three- dimensional point clouds and recover Gaussian attributes from them. All architectures, optimization
Implementation. For implementing
15

---

MH
Input
Splatter Image
TriplanGGausian
LGM
Dream aussian
Ours
Figure 7: Qualitative comparison of image-to-3D generation.
targets, and Gaussian extraction processes remain the same in the Gaussian VAE, except for the encoder. Note that for the Point-to- Gaussian task, the Gaussian LDM is not trained, as we focus solely decoding and extracting Gaussians from the point cloud inputs.
Point-to-Gaussian
B More Comparisons and Ablations
We further conducted comprehensive experiments focusing on two critical tasks: text-to-3D gener- ation and single-view 3D generation. These tasks are central to demonstrating the flexibility and robustness of approach different application scenarios.
B.1 Image-to-3D Generation
We compare DifGSS with various SOTA methods on implicit generation or Gaussian Splatting genera- tion. The visual comparison of image-to-3D generation presented Fig. These results illustrate the superior visual quality and fidelity achieved by our method compared to the SOTA baseline methods including Splatterlmage [57], TriplaneGaussian [86], LGM [58] and DreamGaussian [59]. Our approach consistently produced more detailed and accurate generations, effectively capturing intricate textures and geometries that are often challenging for the compared optimization-ban based and multi-view based methods.
B.2 Text-to-3D Generation
We conduct evaluations under the difficult task of text-to-3D generation. We compare DiffGS with the SOTA data-driven and optimization-bant methods Shap-E [27], LGM [58] and DreamGaussian [59]. We present the visual comparisons in Fig. 8, where DiffGS achieves more visual-appealing results compared the baselines.
We also follow the common setting to conduct quantitative comparisons using the CLIP score, a metric that measures the semantic alignment between the generated 3D models and the input conditions. The results are presented in Tab. 4. According to Tab. 4, our method achieved higher CLIP scores than both DreamGaussian and LGM. This indicates that our approach more faithfully adheres to the condition inputs.
However, it is important to note that DifiGS is trained on the ShapeNet dataset, while some methods (e.g., LGM, TriplaneGaussian) are trained on the Objaverse [13] dataset. a result, the comparisons may not fully reflect the generation capabilities due to different data sources. The above comparisons are provided for reference, and we plan to conduct further training using the same dataset for more accurate comparison.
16

---

chair with cloth seat and white legs
Wooden chair with gray seat, small amreets and legs
HH
DmamG
Shap.P
LGM
Ours
Figure 8: Qualitative comparison of text-to-3D generation.
Table Comparisons of text consistencies.
DreamGaussian
Shap-E
LGM
Ours
CLIP Scores
29.08
32.76
32.52
33.42
B.3 Unconditional Generation
We compare DiffGS with DiffTF [4] the airplane class of the ShapeNet dataset. The visual comparison is shown in Fig. where our proposed DifIGS achieves more visually appealing results than DiffTF. DiffTF often produces shape generations with blurry textures, resulting in poor rendering quality. contrast, our method produces high-fidelity renderings with the generated high-quality 3DGS, accurately capturing both geometry and appearances.
We further make comparison with the GAN-based 3D generative model EG3D [5] and the NeRF. based SSD-NeRF [8] on unconditional generation of car models under ShapeNet dataset. The visual comparison shown Fig. 10, where DifIGS significantly outperforms EG3D and SSD-NeRF on the geometry and appearance details.
DifTTF
Ours
Figure 9: Visual comparisons with state-of-the-ar method DiffTF [4] on unconditional generation of ShapeNet airplanes.
17

---

EG3D
SSD-NeRF
Ours
Figure 10: Visual comparisons with fatt--of-the-ate unconditional generation of ShapeNet cars.
B.4 Ablation Study on Octree Depth
The depth the octree in our Gaussian extraction algorithm is an important hyper- parameter framework. To explore how the octree depth affects the rendering quality, conduct ablations the octree depth as shown in Tab. 5. The results demonstrate that larger octree depth leads to better quality by capturing more geometry details.
The deeper octree depths may lead to increased complexity in Gaussian extraction. To address the trade-off between the efficiency and quality, we conducted an ablation study in Tab. 5 to identify the optimal balance between computational cost and reconstruction quality. The results indicate that moderate increase in octree depths can significantly improve the Gaussian quality with very few additional cost.
Table Ablations on the sample time and Gaussian quality with different octree depths.
Depth
PSNR
SSIM
LPIPS
Sample Time (s)
7
29.77
0,9772
0.0254
0.15
8
31.70
0.9824
0.0156
0.16
9
32.70
0.9842
0.0162
0.21
10
34.01
0.9879
0.0149
0.58
B.5 Ablation Study Framework Designs
We also conduct ablations to demonstrate the effectiveness introducing triplanes as the decoder implementation of our Gaussian VAE. We replace the Triplane decoder with simple Multi-Layer Perceptron (MLP) model and show the performance in Fig. 11. The results indicate that using MLP fails to capture the intricacies of Gaussian modeling adequately. The Triplane model demonstrates superior performance in terms of detail accuracy and geometric fidelity.
C Efficiency Analysis and Comparison
C.1 Parameters and Inference Time
We compare the model sizes and generation time between DiffGS and other SOTA generative models. The results are shown Tab. 6, where we highlight that DiffGS demonstrates significant efficiency compared to the SOTA baselines DiffTF, Shap-E, SSDNeRF and DreanGaussian. DiffGS also offers competitive generation time with GET3D and LGM. DiffGS has a significantly smaller number of parameters compared most baseline methods. The smaller parameter count of DiffGS translates
18

---

PSNR:
34.47
33.71
21.50
PSNR:
37.01
36.62
20.04
(a) Ground-truth
(b) Gaussian Splatting
(c) Triplane
(d) MLP
Figure 11: Ablations on 3D representation implementations. (a) The input shapes from ShapeNet. (b) Gaussians trained on the ground truth shape. (c) Reconstructed Gaussian primitives obtained with our Triplane-based Gaussian Variational Auto-encoder using proposed Gaussian Splatting functions. (d) Replace the Triplane architectures with simple MLPs.
Table 6: Comparison of model parameters and inference time. Inference time is measured on a single NVIDIA RTX 3090 GPU.
GET3D
DiffTF
SSDNerf
Shap-E
DreamGaussian
LGM
Ours
Parameters (M)
34.3
929.9
244.9
759.5
258.7
429.8
127.4
Inference Time (s)
5.0
99.7
27.4
27.1
197
10.5
9.5
into reduced memory usage and potentially faster inference times, which can be advantageous in environments where computational resources are limited.
C.2 Gaussian Numbers during Extraction
We conducted supplementary experiments to analyze the impact of the number of extracted Gaussian points optimization time. The results of this ablation study are presented in Tab. 7. The results show that for generations with a smaller number of Gaussians, e.g., 50K, the optimization is extremely fast, taking only 0.64 seconds to converge. For high-quality Gaussian generations with 350K primitives, the optimization time increases to seconds, which still efficient. In our experiments, we selected configuration of 350K Gaussian points. This choice balances quality and computational efficiency, providing robust representation of the model without excessively increasing processing time.
Table 7: Ablations on Gaussian Number. Optimization time is measured single NVIDIA RTX 3090 GPU.
Num
PSNR
SSIM
LPIPS
Opt. Time (s)
50K
28.61
0.9787
0.0251
0.64
100K
30.35
0.9838
0.0151
1.26
350K
34.01
0.9879
0.0149
2.5
19

---

D Implement Details
We implement DiffGS with Pytorch Lightning. We leverage the Adam optimizer with a learning rate of 0.0001. We train DiffGS with eight 3090 GPUs and the convergence in each class of ShapeNet dataset takes around 5 days. The Guassian encoder and Triplane decoder are implemented based on the IDF-VAE encoder and decoder of DiffusionSDF [12], where we modify the PointNet [50] in the SDF-VAE encoder to receive K dimension 3DGS as the inputs.
E
Limitation
One limitation of our method is that it sometimes produces overly creative color schemes. We illustrate this issue with two examples in Fig. 12. As shown, the failure cases of DifGGS result in excessively colorful appearances. For instance, the chair shown exhibits a color transition from yellow to blue to green and finally to orange. These overly creative generations may not accurately reflect real-world shapes.
Figure 12: Failure cases of our method. DiffGS sometimes generates overly creative shapes with colorful appearances.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We analysis the limitations of our method in Sec.F of the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the detailed information in reproducing our methods in Sec.3, Sec.4 the main paper and the appendix. We also provide a demonstration code of our method in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide our demonstration code as part of our supplementary materials. We will release the source code, data and instructions upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
ceabbbiissioooooccyy. for more details. httee//uiddee/oeeuueeeeeeeeeeeee
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbpiccccceecce geides/ CodeeSbbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the training and testing details in the experiment section (Sec.4) and the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We report the average performance the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computer resources needed to reproduce the experiments are provided in Sec.C& of the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https://neurd: /ttp::////ee.../eeeeeeeeee cc/public////ppccpcccccccccccclc Ceteec////eeeeeeeeeee csQuidel ines?
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
We discuss the applications and potential impacts of our method in the introduction.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We use the open- sourced datasets under their licenses.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.hho....c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "GNmbUen2nk", "reviewer": "Reviewer_p5pi", "summary": "The paper introduces DiffGS, a text/image-to-3D generative model with 3D Gaussian splatting as its output representation. The model generates Gaussians from text/image condition by CLIP-augmented latent diffusion model (LDM), whose output is a latent vector that can be decoded into a triplane representation. DiffGS then translates this triplane representation by first sampling a plausible set of Gaussian locations from the learned implicit function (GauPF), and then reconstructs colors and shapes of these Gaussians with two more learned implicit functions (GauCF, GauTF) which are used as components of the objective function in the optimization-based generation.\n\nTherefore, the model consists of a (1) Gaussian LDM, a (2) latent-to-triplane decoder and (3) three implicit functions (GauPF, GauCF, GauTF). The (2, 3) decoder models (triplane decoer and three implicit functions) are trained in a VAE fashion, and the (1) Gaussian LDM is trained over the top of VAE’s learned latent space. The method is tested with ShapeNet and DeepFasion3D dataset with FID/KID metrics.", "questions": "1. Although the authors claim for the generation of 3D Gaussian splatting, the actual generated sample is the triplane representation, which is realized as a Gaussian splatting with implicit neural representation. Since there are already a well-established triplane-to-NeRF decoding (e.g., TriNeRFLet: A Wavelet Based Triplane NeRF Representation, ECCV 2024, code released), are there **specific technical benefits of decoding this information into Gaussian splatting in this case?**\n2. The proposed work involves a dedicated generative model specific to certain domains (e.g., ShapeNet). However, there are other branches of works that approaches text/image-to-3D generation leveraging the power of 2D diffusion models, e.g., DreamFusion and its offsprings that uses SDS losses, which seem to be mature at this moment. Concerning the recent achievements of the latter branch of works, I am not fully convinced to training a domain-dedicated 3D generator. May I ask **why the authors think training the dedicated 3D generator is a good way of solving 3D generation tasks?**\n3. In line 109: lake → lack\n\nPlease note that these questions are not counted in my overall scoring.", "limitations": "Yes, the limitations are addressed in the appendix.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The authors have attached supplementary materials that effectively show their code and outputs of their method.\n2. The paper contains ablation study (Table 2) that justifies each component of their framework.\n3. Visual results show that the proposed method effectively generates complicated geometries with Gaussian splatting representation.", "weaknesses": "1. Since the paper targets text/image-to-3D generation with Gaussian splatting based on the domain-specific (e.g., ShapeNet/DeepFasion3D) latent diffusion model, I believe that the results should be compared with other similar 3D generative models, at least those mentioned in Section 2.2 by the author. **Particularly, the work seems to solve similar problem with LGM (ECCV 2024, code released), and therefore should be thoroughly compared with it.**\n2. **The detailed structure of the proposed models are not presented in the paper.** For example, that is the dimension of the latent z in Figure 2, and why did the authors have chosen the structure? How does the authors model the Gaussian Splatting encoder that encodes the (fixed or not fixed number of) Gaussians into this latent z? Does the number of Gaussians fixed in this model? How are the architectures of GauPF/GauCF/GauTF designed? How does the DALLE-2’s LDM, which is originally proposed to model the latents of a 2D image, be used to model the latent triplane representation? How does the triplane decoder shaped? How many parameters are used in the models? Such information should be included in the main manuscript (or at least in the appendix) for proper presentation of the idea.\n3. The proposed method uses post-optimization steps to infer the generated 3D Gaussian splatting. This seem to require **additional generation time in the inference steps.** This can be considered a weakness since there are already many large generative models of Gaussian splatting that generate samples in tens of seconds such as LGM. Even if the proposed model requires long delay, such delay can still be justified if the results are sufficiently detailed and the sampling time is sufficiently short (as how we acknowledge the contribution of 2D LDM models). However, there is no information of the exact generation time specified in the paper.\n4. The positive correlation between the number of Gaussians / octree depths and the reconstruction quality enhancement in Table 3 and 4 seems to be trivial. I believe more interesting question is **how much the amount of increment in computational cost trades off against the quality enhancement**, etc.\n5. Since the model uses CLIP to perform text/image-to-Gaussian generation, the paper should report **numbers that show the text-fidelity, e.g., CLIP-scores**. Comparing only with FID score (Table 1) does not seem to effectively demonstrate the superiority of the proposed method.\n\nAs a summary, my key concerns are the presentation of the materials, the lack of comparison with the more recent papers (e.g., LGM) that appears in Section 2.2 of the main manuscript, justification of the sampling-time delays that seem to present, and the lack of quantitative comparison on text-fidelity. Unless these concerns are resolved, I believe that the paper is not ready to be published in the venue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer p5pi for the thorough review and valuable feedback. We have addressed each point in our response below.\n\n**Q1: Comparison with other recent 3D generative models**\n\nWe refer the reviewer to the \"Global-Q1: Evaluation Against Other SOTA 3D Generative Models\" section of the global response for a comparison with other methods.\n\n**Q2: Detailed Structure of Proposed Models**\n\nBelow, we address each point raised by the reviewer.\n\n1. Latent dimension z:\n\nThe latent vector z in Figure 2 of the paper is designed to capture the essential features of the Gaussian Splatting representation. We set the latent dimension to 256 to balance expressiveness and computational efficiency. The choice of 256 dimensions is empirically determined to balance encoding diverse 3D shapes with manageable model size.\n\n2. Gaussian Splatting Encoder\n\nThe Gaussian Splatting encoder transforms the input Gaussians into the latent space. We utilize a PointNet[1]-based architecture as the implementation of the encoder, which efficiently processes gaussian data . In detail, We adjust the dimensions of the input in PointNet within the SDF-VAE encoder to accept a K-dimensional 3DGS as input.\n\n3. Design of GauPF/GauCF/GauTF\n\nGauPF, GauCF, and GauTF share a similar architecture consisting of three MLP-based blocks. The input latent vector has a dimension of 131, comprising the latent dimension sampled from triplane and the XYZ coordinates. The feature channels for the first two blocks are [131,512,512,512,512] and [643,512,512,512,512] with a skip connection preceding the second block. To leverage the properties of Gaussians, the last block of GauTF is designed uniquely. To ensure the generated scale remains within a certain threshold, a truncation operation is applied to the scale attribute. Besides, in order to achieve uniformity in the form of the rotation quadruple, we normalize it so that its last dimension has a unit length. Finally, we use the sigmoid activation function to restrict opacity values to the range of 0 to 1.\n\n4. Use of DALLE-2’s Latent Diffusion Model (LDM)\n\nThe core operation of LDM is performing diffusion on a one-dimensional latent space. This diffusion process is general and applicable to various data types, including 3D representations like triplanes. Although LDMs are frequently used for 2D image processing, their ability to handle latent spaces makes them suitable for broader applications. We begin by encoding the triplane data into a latent space suitable for latent diffusion. This encoding process captures the essential geometric and visual information needed for accurate 3D reconstruction. The LDM operates on this encoded latent space, applying diffusion processes. \n\n5.\tTriplane Decoder Architecture\n\nThe triplane decoder reconstructs the 3D shape from the latent space using transposed convolution layers, interleaved with batch normalization and activation functions. This structure is designed to progressively upscale the 2D feature maps while maintaining high fidelity and detail.\n\n6.\tParameter Counts\n\nThe complete DiffGS model consists of approximately 127.4 million parameters. In Table B, we compare the parameter count of our method with other SOTA 3D generation methods. The results demonstrate the efficiency of DiffGS, which uses significantly fewer parameters than DiffTF (929.9M), ShapE (759.5M), and LGM (429.8M).\n\n**Q3: Addressing Post-Optimization Steps**\n\nWe refer the reviewer to \"Global-Q2: Addressing Post-Optimization Steps\" for results and analyses.\n\n**Q4: Trade-Offs Between Computational Cost and Quality Enhancement**\n\nWe refer the reviewer to ”Global-Q3: Balancing Computational Cost and Quality Enhancement“ for a detailed discussion on balancing computational costs with quality enhancement.\n\n**Q5: Inclusion of Text-Fidelity Metrics**\n\nWe conducted experiments to evaluate the text fidelity of our text-to-Gaussian generation using CLIP-scores in Table A of the rebuttal PDF. Compared to LGM and other baseline models, our method shows a consistent improvement in CLIP-scores, confirming the superior text fidelity of our approach.\n\n**Q6: Decoding Triplane Representation into Gaussian Splatting**\n\nThe triplane representation serves as an effective intermediate step that simplifies the complex task of directly regressing explicit 3D Gaussian attributes. Triplanes offer a structured approach to capture spatial information without being hindered by the non-structural nature of Gaussian splatting. The Gaussian splatting is inherently more efficient for training and rendering compared to volumetric approaches like NeRF. It utilizes point-based representations, which require less computational overhead and facilitate quicker convergence during training. The use of Gaussian splatting enables faster rendering speeds, making our approach suitable for real-time applications such as virtual reality and interactive 3D simulations.\n\n**Q7: Justification for Training a Domain-Dedicated 3D Generator**\n\nSDS-based methods require extensive optimization and are computationally intensive, particularly when distilling scores from 2D to 3D. They are also highly sensitive to hyperparameter settings like learning rates and noise schedules. Additionally, these models often produce the Janus problem in 3D shapes. We make a comparison with SDS-based Gaussian generative model DreamGaussian in Figures A and B, as well as Table B. The results demonstrate that our dedicated 3D generator DiffGS performs faster and produces more accurate, high-fidelity geometric structures than SDS-based methods.\n\n**Q8: Typographical Error**\n\nWe thank the reviewer for identifying the typographical error in line 109, which will be corrected in the revised manuscript.\n\n[1] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qEzmh3liG3", "reviewer": "Reviewer_3Kid", "summary": "This article first transforms 3DGS into a regular representation: the triplane structure, to separately model the location probability, color, and other attributes of each Gaussian. This representation can then be used to train a generative model through a VAE and LDM.\nTo sample Gaussian positions from the triplane structure, this work uses an octree to partition the space based on the magnitude of the location probability. It initializes the Gaussian point positions by sampling from the nodes with higher probability, and then further optimizes the final positions.", "questions": "1) It seems the optimization target in equation (9) is wrong. Should it be -log(\\psi(\\rho))", "limitations": "The author has shown some failure cases and limitation of this papar.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Strength:\n1) This paper is well written and easy to understand\n2) This design supports versatile downstream tasks: test/image conditional GS generation; GS completion; Point-to-GS Generation", "weaknesses": "Weakness:\n1) This work is only trained on a small dataset. Similar works such as \"triplane meets gaussian splatting\" are trained on the Objverse dataset, which can visualize more diverse results. This might be due to the cumbersome data processing, and will limit the scaling up in the future.\n2) The equation (3) is a little weird. Why the choice of \\tau matters as shown in the ablation study?\n3) The 3DGS representation is not properly evaluated. \nGiven an existing 3DGS, is it possible to be reconstructed by firstly represented as GauPF, GauCF, GauTF and then recover by Octree-Guided geometry sampling?\n4) Personally, I'm not sure whether 3DGS is a suitable representation for 3D generation since it's highly irregular, which cannot be embedded into a generative model conveniently.  Moreover, the data processing is really time-consuming, which will further limit the large-scale training.\n5) More comparison with GS-based reconstruction model should be added, such as \"LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful to the reviewer 3Kid for the thoughtful feedback and time invested in evaluating our work. We address each question below.\n\n**Q1: Dataset Size and Scalability**\n\nWe justify that data fitting is a common step for most generative models. For example, DiffRF requires voxelized radiance field fitting during data processing, which may take hours for each shape. NDF [1] and DiffTF require SDF-based and NeRF-based triplane fitting for data preparation. In comparison, our method only takes about 3 minutes to fit the Gaussian representation of an object using a single NVIDIA RTX 3090 GPU. This efficiency demonstrates the capability of our approach to handle 3D data with minimal computational overhead. During the rebuttal period, we faced time constraints that limited our ability to process larger datasets such as Objaverse. Our current focus was on demonstrating the model's capabilities within the constraints of available resources and time. The inherent efficiency of our method makes it well-suited for scaling up to larger datasets like Objaverse in the future.\n\n**Q2: Equation (3) and the Choice of $\\tau$**\n\nIn the paper, we have provided a comprehensive explanation of the role of $\\tau$ and why its selection is critical, as demonstrated in our ablation study (Table B). By controlling the truncation of the GauPF, $\\tau$ helps concentrate the learning process near the surface of objects. The truncation strategy prevents the learning from being influenced by distant regions in space, which are less relevant for accurate surface representation. Besides, in our ablation study, we explored both exponential and linear mappings for distance-to-probability transformations. While both methods are commonly used, our results showed that linear mapping performances better than exponential mapping in our framework. The linear mapping approach with a truncation strategy resulted in more stable and accurate training outcomes, highlighting the significance of selecting an appropriate distance-to-probability mapping technique for the task. And the ablation study identifies a suitable $\\tau$ that balances sensitivity and specificity, ensuring efficient learning of the object's surface while maintaining overall model stability.\n\n**Q3: Evaluation of 3DGS Representation**\n\nDuring rebuttal, we further evaluate our 3DGS representation by randomly selected two airplanes from the dataset and modeled each using GauPF, GauCF, and GauTF. The results are shown in Figure C of the rebuttal PDF, which provide visual comparisons of the original and reconstructed objects, showcasing the accuracy and effectiveness of our approach in capturing intricate details. In Figure C, we also report PSNR scores, indicating a strong ability to reconstruct the original 3DGS with high fidelity. The integration of GauPF, GauCF, and GauTF ensures that both geometric and visual details are preserved during reconstruction. And Octree-Guided sampling allows for efficient handling of complex geometries, reducing computational overhead by focusing resources on critical areas. This makes our method scalable and practical for complex objects.\n\n**Q4: Suitability of 3DGS for 3D Generation**\n\nCompared to NeRF, 3DGS offers several advantages, particularly in terms of rendering efficiency and high visual fidelity. The task of solving 3DGS generation is not only timely but crucial for advancing the field of 3D modeling. As 3D content creation continues to grow in demand, having efficient and robust methods like 3DGS becomes increasingly important for various applications, including gaming, virtual reality, and digital content creation. One of the primary contributions of our work is addressing the challenges posed by the discrete and unstructual nature of 3D Gaussian Splatting. Our approach provides three continuous Gaussian Splatting functions to effectively embed 3DGS into a generative model, allowing for efficient and accurate 3D generation. As mentioned in our previous response, fitting the Gaussian representation for each shape is quite efficient, taking only 3 minutes on a single NVIDIA RTX 3090 GPU. \n\n**Q5: Comparison with GS-Based Reconstruction Models**\n\nUpon review, we conducted supplementary experiments to compare the visual quality of our method with several SOTA 3D generative models, including LGM and other GS-based approaches. The results of these visual comparisons are showcased in Figure A and Figure B of the rebuttal PDF. The visual results clearly demonstrate the strengths of our method in capturing intricate details and consistency with conditions. Our approach benefits from an efficient representation of Gaussians that enhances both processing speed and visual output quality, making it well-suited for high-fidelity content creation. In Table A of the manuscript, we provide a comparison of the CLIP Score between our method and the other SOTA 3D generative methods including  Shap·E, LGM(ECCV 2024) and DreamGaussian(ICLR 2024). Our method demonstrates superior CLIP Scores, indicating that DiffGS generates 3D content that is more faithful to the input conditions. We have also included a comparison of model parameter count and generation time in Table B. DiffGS has a considerably smaller number of parameters compared to LGM, leading to decreased memory consumption and computational overhead. Additionally, our approach enables quicker generation time than other SOTA 3D generative models, showcasing its ability to efficiently create high-quality 3D content.\n\n**Q6: Correction of Optimization Target in Equation (9)**\n\nWe appreciate the reviewer's careful examination of Equation (9) and for pointing out the error in the optimization target. The revised manuscript will reflect this correction.\n\n[1] Shue, J.R., Chan, E.R., Po, R., Ankner, Z., Wu, J., Wetzstein, G.: 3d neural field generation using triplane diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20875–20886 (2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cBl1O7zbsm", "reviewer": "Reviewer_tnRu", "summary": "This paper proposes a new generative model for Gaussian primitive generation based on latent diffusion models. In detail, the method disentangles Gaussian Splatting generation into three functions, i.e., Gaussian probabilities, colors, and transforms. The method can achieve unconditional and conditional generation and extract Gaussians at arbitrary numbers via octree-guided sampling. Experiments on unconditional generation, conditional generation from text, image, and partial 3DGS, as well as Point-to-Gaussian generation, show the advances of the proposed method.", "questions": "Please refer to the weaknesses section for more details.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. DiffGS effectively deals with the discreteness of Gaussian Splats, by disentangling Gaussian Splatting into Gaussian probabilities, Gaussian colors, and Gaussian transforms.\n\n2. DiffGS shows good performances on multiple tasks, including unconditional/conditional generation from text, image, and partial 3DGS, as well as Point-to-Gaussian generation.\n\n3. DiffGS is able to generate high-quality Gaussian primitives at arbitrary numbers.\n\n4. The paper is well-written and easy to follow.", "weaknesses": "1. What is the benefit of predicting features in structural triplanes first, then extracting 3D Gaussian Splatting, compared to directly modeling the 3D using triplanes? The generation speed is not an advantage, and the performances are not compared/ablated in the experiments.\n\n2. Are Gaussian probabilities work better than directly predicting point cloud explicitly? To show the advantage of the GS disentanglement proposed in this paper, it would be better to compare it with previous work, e.g., [1].\n\n3. Experiments can be strengthened by comparing with optimization-based methods for generating GS, e.g., [2], and GS prediction from multi-view Diffusion Models, e.g., [3].\n\n4. Mathematics. \n\n* $i$ and $j$. In lines 131-132, why \"$j=1$\" and \"$j$-th\" are used for $g_i$? Similar for the query points definition.\n\n* Eq. 3 typo. \n\n* Does bigger $\\psi_{pf}$ mean larger probability? If so, do we want to maximize Eq. 9?  \nPlease double-check the math and add the necessary explanations.\n\n---\nReferences:\n\n  [1] Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. CVPR 2024.\n\n  [2] DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation. ICLR 2024.\n\n  [3] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. ECCV 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the comprehensive feedback and time that reviewer tnRu dedicated to evaluating our work. Below, we provide responses to each of your questions. We look forward to your further comments on our responses.\n\n**Q1: Benefits of Structural Triplanes in 3D Gaussian Splatting**\n\nThe triplane structure utilizes 2D planes, which allows for the integration of standard 2D neural networks. This integration provides several benefits, including access to well-established 2D processing techniques and efficient computational frameworks. Triplanes serve as a bridge to utilize the generalization capabilities of diffusion models in 3D generation. By predicting features in triplanes first, we can more effectively capture and model Gaussian distributions. Directly modeling 3D with triplanes may not leverage the full potential of these distributions, especially when complex geometries are involved. Also, triplanes allow for better handling of spatial information, facilitating the reconstruction of detailed and accurate 3D representations. We conducted an ablation study where the triplane structure was replaced with a Multi-Layer Perceptron (MLP) to evaluate its effectiveness in modeling Gaussians. The results, visualized in Figure C of the rebuttal PDF, indicate that using an MLP fails to capture the intricacies of Gaussian modeling adequately. The triplane approach demonstrates superior performance in terms of detail accuracy and geometric fidelity.\n\nWe further provide the efficiency comparison with other SOTA baselines in 3D generation, as shown in Table B of the rebuttal PDF. DiffGS signicantly outperforms other baselines in terms of generation efficiency. Although generation speed is not our sole advantage, our method effectively balances speed and quality, underscoring the strengths of the triplane approach in delivering high-fidelity results efficiently.\n\n**Q2: Advantage of Gaussian Probabilities Over Direct Point Cloud Prediction**\n\nPredicting Gaussian probabilities allows us to extract Gaussians at arbitrary numbers. This flexibility is a significant advantage over directly predicting point clouds, which are typically fixed in number and less adaptable to varying levels of detail. Moreover, direct generation of unstructured Gaussians poses challenges due to their non-structured nature. By disentangling GauPF, GauCF, and GauTF, we create a framework that systematically handles Gaussian attributes, allowing for a structured representation that addresses these challenges.\n\nDuring rebuttal, we directly compare our method with \"Triplane Meets Gaussian Splatting\" (CVPR 2024). The visual comparison results are presented in Figure A of the rebuttal PDF, illustrating the superior visual quality achieved by our approach. Our method consistently produced more accurate and detailed reconstructions compared to the baseline. By disentangling the Gaussian attributes, our approach allows for more efficient modeling of 3D structures, ensuring that both geometric and color details are preserved with high accuracy.\n\n**Q3: Comparison with Other Methods**\n\nWe conducted comparative experiments focusing on two critical tasks: text-to-3D generation and single-view 3D generation. These tasks are central to demonstrating the flexibility and robustness of our approach in different application scenarios. The visual results of our comparisons are presented in Figure A and Figure B of the rebuttal PDF. These figures illustrate the superior visual quality and fidelity achieved by our method compared to the SOTA baseline methods including Shap·E, SplatterImage (CVPR 2024), TriplaneGaussian (CVPR 2024), LGM(ECCV 2024) and DreamGaussian(ICLR 2024). Our approach consistently produced more detailed and accurate generations, effectively capturing intricate textures and geometries that are often challenging for the compared optimization-based and multi-view based methods. We also evaluated the models using the CLIP score, a metric that measures the semantic alignment between the generated 3D models and the input conditions. The results are presented in Table A of the rebuttal PDF. According to Table A, our method achieved higher CLIP scores than both DreamGaussian and LGM. This indicates that our approach more faithfully adheres to the condition inputs.\n\n**Q4: Mathematical Notation and Clarifications**\n\nWe appreciate the reviewer's careful examination on the mathematics.\n\nThe use of $j$ was indeed incorrect and should be replaced with $i$ to maintain consistency in indexing. Each Gaussian $g_i$ should be consistently indexed with $i$, not $j$.\n\nWe will correct any typographical errors in Equation (3), ensuring that it accurately conveys the intended mathematical relationship.\n\n$\\psi_{pf}$: This function represents a probability-related component within our model, implying that larger values of $\\psi_{pf}$ correspond to higher probabilities. And We will revise Equation (9) to correctly reflect the maximization intention.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LIdI2ToABi", "reviewer": "Reviewer_6WmW", "summary": "The paper is about generating 3D objects by using gaussian splatting. The method converts the points (with gaussian properties) to continuous fields. With this idea, the irregular structured data can be easily processed by neural networks.", "questions": "How do the authors optimize the loss in Eq 6?\nI understand how the point are converted to fields thus we can query the probabilities continuously in the space\nBut how about other properties? They are not converted to continuous fields.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The idea is interesting and novel. Usually we need many points (10k-1M) to represent a single object. With this large point clouds, it is very challenging to process the data with neural networks. Thus the paper proposed to convert a point cloud to a field which can be easily processed by neural networks.\n\nThe octree design seems to be also interesting. This is useful to get fine-grained structure.", "weaknesses": "The results shown in the supplementary video is not very satisfying. Even the paper showed some metrics comparing to some existing methods, the visual quality is much worse.\n\nSpecifically, the metrics shown in Table 1 are better than some prior works. However, I believe some important works are missing in the table, e.g., EG3D, DiffRF, SSDNerf. The evaluation protocol seems to be different from these works. I am curious how the authors pick 50k images as the reference set. SSDNerf used testing set as the reference set. I know there is always a debate between training and testing set in the evolution of generative models. But this should be mentioned in the paper.\n\nFor the visual quality, results of EG3D and DiffRF are much better than this paper. It would be better the authors can do a visual comparison with these works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the insightful review provided by Reviewer 6WmW. Your thorough analysis and constructive feedback have significantly enhanced the quality of our work.\n\n**Q1: Visual Quality and Evaluation Protocol**\n\nWe justify that DiffGS achieves SOTA performance in terms of both numerical and visual qualities. In response to the reviewer's comments, we conducted additional experiments comparing our method to the SOTA baseline methods including EG3D(CVPR 2022),  SSDNeRF(ICCV 2023), Shap·E, SplatterImage (CVPR 2024), TriplaneGaussian (CVPR 2024), LGM(ECCV 2024) and DreamGaussian(ICLR 2024).  We are unable to compare our method with DiffRF since it is not open-sourced. However, we have compared our method against other stronger baselines such as DiffTF and SSDNeRF, which have been shown to outperform DiffRF in various benchmarks. The visual comparison results are presented in Figure A, Figure B and Figure D of the rebuttal PDF. The results show that our method outperforms the SOTA baseline in capturing complex details while preserving geometric consistency. Additionally, we evaluate text consistencies against other state-of-the-art methods in Table A of the rebuttal PDF. Our approach shows strong alignment between textual descriptions and the generated 3D models. Furthermore, Table B highlights a comparison of model parameter counts and generation times. DiffGS employs far fewer parameters than LGM, leading to lower memory consumption and reduced computational demands. Additionally, our method achieves quicker generation times than other top 3D generative models, underscoring its efficiency in producing high-quality 3D content.\n\nFor our evaluation, we selected a diverse set of 50K images as the reference set. This selection was designed to reflect a wide range of scenarios within the dataset. We used the testing set as the reference set for rendering. This approach is aligned with practices like those used in SSDNeRF, ensuring that our evaluations are based on unseen data and provide a realistic assessment of model performance.  By choosing the testing set as our reference, we prioritize demonstrating the model's ability to generalize and generate high-fidelity outputs in unseen scenarios. This decision aligns with the goals of ensuring robust evaluation and fair comparison across models.\n\n**Q2: Optimization of Loss in Equation 6**\n\nWe compute the loss directly against real Gaussians, which serves as the ground truth for optimization. This approach ensures that the model aligns with actual data distributions effectively. \n\nEach property of the Gaussians, including position, color, and shape, is represented as a continuous field in the spatial domain. This allows for precise querying and supervision with the real Gaussians.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F18ON44wah", "reviewer": "Reviewer_cAhv", "summary": "This paper proposes DiffGS, a novel diffusion-based generative model that can generate 3D Gaussian Splatting (3DGS) representing an object. As 3DGS is naturally discrete and unstructured, it is not practical to directly diffuse the 3DGS representation. Thus, the authors propose to reconstruct the 3DGS with structured triplanes that represent three functions: a probability function that models the geometry (the mean of Gaussians) of 3DGS, a color function that models the color, and a transform function that models the transformation parameters of the Gaussians. Additionally, the authors propose a VAE that encodes 3DGS into a latent vector. By representing 3DGS with such structured data, it is then possible to train a diffusion model on the latent vectors, which can be recovered to 3DGS for rendering. Experiments show that the proposed DiffGS achieves preferable quality results compared to baselines and by adding conditions to the diffusion model, it can achieve various applications.", "questions": "1. What is the inference time of DiffGS, and how much does the optimization process during inference influence the inference speed?\n2. How many parameters does the model use?", "limitations": "Yes, the limitations are discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The idea of representing 3DGS with disentangled functions is novel and practical. Indeed, it is difficult to diffuse unstructured data like 3DGS, and representing 3DGS using functions that can both encode and decode is a novel direction.", "weaknesses": "1. It can be observed that there might be holes and incomplete shapes in the generated 3DGS (Fig.5(b)). I suspect the reason for this is that the proposed Gaussian extraction algorithm struggles to sample points in high-frequency parts like the legs of chairs. While this can be improved by increasing the number of sampled Gaussians or the depth of the octree, it might be quite computationally intensive.\n2. The inference efficiency is not shown in the experiments. During each sample, DiffGS requires optimization over the sampling points, which may further slow down the inference speed of the model. The authors should provide the influence of the optimization on the inference time. However, it is acceptable if the inference speed is slower than baselines since diffusion models are naturally slow to inference.\n3. An important baseline is missing. Splatter Image [1] is also a 3DGS-based object reconstruction method and shares the similar idea of representing unstructured 3DGS with structured data. Since the Splatter Image models trained on ShapeNet are also available, it is reasonable to compare DiffGS with Splatter Image under the single-view reconstruction task.\n4. The authors do not provide the number of parameters of DiffGS and the comparisons with baselines. In the quality comparisons, the authors should keep DiffGS and other baselines having around the same number of parameters.\n\n[1] Splatter Image: Ultra-Fast Single-View 3D Reconstruction, CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere thanks to Reviewer cAhv for their comprehensive review and valuable suggestions. In response to the concerns raised, we have offered detailed explanations in the subsequent section, and we are eager to engage in further dialogue.\n\n**Q1: Incomplete Shapes in 3DGS**\n\nIt is indeed possible that some models exhibit holes and incomplete shapes, especially in regions with high-frequency details. This issue arises from challenges in capturing fine details during the Gaussian extraction process. One straightforward approach to improve this is to increase the number of sampled Gaussians. By doing so, we can achieve a higher resolution representation, capturing more details in the high-frequency regions. Similarly, increasing the depth of the octree can enhance the model's ability to resolve fine details. We aim to balance quality and computational efficiency. While increasing the number of Gaussians and octree depth improves detail capture, it also escalates computational costs. We focus on a balance between satisfactory results and inference efficiency,, and find it acceptable to occasionally generate shapes with minor artifacts.\n\n**Q2: Inference Efficiency and Optimization Impact**\n\nDuring Gaussian extraction, DiffGS performs optimization over the Gaussian positions to enhance the quality and accuracy of the generated 3D Gaussian Splatting geometries. This optimization step, while essential for ensuring high-quality outputs, does introduce additional computational overhead. We conducted supplementary experiments to analyze the impact of the number of Gaussian points on optimization time. The results of this ablation study are presented in Table C of the rebuttal PDF. In our experiments, we selected a configuration of 350K Gaussian points. This choice balances quality and computational efficiency, providing a robust representation of the model without excessively increasing processing time. The optimization time for this configuration is approximately 2.5 seconds. We also compared our model's generation time with other state-of-the-art 3D generative methods. The results are shown in Table B of the rebuttal PDF, where we highlight that our method demonstrates significant efficiency compared to the SOTA baselines DiffTF, ShapE, SSDNeRF and DreamGaussian,  despite the added optimization step. DiffGS also offers competitive generation time with GET3D and LGM. \n\n**Q3: Comparison with Splatter Image Method**\n\nFor a comprehensive comparison with Splatter Image, we evaluated both methods under the single-view generation task. We conducted supplementary experiments to compare the visual quality of reconstructions generated by DiffGS and Splatter Image. The results are presented in Figure A of the rebuttal PDF. The visual comparison indicates that DiffGS produces higher fidelity reconstructions with better handling of fine details and fewer artifacts compared to Splatter Image.\n\n**Q4: Number of Parameters and Baseline Comparisons**\n\nWe have included the number of parameters for DiffGS and other baseline methods in Table B of the rebuttal PDF. As shown in the table, DiffGS has a significantly smaller number of parameters compared to most baseline methods. The smaller parameter count of DiffGS translates into reduced memory usage and potentially faster inference times, which can be advantageous in environments where computational resources are limited. Despite having fewer parameters, DiffGS demonstrate superior performance, demonstrating that its architecture efficiently captures essential features for high-quality 3D reconstruction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tFBmf1PlpD", "reviewer": "Reviewer_TmLe", "summary": "This paper presents a method to turn the representation of the gaussian splatting as a point cloud into a VAE representation. With that representation it is shown how to perform  several tasks such as unconditional generation, conditional generation, and point-to-gaussian generation, all of which are done using a diffusion model that operates on the latent . It is shown that it performed better than previous methods on these tasks.", "questions": "see weaknesses.", "limitations": "see weaknesses.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The method presented in the paper is a novel way to incorporate the gaussian splatting attributes into generation tasks. The method shows better results on some  tasks than previous methods.", "weaknesses": "The main one is that while reading the paper I did not get the answer to the question why enforcing a VAE to predict gaussian splatting should work better than a simpler method. It was shown that this is the case by experiments, but what is the motivation? As SOTA NeRF methods produce better reconstruction than the vanilla GS. Since the paper presents a method to create a new 3d object from the gaussian splatting primitives the relevant papers from the early (1990-2002) use of  gaussian splatting should be referred to. Additionally the number of quantitative results is limited (even in comparison to the paper of the referred baselines), and makes it hard to evaluate the method. Moreover, the second part of the first sentence in the abstract “... yet the generation of Gaussian Splatting remains a challenge due to its discreteness and unstructural nature.” is IMHO logically challenging, only after reading the introduction it becomes clear that the authors referred to generation novel 3d assets using gaussian splatting. The main concepts and ideas in the paper are interesting, but there is still work required in both writeup and experimental parts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "It is with great appreciation that we acknowledge the insightful feedback from Reviewer TmLe. We have addressed the questions and comments with careful consideration, and we encourage continued conversation to ensure the robustness of our findings.\n\n**Q1: Motivation, Literature, and Experimental Evaluation**\n\nVariational Autoencoders (VAEs) are well-suited for capturing complex distributions in a lower-dimensional latent space, which encodes diverse data into a compact latent representation, allowing for more efficient and structured generation of 3D models. In comparison to Neural Radiance Fields (NeRF), 3D Gaussian Splatting presents multiple benefits, especially regarding rendering efficiency and superior visual fidelity. Addressing the challenge of 3DGS generation is both timely and essential for the progression of the 3D modeling domain. The revised abstract will explicitly state the novelty and motivation behind using Gaussian splatting for novel 3D asset generation, emphasizing its significance in overcoming the limitations of discreteness and structural complexity. \n\nWe carried out further experiments to compare our approach with the state-of-the-art baseline methods, such as EG3D (CVPR 2022), SSDNeRF (ICCV 2023), Shap·E, SplatterImage (CVPR 2024), TriplaneGaussian (CVPR 2024), LGM (ECCV 2024), and DreamGaussian (ICLR 2024). As shown in Figure A, Figure B and Figure D of the rebuttal PDF, our method shows more refined geometry and detailed textures. Besides, we compare text consistencies with the other SOTA methods in Table A of the rebuttal PDF. Our method demonstrates a competitive alignment between the text descriptions and the resulting 3D models. Besides, we provide a comparison of model parameter counts and generation times in Table B of the rebuttal PDF. DiffGS has significantly fewer parameters than LGM, resulting in reduced memory usage and computational overhead. Moreover, our approach achieves faster generation times than other state-of-the-art 3D generative models, demonstrating its efficiency in creating high-quality 3D content.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers for their invaluable feedback and the time they spent evaluating our work. We are delighted that the reviewers recognized the representation and importance of our paper. We respond to each reviewer individually, providing comprehensive analyses, visualizations, and ablation studies to address all the questions raised. We have uploaded a rebuttal PDF containing experimental results and visualizations. In the following responses, we refer to this document as the "rebuttal PDF," as in "in Table A of the rebuttal PDF." Below, we address some of the common questions found in the reviews.

**Global-Q1: Evaluation Against Other SOTA 3D Generative Models**

We conduct additional experiments comparing DiffGS to other recent 3D generative models on text-to-3D and image-to-3D generation. Table A of the rebuttal PDF highlights the competitive performance of our model in text and 3D alignment tasks. As depicted in Figure A and Figure B of the rebuttal PDF, our model can produce high-quality generations with more consistent geometry and more detailed textures. DreamGaussian tends to generate overly saturated images and is affected by the Janus problem. Shape-e faces challenges in producing semantically accurate and complex geometries. LGM generates multi-view images from text or a single viewpoint and reconstructs 3D Gaussian distributions. However, inconsistencies in the generated multi-view and the limited number of output Gaussians often result in inaccurate geometric reconstructions and lower-quality rendered results. In contrast, DiffGS introduces a novel disentangled representation of 3D Gaussian Splatting using three functions—Gaussian Probability Function (GauPF), Gaussian Color Function (GauCF), and Gaussian Transform Function (GauTF). This representation allows us to generate high-quality Gaussians at arbitrary numbers, resulting in sharper and more detailed textures.

**Global-Q2: Addressing Post-Optimization Steps**

The post-optimization process in our method is designed to refine and enhance the accuracy of the generated 3D Gaussian splatting by optimizing the proxy points to the exact Gaussian centers with the largest probabilities. This step ensures the high fidelity and details of the final generated Gaussians. We present the overall generation time in Table B of the rebuttal PDF and the optimization time for various Gaussian quantities in Table C of the rebuttal PDF. The results show that for generations with a smaller number of Gaussians, e.g., 50K, the optimization is extremely fast, taking only 0.64 seconds to converge. For high-quality Gaussian generations with 350K primitives, the optimization time increases to 2.5 seconds, which is still efficient.

**Global-Q3: Balancing Computational Cost and Quality Enhancement**

The increase in computational cost with additional Gaussians and octree depths is primarily due to the increased complexity in handling more data points and subdivisions. To address the trade-off, we conducted a series of experiments in both Table C and Table D of the rebuttal PDF to identify the optimal balance between computational cost and reconstruction quality. The results indicate that a moderate increase in Gaussians or octree depths can significantly improve the Gaussian quality with minimal additional cost. For instance, lifting the number of Gaussians from 50K to 350K led to a 20% increase in PSNR while only increasing computation time about 1.8 seconds. 

---

Thank you again for your insightful feedback and we are looking forward to continuing the discussion

Meta-review: The paper received mixed ratings, and after the rebuttal, reviewers TmLe and 3Kid did not provide strong input, but the other reviewers recommended accepting the paper. The main concerns from two negative reviewers mainly focus on the presentation (3Kid) and justification of the use of 3DGS (TmLe). Both concerns are valid, but given the recent interest in the 3D vision community, the work on 3DGS seems to be, at least, of interest to many researchers, as demonstrated by the other reviewers who favour the paper. Also for the presentation, there are mixed evaluations, which makes the AC believe while it is at the border, it is acceptable. The AC therefore recommends accepting the paper.

The AC would further like to note that the rebuttal played an important role and, thus should be integrated into the revision.
Decision: Accept (poster)