GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling
Authors: Bowen Zhang
Yiji Cheng
Jiaolong Yang
Chunyu Wang
Feng Zhao
Yansong Tang
Dong Chen
Baining Guo
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Generative Modeling; Gaussian Splatting; Optimal Transport
Primary area: generative_models
TL;DR: We present a structured and explicit representation for 3D generative modeling by structuring Gaussian Splatting using Optimal Transport, achieving state-of-the-art generation quality.

Abstract: We introduce a radiance representation that is both structured and fully explicit and thus greatly facilitates 3D generative modeling. Existing radiance representations either require an implicit feature decoder, which significantly degrades the modeling power of the representation, or are spatially unstructured, making them difficult to integrate with mainstream 3D diffusion methods. We derive GaussianCube by first using a novel densification-constrained Gaussian fitting algorithm, which yields high-accuracy fitting using a fixed number of free Gaussians, and then rearranging these Gaussians into a predefined voxel grid via Optimal Transport. Since GaussianCube is a structured grid representation, it allows us to use standard 3D U-Net as our backbone in diffusion modeling without elaborate designs. More importantly, the high-accuracy fitting of the Gaussians allows us to achieve a high-quality representation with orders of magnitude fewer parameters than previous structured representations for comparable quality, ranging from one to two orders of magnitude. The compactness of GaussianCube greatly eases the difficulty of 3D generative modeling. Extensive experiments conducted on unconditional and class-conditioned object generation, digital avatar creation, and text-to-3D synthesis all show that our model achieves state-of-the-art generation results both qualitatively and quantitatively, underscoring the potential of GaussianCube as a highly accurate and versatile radiance representation for 3D generative modeling.

GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling
Bowen Zhang
Yiji Cheng?*
Jiaolong Yang
Chunyu Wang
Feng Zhao !1
Yansong Tang"
Dong Chen
Baining Guo
'University of Science and Technology of China
TTinnhua University
Microsoft Research Asia
Abstract
We introduce a radiance representation that is both structured and fully explicit and thus greatly facilitates 3D generative modeling. Existing radiance representations either require an implicit feature decoder, which significantly degrades the modeling power of the representation, to integrate with mainstream 3D diffusion methods. We derive GaussianCube Gaussian fitting algorithm, which yields high-accuracy fitting using a fixed number of free Gaussians, and then rearranging these Gaussians into predefined voxel grid via Optimal Transport. Since GaussianCube is structured grid representation, it allows us to use standard 3D U-Net as our backbone in diffusion modeling without elaborate designs. More importantly, the high-accuracy fitting of the Gaussians allows to achieve a high- quality representation with orders of magnitude fewer parameters than previous structured representations for comparable quality, ranging from one to two orders of magnitude. The compactness of GaussianCube greatly eases the difficulty of 3D generative modeling. Extensive experiments conducted on unconditional and class- conditioned object generation, digital avatar creation, and text-to-3D synthesis all show that our model achieves state-of-the-art generation results both qualitatively and quantitatively, underscoring the potential of GaussianCube as a highly accurate and versatile radiance representation for 3D generative modeling. Project page: http:aawwsssinuuuggiiibbb...ee.e
spatially unstructured, making them difficult
by first using novel
Introduction
The field of 3D generation [59, 39, 57. 49, 8. 19, 11, 61, 76] has witnessed remarkable growth, driven by advancements in generative modeling [25, 20, 41. 17, 72, 29]. Most of the prior works in this domain leverage variants of Neural Radiance Field (NeRF) [38, 57, 40] as their underlying 3D representations, which typically consist of an explicit structured proxy representation and an implicit feature decoder. However, such hybrid NeRF variants exhibit degraded representation power, particularly when used for generative modeling where single implicit feature decoder is shared across all objects. Additionally, the high computational complexity of volumetric rendering leads to both slow rendering speed and extensive memory costs.
Recently, the emergence of 3D Gaussian Splatting (GS) [30] has enabled improved reconstruction quality and real-time rendering capabilities [69, 36, 63. 35]. The fully explicit nature of 3DGS eliminates the need for a shared implicit decoder, providing another key advantage over NeRFs. Although 3DGS has been widely studied in scene reconstruction tasks, its spatially unstructured nature presents a significant challenge when applied mainstream generative modeling frameworks.
Interns Microsoft Research Asia. EEuual advising. *Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Representation
F Fully-explicit
Real-time Rendering
Rel. Parameters,
Instant-NGP [38]
X
26.63x
Neural Vexels [57]
X
K
145.9x
Triplane [8]
X
X
1
Gaussian Splatting [30]
4.0x
Our GaussianCube
1.0x
Table Comparison with previous 3D representations with respect to spatial structure, explicitness. real-time rendering capability, and relative parameter count (Rel. Parameters) for representations of comparable quality.
To overcome these barriers, we introduce GaussianCube - an innovative radiance representation that is both structured and fully explicit, with strong fitting capabilities (see Table 1 for comparisons with prior works). The proposed approach first ensures high-accuracy fitting with predefined number of free Gaussians, and subsequently organizes these Gaussians into a structured voxel grid. Such an explicit grid-based structure permits the seamless application of standard 3D convolutional architec- tures, such as U-Net, thereby eliminating the need for complex, specialized network designs [77, 59] that are often necessary with unstructured or implicitly decoded representations.
Structuring 3D Gaussians without sacrificing fitting quality is not trivial task. A naive starting point would be obtaining a fixed number of Gaussians by omitting the densification and pruning steps in GS. However, such simplification fails to lead the Gaussians close the object surfaces and results in significant quality degradation. In contrast, we propose a censsffiaatinnccosttiiiii which retains the original pruning process yet constrains the number of Gaussians that perform densification, ensuring the total does not exceed a predefined maximum No For the subsequent structuralization, we allocate the Gaussians across N N N voxel grid using Optimal Transport (OT). Consequently, our fitted Gaussians are systematically arranged within the voxel grid, with each voxel containing the features of Gaussian. The proposed OT-based structuralization achieves maximal spatial correspondence, characterized by minimal total transport distances, while preserving the expressive power of 3DGS.
fitring strategy,
The structured nature of GaussianCuhe enables us to perform efficient 3D diffusion [25] modeling for the following three reasons: 1) It allows the use of standard 3D U-Net as backbone for diffusion modeling without elaborate designs. 2) The spatial coherence of GaussianCube permits the use of standard 3D convolutions capture the correlations among neighboring Gaussians. facilitating effi- cient feature extraction. 3) GaussianCube enables high-quality fitting with orders of magnitude fewer parameters than prior structured representations of similar quality. Since recent works [32, 3] have demonstrated diffusion models' struggle in handling high- sigh-dimensional distributions, the compactness of GaussianCube significantly reduces the modeling difficulty of the generative framework.
We conduct comprehensive experiments to verify the efficacy of our approach. The model's ca- pability for unconditional and candiicondi generation is evaluated on the ShapeNet [9] and OmniObject3D [64] datasets. Both the quantitative and qualitative comparisons indicate that our model surpasses all previous methods. We also perform digital avatar generation dataset [62]. Our model is capable of producing high-fidelity 3D avatars conditioned on single portrait images, excelling beyond prior art in both identity preservation and detail creation. Additionally, we assess our model's capacity for the challenging text-to-3D creation task on Objaverse [14]. Our model demonstrates competitive performance both quantitatively and qualitatively, producing results consistent with the given text prompts in just 2.3 seconds. All experiments show the strong capabili- ties of our GaussianCube and suggest its potential as a powerful and versatile 3D representation for variety of applications. Some generated samples of our method are presented in Figure 1.
synthetic avatar
2 Related Work
Radiance field representation. Radiance fields model ray interactions with scene surfaces and can be in either implicit or explicit forms. Early works of neural radiance fields (NeRFs) [38, 74, 43, 1, 45] are often in an implicit form, which represents scenes without defining geometry. These works optimize continuous scene representation using volumetric ray- ray-marching that leads to extremely high computational costs. Recent works introduce the use of explicit proxy representation [8, 26, 18, 51, 40, 68) followed by an implicit feature decoder to enable faster rendering. Recently, the 3D Gaussian Splatting methods [30, 69, 63, 13, 31, 10, 71] utilize 3D Gaussians as their underlying representation and offer impressive reconstruction quality. The fully explicit representation also provides real-time rendering speed. However, the 3D Gaussians are unstructured representation, and

---

ShapeNet Car
ShapeNer Chair
Unceeditioaal
OmniObjectBD
Text-to-3 Portrait-lo- Avatar Tessonniiinol
Image-onniions
"a donut with blue frasting sprinkles.
"a yellow and black bee with white wing.
ta silver helmet with on top."
"a wooden pallet."
Our diffusion model is able to create diverse objects with complex geometry and rich texture details (top three rows). Our method also supports creating high-fidelity digital avatars (the forth row) conditioned on single portrait images (visualized in dashed boxes) and high-quality 3D assets given text prompts (the fifth row).
Figure 1:
require per-scene optimization to achieve reato-realistic quality. In contrast, our work proposes a structured representation termed GaussianCube for 3D generative tasks.
3D generation. Previous works of SDS-based optimization [44, 55, 67, 52, 12, 53, 70, 56] distill 2D diffusion priors [47] to 3D representation with the score functions, but these works are notably time-intensive, often taking several minutes to hours. While 3D-aware GANs [8, 19, 7, 21, 42, 16, 66] facilitate view-dependent image generation from single-image collections, they struggle to capture the complexity diverse objects with intricate geometric variations [65]. Although recent works [59, 39, 22, 57, 49, 73] have utilized diffusion models with structured proxy representations for 3D generation, the use of a shared implicit feature decoder across different assets restricts expressiveness and the computational demands of NeRF hinder efficient training. In contrast, introduce a structured and fully explicit radiance representation for 3D generative modeling, building upon 3DGS [30]. A coneurrent work of [23] includes elaborate designs to form the Gaussians into volumetric representation during fitting, yet does not thoroughly address global correspondence. In contrast, our approach only restricts the total count of Gaussians while allowing freedom in their spatial distribution during the fitting. We then organize these Gaussians into a voxel grid using Optimal Transport, which yields spatially coherent arrangement with minimal global offset cost, effectively easing the difficulty of generative modeling.
Method
Following prior works, our framework comprises two primary stages as shown in Figure 2: repre- sentation construction and diffusion modeling. In representation construction phase, we first apply ddenifiiation-constranne 3DGS fitting algorithm for each object obtain constant number of Gaussians. These Gaussians are then organized into the proposed spatially structured GaussianCube

---

Representation Construction
3D Diffusion
Druukaation-anntraannd ritting
Grunsia Structurntiptioo
Optimal Tranport
Timesteps
Class/magge/text
Rasterization
3D U-Net
Multi-vees
3DGS
Tranuport
Optimal
GaussianCube
Noise
Generoted GounsionCube
Figure 2: Overall framework. Our framework comprises two main stages of representation con- struction and 3D difffusion. In the representation construction stage, given multi- view renderings of cnleffication-consttiiii
3D asset, we perform Subsequently, the Gaussians structured into Gaussian Cube via Optimal Transport. In the 3D diffusion stage, our 3D diffusion model is trained to generate GaussianCube from Gaussian noise.
fitting to obtain 3D Gaussians with constant numbers.
via Optimal Transport between the positions of Gaussians and centers of predefined voxel grid. For diffusion modeling, we train a 3D diffusion model to learn the distribution of GaussianCubes, We will detail our designs for each stage subsequently.
3.1 Representation Construction
We build our GaussianCube upon 3DGS, explicit representation that offers impressive fitting quality and real-time rendering speed. However, it fails to yield Gaussians of fixed length since the adaptive density control during GS fitting can lead to varying number of Gaussians for different objects. Furthermore, the lack of a predetermined spatial ordering for Gaussians leads to a disorganized spatial structure. These aspects pose significant challenges to 3D generative modeling. To overcome these obstacles, we first introduce our manniifiaatio-constatiii free Gaussians. Then, we systematically arrange the resulting Gaussians within a predefined voxel grid via Optimal Transport, thereby achieving a structured and explicit radiance representation.
fitting strategy to obtain fixed number of
Formally, a 3D asset is represented by collection of 3D Gaussians as introduced in Gaussian Splatting [30]. The geometry of the i-th 3D Gaussian g, is given by
(x) = exp (-2 (x p4) E (x 14:)
where μe, F is the center of the Gaussian and E, E 33x3 the covariance matrix defining the shape and size, which can be decomposed into a quaternion E E and a vector e R for rotation and scaling, respectively. Morcover, each Gaussian gi have an opacity value a, E R and a color feature e; E R for rendering. Combining them together, the C-channel feature vector 0, {Fi S;, qi, O; c.) TRC fully characterizes the Gaussian gt.
fitting. Our approach begins with the aim of maintaining a constant number of Gaussians g E N across different objects during the fitting. A simplistic approach might involve omitting the densification and pruning in the original GS. However, we argue that such simplifications significantly harm the fitting quality, with empirical evidence shown in Table 6. Instead, we propose to retain the pruning process while imposing a new constraint on the densification phase as shown Figure 3 (a). The fitting process encompasses several distinct stages: 1) Densification Detection: Assuming the current iteration includes N. Gaussians, we identify densification candidates by selecting those with view-space position gradient magnitudes exceeding a predefined threshold We denote the number of candidates as Ng. 2) Candidate sampling: To prevent exceeding the predefined maximum of Nmax Gaussians, we select min (N - No, N&) Gaussians with the largest view-space positional gradients from the candidates for densification. 3) Densification: We modify the densification approach by alternating between cloning and splitting actions into separate steps. 4) Pruning Detection and Pruning: We identify and remove the Gaussians with a less than small threshold 6. After completing the fitting process, we pad Gaussians with a 0 to reach the target count of Nax without affecting the rendering results. Benefiting from proposed strategy,
Densification-consraiin

---

Na
Denoifcation
Detection
Condidate
Sampling
Densification
Pruning
Detection
Pruning
Nmax
(a)
(b)
Figure 3:
Illustration of representation construction. First, we perform ceassffiaatio-constrainn
fitting to yield fixed number of Gaussians, as shown in (a). We then employ Optimal Transport to organize the resultant Gaussians into a voxel grid. 2D illustration of this process presented (b).
attain a high-quality representation with orders magnitude fewer parameters compared to existing works of similar quality, which significantly reduces the modeling difficulty for the diffusion models.
Gaussian structuralization via Optimal Transport. To further organize the obtained Gaussians into spatially structured representation for 3D generative modeling, we propose to map the Gaussians to a predefined structured voxel grid v E IR N,N,, X X. where N, = V Naax. Intuitively, we aim to "move" each Gaussian into voxel while preserving their geometric relations much possible. While naive approaches such as nearest neighbor transport fall short in conserving these relations due to disregard for global arrangement with evidence shown in Figure 10, formulate this an Optimal Transport (OT) problem [58, 4] between the Gaussians' spatial positions {p,, i 1,..., _Max } and the voxel grid centers {z,.j I,..., Nmax ). Let D be distance matrix with Dij being the moving distance between μ4 and T,,, i.e., Dij = ||,. 22, The transport plan is represented by binary matrix T E R Nam Nan, and the optimal transport plan is given by:
minimize T
Nuxx
Nan j=1
D.
i=1
subject to
Nus ==
Tú Vie fI....,
Mun
Tú =1 (jE Nmax
1=1
10
E Nmax) fl...., Nmax).
The solution is a bijective transport plan T that minimizes the total transport distances. We employ the Sonker-Volgenaal algorithm [27] to solve the OT problem. We provide 2D illustration in Figure (b). We organize the Gaussians according to the solutions, with the j-th voxel encapsulating the feature vector of the corresponding Gaussian 0 {ur Iy. $k qk: ag. c& RC where is determined by the optimal transport plan (i.e., T 1). Note that we replace the original Gaussian positions with offsets of the current voxel center to reduce the solution space for diffusion models. As result, our fitted Gaussians systematically arranged within voxel grid and preserve the spatial correspondence of neighboring Gaussians, which further facilitates generative modeling.
3.2 3D Diffusion on GaussianCube
We now introduce our 3D diffusion model incorporated with the proposed expressive, efficient and spatially structured representation. After organizing the fitted Gaussians g into GaussianCube y for each object, aim to model the distribution of GaussianCube, i.e., p(y).
Formally, the generation procedure can be formulated into the inversion of a discrete-time Markov forward process. During the forward phase, we gradually add noise to yo ~ p(y) and obtain sequence of increasingly noisy samples {yel [0, T)} according to Y:: := a:Yo + d,c, where constitute the noise schedule. As
E (0, I) represents the added Gaussian noise, and result, yr will finally reach isotropic Gaussian noise after sufficient destruction steps. By reversing the above process, we are able to perform the generation process by gradually denoise the sample starting from pure Gaussian noise yr ~ (00 I) until reaching yo- Our diffusion model trained to denoise y, into yo for each timestep t, facilitating both unconditional and conditional generation.
5

---

Table 2: Comparison with prior 3D representations of spatial structure, fitting quality, relative fitting speed (Rel. Speed) and parameter sizes on ShapeNet Car. denotes that the implicit feature decoder is shared across different objects. All methods are evaluated at 30K iterations.
Representation
Sputially-structured
LPIPS.
SSIM
Speed
(M))
Instant-NGP
33,98
0.0386
0.9809
1x
12:25
Gaussian Splatling
35.32
0.0303
0.9874
2.58x
1.84
Vosels
0.9664
0.15x
Voxels"
30.25
0.0926
0.9541
0.15x
67.12
Triplanc
32.61
0,0611
0.9709
1.05x
Triplane"
31.39
0,0759
0,9635
1.05x
6.30
Our GaussianCube
34.94
0.0347
0.9863
3.33x
0.46
Ground-truth
Instant-NGP
Gaussian Splatting
Voxel
Triplane"
Our GaussianCube
Figure 4: Qualitative results of object fitting.
Model architecture. Thanks to the spatially structured organization of the proposed GaussianCube. standard 3D convolution is sufficient to effectively extract and aggregate the features of neighboring Gaussians without elaborate designs. We leverage the standard U-Net network for diffusion [41, 17] and simply replace the original 2D operators including convolution, attention, upsampling and downsampling with their 3D counterparts.
Conditioning mechanism. Our model supports variety of condition signals to control the gener- ation process. When performing lass-conditioned diffusion modeling, we employ adaptive group normalization (AdaGN) [17] to inject the class labels into our model. For image-conditioned digital avatar creation, we leverage pretrained vision transformer [6] to encode the conditional image into sequence of feature tokens. We subsequently adopt cross-attention to make the model learn the correspondence between 3D activations and 2D image feature tokens following [5]. We also leverage cross-attention our condition mechanism when creating 3D objects from text, similar to previous text-to-image diffusion models [47].
Training objective. In our 3D diffusion training, we parameterize our model ÿo to predict the noise-free input yo using:
Ssimpe yo.. [leo (axyo or6, L, Gass) yo
where the condition signal Cas only needed when training conditional diffusion models. We addi- tionally impose image-level supervision to improve the rendering quality of generated GaussianCube, which has been demonstrated to effectively enhance the perceptual details in previous works [59, 39]. Specifically, we penalize the discrepancy between the rasterized images I of the model prediction at timestep t and the ground-tre -ruth images Igt using:
Limage = E
Σ V' (Ipeea wi (Lx) + E (Il - Ige 112)
(4)
where v/ is the multi-resolution feature extracted using the pre-trained VGG [50]. Benefiting from the efficiency of both rendering speed and memory costs from GS [30]. we are able to perform fast training with high-resolution renderings. Our overall training loss can formulated as:
L Lsimple XLinage
(5)
where À is balancing weight.

---

Table 3: Quantitative results of unconditional generation ShapeNet Car and Chair [9] and class-
conditioned generation on OmniObject3D [64].
ShapeNet Car
Shape Net Chair
OmniOhject3D
Method
FID-50K,
KID-)0K(%d))
FID-50K,
KID-50K(%o))
FID-50K}
KID-50K(%o)
EG3D
30.48
20.42
27.98
16.01
GET3D
17.15
9.58
19.24
10.95
DiffTF
51.88
41.10
47.08
31.29
46.06
22.86
Ours
13.01
8.46
15.99
9.95
11.62
2.78
ShapoNet
Shape Chair
EG3D [8]
GET3D [19]
DifFTF [5]
Ours
Figure 5: Qualitative comparison of unconditional 3D generation ShapeNet Car and Chair datasets.
Our model is capable of generating results of complex geometry with rich details.
DifTTF [5]
Ours
Figure 6: Qualitative comparison class-conditioned 3D generation on large-oocabulary OmniOb-
ject3D [64]. Our model is able to handle diverse distribution with semantically accurate results.
Experiments
4.1 Dataset and Metries
To measure the expressiveness and efficiency of various 3D representations, we fit 100 objects in
ShapeNet Car [9] using each representation and report the PSNR, LPIPS [75] and Structural Similarity Index Measure (SSIM) metrics when synthesizing novel views. Furthermore, we conduct experiments
of cingle-category unconditional generation on ShapeNet [9] Car and Chair, and cass-condiiioned generation on real-world scanned dataset OmniObject3D [64]. We compute the FID [24] and KID [2] scores between 50K generated renderings and 50K ground-truth renderings. For image-conditioned digital avatar generation, we utilize the synthetic avatar dataset [62], which comprises
3D avatars created by synthetic pipeline. We assess the generation quality of 5K rendering from 500 test avatars and additionally include cosine similarity of identity embedding [15] (CSIM) to measure ID preservation. The experiments of text-to-3D generation are performed on the large-scale challenging Objaverse dataset [14]. We numerically evaluate the text alignment quality using CLIP score [46] of 300 test prompts. All images are rendered with 512 512 resolution. For more details of data, please refer to Appendix A.I.

---

Table 4: Quantitative results of digital avatar creation conditioned on single portrait image.
Method
PSNR
LPIPS
SSIM
CSIM
FID-3K,
KID-5K(%oL,
Rodin w/o 2D SR
18.80
0.2842
0.7439
0.6594
32.07
24.78
Rodin
18.59
0.2821
0.7373
0.6466
20.02
9.24
Ours
21.87
0.1768
0.7703
0.7821
8.32
2.67
Reference
Rodin [59]
Ours
Figure 7: Qualitative comparison of 3D avatar creation conditioned on single frontal portraits.
4.2
Implementation Details
For GaussianCube construction, we set N Nmax to 32,768 and C to 14 across all datasets. We perform the proposed densification-constre constrainndd fitting for 30K iterations, which requires approximately 2.67 minutes on single V100 GPU for each object. After OT-based structuralization, we obtain 32 x 32 x 32 x 14 GaussianCube for each object. The OT-based structuralization takes around 2 minutes per object on an AMD EPYC 7763v CPU. For the 3D diffusion model, we adopt the ADM U-Net network [41, 17J. We perform full attention at the resolution of 83 and 43 within the network. The timesteps of diffusion models are set to 1, 000 and we train the models using the cosine noise schedule [41] with loss weight À set to 10. We deploy 16 Tesla V100 GPUs for the ShapeNet Car, ShapeNet Chair, OmniObject3D, and Synthetic Avatar datasets, whereas 32 Tesla V100 GPUs are used for training on the Objaverse dataset. It takes about one week to train our model on ShapeNet Car, ShapeNet Chair, and OmniObject3D, and approximately two weeks for the Synthetic Avatar and Objaverse datasets. For more training details, please refer to Appendix A.l.
4.3
Main Results
3D fitting, We first evaluate our representation capability of object fitting against previous NeRF- based representations including Voxels [57] and Triplane [8], which are widely adopted in previous 3D generation works [8. 59, 5, 39, 57]. We set the representation size of Voxels and Triplane to 128 x 128 x 128 x 32 and 256 256 32 respectively for comparable fitting quality. We also include Instant-NGP [40] and original Gaussian Splatting [30] for reference despite their unsuitability for generative modeling due their unstructured spatial nature. shown in Table 2, our GaussianCube outperforms all NeRF-based representations among all metrics. Figure 3 illustrates that GaussianCube can faithfully reconstruct geometry details and intricate textures. Moreover, we achieve such high- quality fitting with orders of magnitude fewer parameters than previous structured representation due to the cessificationconnstrine feature decoder in the multi-object fitting of NeRF-based methods leads to significant decreases in quality compared to single-object fitting as evidenced in Table While the fully explicit nature of GS results in quality gap between single and multiple object fitting.
fitting, showcasing our compactness. Notably, the shared implicit
Single-category unconditional generation. For unconditional generation, we compare our method with the state-of-tt sate-of-thee-nt the-art 3D generation works including 3D-aware GANs [8, 19] and Triplane diffusion models [5]. As shown in Table 3, our method surpasses all prior works in terms of both FID and KID scores and sets new records. We provide visual comparisons in Figure 5, where EG3D and DiffTF tend to generate blurry results with poor geometry, and GET3D fails to provide satisfactory textures. In contrast, our method yields high-fidelity results with authentic geometry and sharp textures.
8

---

Table 5: Quantitative results of text-to-3D creation. Inference time is measured single A100 GPU. While Shape-E, LGM achieve comparable CLIP scores as ours, they either utilize millions of training data or leverage 2D diffusion prior.
DrusiGaussian
VolumeDiftusion
Shap-E
LGM
Ours
CLIP Score" Inference Time (s)
26.38
24.41
30.52
30.06
30,56
~ 120
4.95
4.42
1.55
2.30
"a red and white shoz."
"a white ceramic sink with silver faucet.'
DreamGaussian [53] VolumeDDifussio [57] Shap-E [28]
LGM [54]
Ours
Figure 8: Qualitative comparison of text-to-3D generation Objaverse [14]. Our model is able to
generate high-quality samples according to the given text prompts.
Large-vocabulary class-conditioned generation. We also compare class- conditoond generation with DiffTF [5] on more diverse and challenging OmniObject3D [64] dataset. We achieve significantly better FID and KID scores than DiffTF as shown in Table 3. Visual comparisons in Figure 6 reveal that DifTTF often struggles to create intricate geometry and detailed textures, whereas our method is able to generate objects with complex geometry and realistic textures.
Image-conditioned avatar generation. For 3D avatar generation conditioned on single reference image, we compare our method with state-of-the-un Triplane diffusion models, Rodin [47]. Our model surpasses Rodin among all evaluated metrics as shown in Table 4. Although Rodin utilizes a 2D refiner [60] to boost the visual quality of facial areas, which significantly compromises 3D consistency. Our model still outperforms it by direct real-3D generation. Results in Figure/ 7 demonstrate that our model faithfully preserves the identity, expression and accessories of the references with rich details, while Rodin struggles provide satisfactory results even using 2D refinement.
Text-to-3D generation. We compare text-to-3I) generation with prior arts including diffusion models [28, 57]. optimization-base method [53] and ford-forward method [54]. Our model achieves competitive text-3D alignment results as shown in Table 5. The visual comparison in Figure shows that our model is able to create high-quality samples aligning with text prompts in just 2.3 seconds. DreamGaussian tends to create over-saturated results and suffers from Janus problem. VolumeDiffusion produces unsatisfactory textures with poor text alignment. Shap-E can produce semantically accurate results but struggles to generate complex geometry. LGM reconstructs 3D Gaussians from multi view images generated by text-conditioned multi-view diffusion pipeline [48], whereas the inconsistency [54] of the generated multi-views often results in inaccurate geometric reconstruction.
4.4 Ablation Study
We first examine the key factors in representation construction on ShapeNet Car. To spatially structure the Gaussians, simplistic approach would be anchoring the positions of Gaussians to predefined voxel grid while omitting densification and pruning, which leads to severe failure when fitting the objects as shown in Figure 9. Even by introducing learnable offsets to the voxel grid, the results still lack details. We observe the offsets are typically too small to effectively lead the Gaussians close to the object surfaces, which indicates the importance of densification in the fitting process. Instead, GaussianCube can capture both complex geometry and intricate details as shown Figure 9. The numerical comparison in Table 6 also demonstrates the superior fitting quality of GaussianCube.

---

Table 6: Quantitative ablation of both representation fitting and generation quality on ShapeNet Car.
Method
Densify Prune
PSNRT
Representation Fiiting
LPIPSL
SSIMI
FID-50K.
Generation
KID-50K(%0))
A. Voxel grid wio offsct
x
25.87
0.1228
0.9217
B. Vaxel grid w/ offset
x
30.18
0.0780
0.9628
40.52
24.35
C. Ours wio OT
/
34,94
0.0346
0.9863
21.41
14.37
0.0346
0.9863
13.01
8.46
2016
Ground-truth
Table
Table 6 B.
Table D. (Ours)
Figure 9: Qualitative ablation of representation fitting.
OT (Ours)
Nearest Neighbor
Table G.
Table C
Table D. (Ours)
(b)
Figure 10: Visual ablation of the Gaussian organization methods and 3D generation. For visualization of Gaussian structuralization in (a), we map the coordinates of the corresponding voxel of each Gaussians to RGB values visualize the organization. Our OT-based solution also results in the best generation quality shown in (b).
We also evaluate how the representation affects 3D generative modeling ShapeNet Car as shown in Table 6 and Figure 10. Limited by the poor fitting quality, performing diffusion modeling on voxel grid with learnable off'sts leads to blurry generation results as shown in Figure 10. To validate the importance of organizing Gaussians via Optimal Transport (OT), we compare with the organization based on nearest neighbor transport. We linearly map each Gaussian's corresponding coordinates of voxel to RGB color to visualize different organizations. As shown in Figure 10 (a), our proposed OT approach yields smooth color transitions, indicating that our method successfully preserves the spatial correspondence. However, nearest neighbor results in abrupt color transitions due to their disregard for global structure. Both the quantitative results in Table 6 and visual comparisons Figure 10 indicate that our globally structured arrangement facilitates generative modeling by alleviating its complexity. successfully leading to superior generation quality.
Conclusion
We have presented GaussianCube, a structured and explicit radiance representation crafted for 3D generative models. We begin by fitting each 3D object with a constant number of Gaussians using our proposed cussffiattion-costtainnee fitting algorithm. We further organize the obtained Gaussians into a spatially structured representation by solving the Optimal Transport between the positions of Gaussians and the predefined voxel grid. The proposed GaussianCube is spatially structured, allowing to use standard 3D U-Net for diffusion modeling without elaborate designs. Moreover, Gaussian achieve high-quality fitting using much fewer parameters compared prior works of similar quality, which further eases the difficultyy of generative modeling. Our 3D diffusion models equipped with GaussianCube achieve state-of-the-arr generation quality on the evaluated datasets, underscoring its potential of GaussianCube as versatile and powerful radiance representation for 3D generation.
Acknowledgments: This work was supported in part by the Anhui Provincial Natural Science Foundation under Grant 2 2108085UD12. We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC. We also thank anonymous reviewers for their valuable comments.
10

---

References
[1] Jonathan Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[2] Mikolaj Binkkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans, arXiv preprint arXiv:1801.01400 2018.
[3] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVT Conference on Computer Vision and Puttern Recognition, pages 22563 225755, 2023.
[4] Rainer E Burkard and Eranda Cela. Linear assignment problems and extensions. In Handbook of combinatorial optimization: Supplement volume A, pages 75-149. Springer, 1999.
[5] Ziang Cao, Fangzhou Hong. Tong Wu, Liang Pan, and Ziwei Liu. Large-vocabulary 3d diffusion model with transformer. arXiv preprint arXiv:2309. 07920, 2023.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self seff-superviee vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[7] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5799-5800 2021.
[8] Eric R Chan, Connor Z. Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry- aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123-16133, 2022.
[9] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 152. 2015.
[10] Guikun Chen and Wenguan Wang. A survey on 3d gaussian splatting, arXiv preprint arXiv:240..33890. 2024.
[11) Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction. arXiv preprint arXiv:3344.06714, 2023.
[12] Yiji Cheng, Fei Yin. Xiaoke Huang, Xintong Yu, Jiaxiang Liu, Shikun Feng, Yujiu Yang, and Yansong Tang. Efficient text-guided 3d-aware portrait generation with score distillation sampling on distribution. arXiv preprint arXiv:2306.02088 2023.
[13] R James Cotton and Colleen Peyton. Dynamic gaussian splatting from markerless motion capture reconstruct infants movements. In Proceedings of the IEEE/VVF Winter Conference on Applications of Computer Vision, pages 60-68, 2024.
[14] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Osear Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 131 13142-13153, 2023.
[15] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690 4699, 2019.
[16] Yu Deng, Jiaolong Yang. Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds for 3d-aware image generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10673-10683, 2022.

---

[17] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 87994, 2021.
[18] Sara Fridovich- Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501-55101 2022.
[19] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. arXiv preprint arXiv:2209. 1166,, 2022.
[20] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1):139914444 2020.
[21] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXIv:2110.08985, 2021.
[22] Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.05377 003377 2023.
[23] Xianglong He, Junyi Chen, Sida Peng, Di Huang, Yangguang Li, Xiaoshui Huang, Chun Yuan, Wanli Ouyang, and Tong He. Gvgen: Text-to-3d generation with volumetric representation. arXiv preprint arXiv:2403.12957, 2024.
[24] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
[25] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 (1880066511,, 2020.
[26] Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang. Lin Gao, Xiao Liu, and Yuewen Ma. Tri- miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19774-19783, 2023.
[27] Roy Jonker and Ton Volgenant. A shortest augmenting path algorithm for dense and sparse linear assignment problems. In DGOR/NSOR: Papers of the 16th Annual Meeting of DGOR in Cooperation with NSOR/Vorträge der 16. Jahrestagung der DGOR zusammen mit der NSOR, pages 622-622. Springer, 1988.
[28] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint 2rX.v:2305.022633 2023.
[29] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[30] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
[31] Mengtian Li, Shengxiang Yao, Zhifeng Xie, Keyu Chen, and Yu-Gang Jiang. Gaussianbody: Clothed human reconstruction via 3d gaussian splatting. arXiv preprint arXiv:2401.09722 2024.
[32] Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski. Generative image dynamics. arXiv preprint arXiv:2309.07906, 2023.
[33] llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, ICLR, 2019.
[34] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
12

---

[35] Guanxing Lu, Shiyi Zhang, Ziwei Wang, Changliu Liu, Jiwen Lu, and Yansong Tang. Mani- gaussian: Dynamic gaussian splatting for multi-task robotic manipulation. arXiv preprint arXiv:2403.0832/, 2024.
[36] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan, Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint urXiv:2308.0977 13, 2023.
[37] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108. 01073, 2021.
[38] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106 2021.
[39] Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner. Diffrf: Rendering-guiide 3d radiance field diffusion. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4328-4338, 2023.
[40] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4); 1-15, 2022.
[41] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 116-8171. PMLR, 2021.
[42] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1453 1 1464, 2021.
[43] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[44] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using
2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
[45] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 8748 8763.773..76 PMLR, 2021.
[47] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[48] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi- view diffusion for 3d generation. arXiv preprint arXiv:2308 16512, 2023.
[49] J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[50] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
[51] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 5459-5469 2022.
13

---

[52] Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. arXiv preprint arXiv:2310. 16818, 2023.
[53] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprini arXiv:2309. 16653, 2023.
[54] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint 2024.
[55] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22819-22829, 2023.
[56] Junshu Tang. Yanhong Zeng, Ke Fan, Xuheng Wang. Bo Dai, Kai Chen, and Lizhuang Ma. Make-it-vivid: Dressing your animatable biped cartoon characters from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6243-6253,
2024.
[57] Zhicong Tang, Shuyang Gu, Chunyu Wang, Ting Zhang, Jianmin Bao, Dong Chen, and Baining Guo. Volumediffusion: Flexible text-to-3d generation with efficient volumetric encoder. arXiv preprint arXiv:2312.11455, 2023.
[58] Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2009.
(59) Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen. Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4563-4573, 2023.
[60] Xintao Wang, Yu Li, Honglun Zhang. and Ying Shan. Towards real-world blind face restoration with generative facial prior. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9168-9178, 2021.
[61] Zhenwei Wang, Tengfei Wang. Zexin He, Gerhard Hancke, Ziwei Liu, and Rynson WH Lau. Phidias: A generative model for creating 3d content from text, image. and 3d conditions with reference-augmented diffusion. arXiv preprint arXiv:2409. / 2024.
[62] Erroll Wood, Tadas Baltrušaitis, Charlic Hewitt, Sebastian Dziadzio, Thomas J Cashman, and Jamie Shotton. Fake till you make it: face analysis in the wild using synthetic data alone. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3681-3691, 2021.
[63] Guanjun WU, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXi::2310.08522 2023.
[64] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, et al. Omniobject3d: Lorge-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 803-814, 2023.
[65] Weihao Xia and Jing-Hao Xue. A survey on deep generative 3d-aware image synthesis. ACM Computing Surveys, 56(4):1 2023.
[66] Jianfeng Xiang, Jiaolong Yang, Yu Deng, and Xin Tong. Gram-hd: 3d-consistent image genera- tion high resolution with generative radiance manifolds. arXiv preprint arXiv:200.007255 2022.
[67] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: sero-shot text-to-3d synthesis using shape prior and text-to-image diffusion models. arXiv preprint arXh:2211..14704 2022.
14

---

[68] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2022.
[69] Yuelang Xu, Benwang Chen, Zhe Li, Hongwen Zhang, Lizhen Wang, Zerong Zheng, and Yebin Liu. Gaussian head avatar: Ultra high-fidelity head avatar via dynamic gaussians. arXiv preprint arXiv:2312.03002 2023.
[70] Taoran Yi, Jiemin Fang. Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamerr Fast generation from text to 3d gaussian splatting with point cloud priors. arXiv preprint arXiv:2310.08529, 2023.
[71] Youyi Zhan, Tianjia Shao, He Wang, Yin Yang, and Kun Zhou. Interactive rendering of relightable and animatable gaussian avatars. arXiv preprint arXiv:240.117077, 2024.
[72] Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo. Styleswin: Transformer- gan for high-resolution image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11304-11314, 2022.
[73] Bowen Zhang, Yiji Cheng, Chunyu Wang, Ting Zhang, Jiaolong Yang. Yansong Tang, Feng Zhao, Dong Chen, and Baining Guo. Rodinhd: High-fidelity 3d avatar generation with diffusion models. arXiv preprint arXiv:2407.069.: 2024.
[74] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arX1v:2010.07492,2000 2020.
[75] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018.
[76] Junsheng Zhou, Weiqi Zhang, and Yu-Shen Liu. Diffgs: Functional gaussian splatting diffusion. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[77] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826-5835, 2021.
15

---

A Appendix
A.1 Additional Implementation Details
Dataset preparation. We conduct experiments on ShapeNet Car [9]. ShapeNet Chair [9], OmniOb- ject3D [64]. Synthetic Avatar [62] and Objaverse [14] datasets. For each dataset, we report the total number of objects used for training, the number of views rendered per object for GaussianCube fitting and the distribution of camera poses used for rendering in Table 7. For the Objaverse dataset, excluded low-quality objects, such as those without textures or with defective reconstructions follow- ing [57]. We also report the object bounding box b in the world coordinate system of each dataset in Table 7, which is used to construct the predefined voxel grid within [-b, b)s during OT-based Gaussian structuralization.
Representation construction. We set Nmx to 32768 and C to 14 omitting the view-dependent spherical harmonics. This simplification appears to have a negligible impact object fitting while concurrently reducing the data dimension, thereby alleviating the difficulty of diffusion modeling. During densification-constee
fitting procedure, primarily follow the dyyer--prameee original Gaussian Splatting [30]. For OT-based Gaussian structuralization, we adopt an approximate solution for the OT problem due to the O (N Amxx) time complexity of Jonker- Volgenant algorithm [27]. This is achieved by dividing the positions of the Gaussians and the voxel grid into four sorted segments and then applying the Jonker- Volgenant solver each segment individually. We empirically find this approximation successfully strikes a balance between computational efficiency and spatial structure preservation. The proposed censification-conttrane object of 30K iterations and the OT bT-based voxelization takes around 2 minutes which can be run on CPU in parallel.
in
fitting takes around 2.67 minutes for each
3D Diffusion, To train the 3D diffusion model, we initially compute the instance-wist statistics of mean f I N,N,N, N. N. xC and standard deviation o R N. N. N,,CC' from the GaussianCubes of each training dataset respectively. These statistical measures are then utilized to normalize the training data. For our 3D diffusion model architecture, we adopt the ADM-UNet from [17] and replace the convolution, upsampling, downsampling and attention operations with 3D implementations. We train our model using AdamW optimizer [33], and apply exponential moving average (EMA) with a rate of 0.9999 during training. We clamp the prediction of opacity a to [0, 1) and clamp the minimum value of predicted scaling s train the model with a base learning rate 5e 5 for 850K iterations and then decay the learning for another 150K iterations. For 3D digital avatar creation from a single portrait image, we adopt the pretrained DINO ViT-B/16 [6] to encode the 512 x 512 conditional images into 1025 x 768 conditional feature tokens. For text-to-3D creation, we take CLIP-L/14 [46] to encode the text prompts into 77 x 768 conditional feature tokens. We provide more detailed configurations of the model architectures, diffusion training and inference for each dataset in Table
ensure validity. For unconditional generation on ShapeNet, we
rate to 5e
Implementation of Gaussian organization visualization in Figure 10 (a). For the i-th Gaussian, obtain its corresponding voxel grid centers Ty E R according to Optimal Transport plan T (i.e., ik 1) as illustrated in Section 3.1. To visualize the coordinates of 2k, we map them to RGB color C E R33 using:
(mgg b) 2b
C2
X 255,
(6)
where b is the bounding box in the world coordinate system. The resultant point cloud like vi- sualizations are shown in Figure 10 (a), where smooth color transitions indicate coherent spatial correspondence preservation.
A.2 Additional Ablation Study and Analysis
fitting, We conduct experiments to evaluate how Nmax affects fitting on ShapeNet Car. The results Table 9 indicate that there clear trend where increasing Nmm leads to improved fitting accuracy. However, larger Nmux also incurs higher computational costs during diffusion training. Therefore, we set Nmax to 32,768 to strike a balance between high-quality fitting and computational efficiency.
Ablation of Nmax in densificationn-onstrraiie
16

---

Table 7: Details of each dataset.
Dataset
# #Ojjects
# Views per object
Rotation Angle
Elevation Angle
Bounding Box
ShapeNet Car
7,462
150
0, 2m
1 n 5
0.45
ShapeNet Chair
6,775
150
[0, 2m
0.35
OmniObject3D
5,795
100
[0, 2m
0, 2
1.0
Synthetic Avatar
98,000
300
00, 2m
[ TT 0, 0,
40.0
Objaverse
125,653
1.50
00,2m
0.5
Table 8: Detailed configuration of model architecture, diffusion training and inference each dataset.
ShapeNet Car
ShapeNee Car
OmniObject3D Synthetic Avatar
Objaverse
Diffusion Steps
1,000
1,000
1,000
1,000)
1,000
Noise Schedule
Cosine
Cosine
Cosine
Cosine
Cosine
NFEs
300
300
300
250
44
Inference Time (s)
10.06
10.06
10.06
13.80
2.30
Inference Sampler
DPM-solver [34] DPM-solver [34] DPM-solver [34] DPM-solver [34]
DPM-solver [34]
DPM-solver Order
3
3
3
2
2
DPM-solver Mode
Multi-step
Multi-step
Multi-step
Multi-step 1.3
Adaptive
CFG Scale
2.0
3.5
Model Size
82M
82M
82M
339M
339M
Channels
64
64
64
128
128
Channel Mult.
(1,2,3,4)
(1,2,3,4)
(1,2,3,4)
(1,2,3,4) 3
(1,2,3,4)
Num. Res. Blocks
3
3
3
3
Attn Resolutions
(8, 4)
(8.4)
(8, 4)
4)
4)
Num. Head Channels
64
64
64
64
Dropout
0
0
0
O
0
Scale Shift Norm
True
True
True
Truc
True
Training Steps
1,000K
1,000K
700K
1,200K
1,800K
Training GPUs
16
16
16
32
Batch Size
128
128
128
128
256
Base Lr
5e -
5e - 5
5e -
5e - 5e - 5
Lr Decay Steps
850K
850K
Ablation of classifier-free guidance class-conditioned generation. We study how classifier-fres free guidance (CFG) impacts our generation quality when inference class-conditioned diffusion models. We report the FID and KID metries in Table 10 under different CFG scales.
Visualization of intermediate results in the denoising process. During inference, our model starts from Gaussian noise and progressively denoises yield the high-quality GaussianCube. We present visualizations of the intermediate renderings at various timesteps E [0, 7' throughout the denoising process. offering a detailed insight into the GaussianCube diffusion procedure. As illustrated in Figure 11, our model first establishes the global structure and then incrementally enhances the details, which similar previous 3D diffusion models [59, 49].
Nearest neighbors analysis. We perform nearest neighbor search of some unconditionally generated samples in the paper according to the similarity of pretrained CLIP [46] features. The results in Figure 12 demonstrate that our model is capable of generating novel geometry and textures rather than simply memorizing the training data.
visualize the offset distribution of IK randomly selected GaussianCubes from each experimental dataset in Figure 14. We observe that most distributions exhibit a bell curve, similar to a normal distribution. However, the Digital Synthetic Avatar dataset presents a more uniform distribution with multiple peaks. We believe these distributions offer valuable insights into how well the fitted 3D Gaussians align with voxel grid centers. Bell-shaped distributions akin to normal distribution, such as in the ShapeNet Car and Chair datasets, suggest a strong initial alignment and lower complexity. On the other hand, broader distributions (e.g., the Digital Synthetic Avatar dataset) indicate higher level of detail (for instance, hair) and a greater need for adjustments during organization.
Distribution visualization of offset from voxel grids of fitted GaussianCubes.
A.3 Additional Visual Results
For 3D avatar generation, while trained on synthetic dataset, our model capable of generalizing to in-the-wild portrait input. We provide more visual comparison of 3D avatar creation conditioned on in-
17

---

Table 9: Quantitative ablation of Nmx in densification--oosttaine fitting. We set Mmxx to 32,768 in
this paper.
Nmax
N.
PSNRT
LPIPS
SSIM
4096
16
32.56
0.0547
0,9765
13824
24
34.32
0.0396
0.9842
32768
32
34.94
0.0347
0.9863
110592
48
35.29
0.0307
0.9874
262144
64
35.34
0.0301
0,9875
Table 10:
Quantitative ablation of CFG scale in the class-conditioned generation of OmniOb-
ject3D [64].
Scale
w/o CFG
1.3
1.5
2.0
3.0
6.0
FID-50K
13.39
12.07
11.72
11.62
12.99
32.80
KID-50K(%o)
4.01
3.12
3.00
2.78
3.17
14.36
the-wild portraits with Rodin [59] in Figure 15. We also include additional comparison conditioned on synthetic imput from our test in Figure 16. Our model can faithfully retain the identity of the reference portrait and is able to provide high-fidelityy results with rich details, e.g. hair, glasses and clothing. Although utilizing pretrained 2D super-resolution module which significantly compromises 31) consistency, Rodin struggles to follow the conditional images and fails to produce detailed textures in non-facial areas e.g. clothing and hair.
include additional qualitative comparison and generated samples of text-to-3D generation Fig- ure 17 and Figure 18 respectively. Our model yields samples with better visual quality, and is capable of handling challenging prompts. The results in Figure 19 show the generation diversity of our results given the same text prompt. Our model is also capable of performing rext-guided editing of generated objects by leveraging SDEdit [37] as depicted Figure 20, demonstrating the promise of achieving controllable 3D generation.
We provide more generated samples of unconditional and coadsicond generation in Fig- ure 21, Figure 22 and Figure 23. The additional results demonstrate the strong capability of our model create high-quality 3D assets with complex geometry and intricate textures.
Furthermore, we also provide an additional video in supplementary material, which intuitively illustrates our approach and visualizes the generated results.
A.4 Limitations
While GaussianCube represents substantial step forward in developing an ideal representation for 3D content generation, still has some limitations. Specifically, although the GaussianCube construction procedure is considerably more rapid than that of NeRF-basee methods and can be executed in parallel, it still requires approximately minutes construct each object. This presents challenge for scaling up training on extensive 3D datasets. In future work, we plan to investigate more time-efficient methods for GaussianCube construction. Additionally, akin to prior 2D diffusion models, our text-to-3I diffusion model encounters difficulties in presenting the specified number of objects within prompts as shown in Figure 13. To address this, we will look into enhancing the precision and controllability of 3D generation in the future.
A.5 Broader Impacts
The proposed GaussianCube enables high-quality 3D asset fitting with few parameters, which sig- nificantly simplifies the challenges of 3D generative modeling. Our diffusion model is capable of generating high-quality 3D assets complex geometry and intricate textures while also accom- modating variety of conditional signals to steer the creating procedure. The strong capability of GaussianCube suggests its potential to serve as a versatile 3D representation for a variety of applications in future 3D research endeavors.
Like all generative models, particular caution is required when dealing with sensitive tasks involving human representations. Our avatar creation model is trained exclusively on synthetic dataset [62] composed of large-scale 3D digital avatars which are generated through graphics pipeline. We conceptualize digital avatars analogous those created by specialized 3D artists, rather than
18

---

photorealistic human images. This strategy in selecting training data mitigates privacy and copyright issues that might arise from utilizing real human photo collections. Nevertheless, it is crucial to acknowledge that avatars generated by our model from real-world imagery could still be misused for spreading disinformation. As such, we advocate implementing rigorous safeguards and promoting responsible use of our technology other related ones to mitigate such risks.
Figure 11: Visualization generation results intermediate diffusion timesteps.
Generated
Top-3 nearest neighbors
Figure 12: Visualization nearest neighbor search on ShapeNet Car and Chair.
"a wooden shelf with three shelves."
"a white chandelier with six lights."
Condition
Generated Sample
Text Condition
Generated Sample
Figure 13: Failure cases.
Figure 14: Distribution of offsets from voxel centers random selection of 1K GaussianCubes on each experimental dataset.
19

---

Reference
Rodin [59]
Ours
Figure 15: Additional qualitative comparison of 3D) avatars creation conditioned single in-the-wild portraits.
Reference
Rodin [59]
Ours
Figure 16: Qualitative comparison generated digital avatars conditioned on synthetic portraits.
20

---

"a yellww tes kettie."
"o sunglesses with blue lenses."
white toble with choirs cround
DreanGaussian [53]
VolumnDittusion [57]
Shap-Ei E [28]
LGM [54]
Ours
Figure 17: Additional qualitative comparison of text-to-3D generation Objaverse [14]. Our model
capable creating high-quality samples following input text prompts.
"a blue and yellow fish with eye."
free stump."
"a red plastic tooth model,"
ned heart."
white cartoon character of Sonic the Hedgehag.
cartoon hamburger with lettuce and cheese."
"a wooden benck."
"a coctus in pot."
Figure 18: Additional results of text-to-3D generation.
"a purple chair with curved back and curved seat."
"a wooden barrel with metal band around the middle."
Text Condition
Sample 1
Sample 2
Sample 3
Figure 19; Variation of text-to-3D generation. Our model is able to generate diverse results condi- tioned on the same text prompt.
Object
Objects
ta ned pickup truck"
"o green pickup truck"
"a burnt and rusted pickup truck"
"a pickup truck with colorful point job.'
Figure 20: Example of text lext-guided 3D editing.
21

---

Figure 21: Additional generated samples on ShapeNet Car.
22

---

Figure 22: Additional generated samples on ShapeNet Chair.
23

---

Figure 23: Additional generated samples on OmniObject3D.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include the discussion of limitations in Appendix A.4.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
this paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided comprehensive implementation details in the main paper and supplementary materials to ensure that our method can be faithfully reproduced. We also release our code and model checkpoints at http::/github..cm/ caaalaub.om/Gaaaaa Conmi/GassiaaCuub/aaaaaatCuuu
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
We have released the source code and model checkpoints at http:///b..mmmuussaaaauuu///aaae Guidelines:
Justification:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
ceeiiissssoooooo ceeiionoolcyy olicyyyy)))lo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ htc/public/ggutiieeeeeeeeeeeeeee http//gudeee/eeeee Ceaees/CooSSubmieec cessinnoolccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include comprehensive training and test details in Appendix A.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results are accompanied by statistical significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include sufficient information on the computer resources needed to repro- duce the experiments in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tte::////uurrree cc wttppuuiice.iiiccccciieeeeeleeee ines?
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Appendix A.5,
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We also include the safeguards in Appendix A.5.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators original owners of assets are properly credited in this paper. The license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "bD4xpwitha", "reviewer": "Reviewer_hQzy", "summary": "The paper proposes a new structured and explicit representation based on 3D Gaussian Splatting for 3D generation. The key idea is to properly organize the 3D Gaussians into a fixed-size volume, allowing for the use of the standard 3D UNet for diffusion. First, the paper uses a densification-constrained fitting algorithm to constrain the number of 3D Gaussians to be no more than a fixed value. Then, the paper proposes to use Optimal Transport to obtain a better spatial structure for the 3D Gaussians. The proposed representation can be easily used for both unconditional and conditional 3D generation. Experiments show that the proposed method outperforms baseline approaches.", "questions": "- The ShapeNet dataset is axis-aligned, but Objaverse is not. Objects in the Objaverse dataset might have different orientations. Is the method still able to generate meaningful results when the dataset size of Objaverse is small (e.g., only using ~5k cars from Objaverse)?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n\nThe paper proposes a new method to encode 3D Gaussians into a structured and explicit voxel grid, which is novel. The key novelty differs from the concurrent work [21]. While [21] also uses a voxel grid to organize 3D Gaussians, this paper proposes using Optimal Transport to make learning easier. This might inspire future research.\n\nQuality:\n\nThe paper is technically sound. The author provides detailed implementation details.\n\nClarity:\n\nThe paper is mostly clear and easy to read.\n\nSignificance:\n\nApplying diffusion models to 3D Gaussians for 3D generation is an important research problem, and the paper provides one feasible solution to this problem. This problem is non-trivial due to the irregularity of 3D Gaussians. The paper directly addresses this issue and proposes using OT for better spatial organization. Overall, I think this paper might inspire future research.", "weaknesses": "There are some concerns that need to be clarified:\n\n- In Table 2, how is the number of the Triplane parameters computed? Why is it larger than Voxels? In literature, people use Triplane because it is more efficient than voxels. In addition, L11-L13: when encoding a large dataset, do Triplane/voxels (with shared implicit decoder) still have more parameters than GaussianCube?\n\n- Is Equation (4) used to train the diffusion model? How do you obtain I_pred? (Do you run multiple denoising steps to generate it?) How does the expectation depend on t here?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions. We address the reviewer's concerns below:\n\n> Q: Parameters of Triplane.\n\nA: In Table 2 of the main paper, we assigned the size of the Triplane as $3\\times256\\times256\\times32$. We set the size of Voxels to $32\\times32\\times32\\times14$ with the intent to draw a comparison with GaussianCube of a similar representation size. We additionally compare with Triplane of size $3\\times128\\times128\\times10$, which has similar representation size with GaussianCube and Voxels of $32\\times32\\times32\\times14$. Furthermore, we include results from Voxels of size $128\\times128\\times128\\times32$, which yield much higher fitting quality than the $32\\times32\\times32\\times14$ counterpart and exhibit a comparable fitting quality to both GaussianCube and Triplane. The additional results are included in the attached PDF of the top-level comment. As indicated in Table 1 of the attached PDF, the Triplane is considerably more efficient than the Voxels, offering superior fitting quality than Voxels with larger parameters. However, our GaussianCube still surpasses Triplane and Voxels in terms of fitting quality using the fewest parameters.\n\n> Q: When encoding a large dataset, do triplane or voxels (with shared implicit decoder) still have more parameters than GaussianCube?\n\nA: Yes. Due to the GaussianCube's explicit nature, it eliminates the requirement for a shared implicit decoder across various objects, leading to no significant difference in fitting quality when fitting larger datasets. Our representation size remains consistent at $32\\times32\\times32\\times14$ across all datasets. We additionally conducted an experiment involving the fitting of the Objaverse dataset, which is not only larger but also exhibits a much more diverse distribution compared to ShapeNet. The experiment can be found in the attached PDF of the top-level comment, where we report the fitting quality of 100 randomly selected objects. As demonstrated in Table 2 of the attached PDF, our GaussianCube still achieves superior fitting quality while utilizing the minimum number of parameters compared with Voxels and Triplane. \n\n> Q: Is Equation (4) used to train the diffusion model?\n\nA: Yes, Equation (4) is used for diffusion training.\n\n> Q: How to obtain $I\\_{\\text{pred}}$?\n\nA: Our model is parameterized to predict the noise-free input $\\mathbf{y}\\_0$. For each training step, let $\\hat{\\mathbf{y}\\_0} = \\hat{\\mathbf{y}\\_\\theta}\\left(\\alpha_t \\mathbf{y}\\_0+\\sigma_t \\mathbf{\\epsilon}, t,  \\mathbf{c}\\_{\\text{cls}}\\right)$ be the prediction of our model. We can directly rasterize $\\hat{\\mathbf{y}\\_0}$ to obtain $I\\_{\\text{pred}}$ without multiple denoising steps during diffusion training.\n\n> Q: How does the expectation depend on $t$ in Equation (4)?\n\nA: Given the model's prediction $\\hat{\\mathbf{y}\\_0}$ and the camera parameters $\\textbf{\\textit{E}}$ used for rasteraization, we obtain $I\\_{\\text{pred}}$ by\n$$\n\\begin{equation}\n\\begin{array}{ll}\n    I\\_{\\text{pred}} &= \\text{Rasterize}(\\hat{\\mathbf{y}\\_0}, \\textbf{\\textit{E}}) \\\\\\\\\n    &= \\text{Rasterize}(\\hat{\\mathbf{y}\\_\\theta}\\left(\\alpha\\_t \\mathbf{y}\\_0+\\sigma\\_t \\mathbf{\\epsilon}, t,  \\mathbf{c}\\_{\\text{cls}}\\right), \\textbf{\\textit{E}}).\n\\end{array}\n\\end{equation}\n$$\n\nTherefore, Equation (4) can be written as\n$$\n\\begin{equation}\n\\begin{array}{ll}\n    \\mathcal{L}\\_{\\text {image }} &= \\mathbb{E}\\_{t, I\\_{\\text{pred }}}\\left(\\sum\\_l\\left\\\\|\\Psi^l\\left(I\\_{\\text{pred}}\\right)-\\Psi^l\\left(I\\_{\\text{gt}}\\right)\\right\\\\|\\_2^2\\right) +\\mathbb{E}\\_{t, I\\_{\\text {pred}}}\\left(\\left\\\\|I\\_{\\text {pred}}-I\\_{\\text {gt }}\\right\\\\|\\_2\\right), \\\\\\\\\n    &= \\mathbb{E}\\_{t, \\mathbf{y}\\_0, \\mathbf{\\epsilon}}\\left(\\sum\\_l\\left\\\\|\\Psi^l\\left(\\text{Rasterize}(\\hat{\\mathbf{y}\\_\\theta}\\left(\\alpha\\_t \\mathbf{y}\\_0+\\sigma\\_t \\mathbf{\\epsilon}, t,  \\mathbf{c}\\_{\\text{cls}}\\right), \\textbf{\\textit{E}})\\right)-\\Psi^l\\left(I\\_{\\text {gt}}\\right)\\right\\\\|\\_2^2\\right) \\\\\\\\\n    &+\\mathbb{E}\\_{t, \\mathbf{y}\\_0, \\mathbf{\\epsilon}}\\left(\\left\\\\|\\text{Rasterize}(\\hat{\\mathbf{y}\\_\\theta}\\left(\\alpha\\_t \\mathbf{y}\\_0+\\sigma\\_t \\mathbf{\\epsilon}, t,  \\mathbf{c}\\_{\\text{cls}}\\right), \\textbf{\\textit{E}})-I\\_{\\text {gt }}\\right\\\\|\\_2\\right),\n\\end{array}\n\\end{equation}\n$$\n\nwhich illustrates that the expectation of Equation (4) depends on $t$. We thank the reviewers for this question and will further revise Equation (4) in the revision to improve its clarity.\n\n> Q: Is our method able to generate meaningful results on a small dataset containing objects of different orientations?\n\nA: That is an interesting question. Our method does not require the objects to be axis-aligned and has demonstrated strong capability to model complex distribution (e.g., Objaverse). Consequently, we believe our model is capable of providing meaningful results with a small dataset comprising objects of varying orientations. However, it is still challenging to fully represent the complete data manifolds using only a small amount of data samples. If the dataset size is excessively small, the model may be exposed to the risk of having limited extrapolation capabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e0HboFeIYY", "reviewer": "Reviewer_E9Nw", "summary": "The paper proposes an approach to 3D generation at the object level with the help of diffusion models. The main challenge in using diffusion model to generate 3D is the choice of 3D representation that fits well with the denoising network. The paper proposes using fixed number of Gaussians lying on a regular 3D grid. Each cell of the 3D grid stores the parameters of the Gaussians along with an offset from the cell. However, the Gaussians trained using Gaussians Splatting can lie on arbitrary coordinates. In order to assign a Gaussian to the cell of the grid, the paper proposes using Optimal Transport algorithm. This datastructure is called GaussianCube and result in shape being represented with smaller number of parameters.\n\nBased on this GaussianCube data structure, we can train a 3D diffusion model to generate 3D shapes via Gaussians. This network can also be used to predict Gaussians in conditional manner e.g. image, text etc.\n\nThe resultant generative model produces 3D consistent representation via Gaussian splatting, follows conditioning signal accurately, fast inference (5 sec) and lower number of parameters.", "questions": "1. In my opinion, Figure 3 is not required. The textual explanation is enough.\n2. What are the failure cases of optimal transport? Have authors considered a learnable scattering of gaussians to cubic grid? (Authors should discuss this in the rebuttal.)", "limitations": "Limitations and societal impacts are described in the Supplementary material.\n\nThough the scope of the paper is 3D asset generation and experiments are done accordingly, I would have appreciated some discussion on large scale scene generation also.\n\nI suspect that there are failure cases of optimal transport. If so, they should be described.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a novel approach to consistent 3D generation with the help of diffusion model. The formulation of 3D dataset structure GaussianCubes is interesting as it allows 3D convolution on an unstructured data i.e. set of Gaussians.\n\nThe paper is mostly well written and describes the proposed approach well.\n\nExperiments are performed on relevant dataset and evaulated against valid baselines.\n\nOverall it is a good work showcasing how to combine diffusion model with Gaussian splatting in an end to end training resulting in better performance across several benchmarks.", "weaknesses": "1. Despite good results, my main concern is in the scalability of this approach. Given enough compute, we will be limited by the amount of 3D data. This is one of the reason of many efforts in 3D generative models based on images or videos only, as they are in abundance. I am skeptical can be used to generate assets for real applications. (Authors rebutting this should focus on how data scalability be achieved?)\n2. How expressive and generalizable is the model in term of fitting many datasets (shapenet, objaverse etc) jointly? (Authors should not do experiments if they have not already tried training on all datasets.)\n3. I find it quite surprising that the GaussianCube data structure works well even when trying to fit shapes with different scales and dimension. I expect that this representation will be hard to scale for complicated topology and intricate details where number of gaussians required to approximate the shape will be quite large. (Authors should discuss it in the rebuttal.)\n4. Missing training time and compute requirement for training diffusion model. (Authors should provide this data in the paper.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions. We address the reviewer's concerns below:\n\n> Q: Given that many 3D generative models are based on images or videos due to the abundance of 2D data, how data scalability can be achieved for our approach?\n\nA: We acknowledge the reviewer's point that data holds significant importance in scaling up models. Despite that some 3D generation works are based on images and videos (e.g., MVDream, SV3D), a substantial amount of 3D data is still necessary for training in order to generate 3D consistent results. Therefore, we believe large-scale, high-quality 3D data is important for both our approach and the 2D-based methods. Recent trends indicate an active research interest in the development of expansive 3D datasets, such as Objaverse-XL[1], which boasts over 10 million 3D objects. The availability of such vast volumes of 3D data paves the way for us to scale up our model. Nevertheless, our approach is not limited to using 3D data only. We are also able to harness 2D image or video priors to further refine our generated results (e.g. leveraging Score Distillation Sampling with a pretrained 2D diffusion model).\n\n> Q: Expressiveness and generalization capability of our model to fit many datasets jointly.\n\nA: Regarding data fitting, the explicit nature of our representation eliminates the need for the shared implicit feature decoder and enables us to achieve high-quality fitting across all datasets. The fitting experiments in Table 1 and Table 2 of the attached PDF of the top-level comment demonstrate the expressiveness and generalization capability when jointly fitting different datasets (fitting results of GaussianCube are done before paper submission). Regarding modeling the distribution of GaussianCube using diffusion models, Objaverse serves as an exemplary case. This large-scale dataset with intricate data distribution encompasses most categories in ShapeNet. Given that our model yields high-quality generation results on Objaverse, we believe our method is effective and generalizable to handle complex datasets jointly.\n\n> Q: How to scale the representation for complicated topology and intricate details?\n\nA: GaussianCube is a highly expressive representation as we have demonstrated. A modest number of Gaussians (e.g., $32^3=32,768$) can effectively capture complex objects with high fidelity (e.g., the tire tread and hair of avatars in Figure 4 and Figure 7 of the main paper). Furthermore, we evaluate our method on the more challenging NeRF Synthetic dataset[2]. Our GaussianCube can approach the fitting quality of original GS using only $32,768$ Gaussians. For more complicated topology like Mic, our method achieves on-par quality by enlarging the corresponding voxel size and number of Gaussians (e.g., to $48^3=110,592$) without extra elaborate design to our pipeline. Please see the attached PDF in our top-level comment for visual comparison.\n\n| Number of Gaussians | Mic | Drums | Ficus | Ship  | Avg. |\n| :-------: | :------: | :-------: | :-------: |:-------: | :------: |\n| 32,768    | 33.25 | 25.76 | 34.55 | 29.90 |  30.87 |\n| 110,592   | 35.32 | 26.08 | 34.78 | 30.97 |  31.79 |\n| Original GS | 35.36 | 26.15 | 34.87 | 30.80 | 31.80 |\n\n> Q: Training time and compute requirement for diffusion model training.\n\nA: All of our diffusion model is trained using Nvidia Tesla V100 32G GPUs. It takes about one week to train our model on ShapeNet Car, ShapeNet Chair, and OmniObject3D, and approximately two weeks for the Synthetic Avatar datasets using 16 GPUs. For Objaverse dataset, the training duration extends to around two weeks with 32 GPUs. We will add the training time and compute information in the revised paper.\n\n> Q: Necessity of Figure 3.\n\nA: Thanks for your suggestion. We will consider revising it in the revision.\n\n> Q: Failure cases of optimal transport.\n\nA: We utilize the Jonker-Volgenant algorithm to resolve the optimal transport problem, which invariably yields the optimal solution. Consequently, we don't identify any failure cases of optimal transport in our model. If possible, we would appreciate further elaboration from the reviewer on what constitutes a failure case in the context of optimal transport. This would facilitate a more productive discussion during the reviewer-author dialogue period.\n\n> Q: Have we considered a learnable scattering of Gaussians to a cubic grid?\n\nA: That is indeed a compelling proposition. One potential advantage of a learnable scattering mechanism is the elimination of the time cost of OT. However, integrating it into the current pipeline presents several challenges. For example, the scattering process would need to be meticulously designed to ensure it is both differentiable and bijective. Additionally, jointly training the scattering with the diffusion model could potentially lead to instability issues. We will leave it to future research and thank the reviewer for the suggestion.\n\n> Q: Discussion of large-scale scene generation.\n\nA: Scene generation is an interesting topic. While the experiments conducted with GaussianCube primarily focus on objects, the generality of our pipeline suggests the potential applicability to scene generation as well. Nonetheless, scene generation comes with its own set of challenges. For one, handling unbounded scenes would require specialized design (e.g., contracting unbounded space into a ball in MipNeRF-360), given that GaussianCube necessitates a voxel grid within the bounding box. Additionally, the scarcity of large-scale scene data poses significant challenges for generative modeling. We plan to delve into these aspects in our future research.\n\n[1] Deitke M, Liu R, Wallingford M, et al. Objaverse-xl: A universe of 10m+ 3d objects[J]. Advances in Neural Information Processing Systems, 2024, 36.\n\n[2] Mildenhall B, Srinivasan P P, Tancik M, et al. Nerf: Representing scenes as neural radiance fields for view synthesis[J]. Communications of the ACM, 2021, 65(1): 99-106.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZEw4dfWbAI", "reviewer": "Reviewer_Z6fD", "summary": "The paper is about 3d object generative model. The generative objects are represented with gaussian splatting. Thus the main obstacle is the large point clouds. It is difficult to generate such large point clouds. The authors proposed a way to put point cloud to a structured grid using optimal transport. Some generative experiments are shown.", "questions": "The digital avatar experiment is image-conditioned generation? Is it possible to train an unconditional generative model?\n\nI do not have further questions for this paper.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of using optimal transport here is creative. And there is a motivation behind this.\n\nThe figures are also nice to illustrate the idea.\n\nThere are some kinds of experiments to verify the idea, including, objects and human avatars.\n\nGenerally I like the proposed idea and the convincing results shown in the main paper (and the video).", "weaknesses": "I would like to see some time and memory analysis about the method, including data preprocessing (object fitting and optimal transport), training resources, and sampling speed. These are important for readers to understand how difficult it is to reimplement the method.\n\nAnother thing might be useful is novelty analysis. Shapenet is a relatively small dataset, which can be easily memorized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions. We address the reviewer's concerns below:\n\n> Q: Time and memory analysis of our method.\n\nA: Thanks for your suggestions. As detailed in the supplementary material (Lines 501-503), the proposed densification-constrained fitting requires approximately 2.67 minutes on a single V100 GPU for each object over 30K iterations. The OT-based structuralization takes around 2 minutes per object on an AMD EPYC 7763v CPU. Fitting a single object consumes around 1GB of GPU memory. OT is run on CPU and costs around 5.4GB memory. For the diffusion training, we deploy 16 Tesla V100 GPUs for the ShapeNet Car, ShapeNet Chair, OmniObject3D, and Synthetic Avatar datasets, whereas 32 Tesla V100 GPUs are used for training on the Objaverse dataset. It takes about one week to train our model on ShapeNet Car, ShapeNet Chair, and OmniObject3D, and approximately two weeks for the Synthetic Avatar and Objaverse datasets. We provide the detailed inference configuration of DPM-solver and speed under various inference timesteps in the table below. Please note that all the inference times of the diffusion models in Table 5 of the main paper are reported using 100 steps to ensure a fair comparison.\n\n| Model size | Orders of Solver | Inference Mode | Inference Timesteps | Time (s) |\n|:------------:|:--------:|:-------------:|:-----------------:|:----------:|\n| 82M        | 3      | Multi-steps | 300             | 10.06    |\n|            | 3      | Multi-steps | 200             | 6.57     |\n|            | 3      | Multi-steps | 100             | 3.27     |\n| 339M       | 2      | Multi-steps | 300             | 15.54    |\n|            | 2      | Multi-steps | 250             | 13.80    |\n|            | 2      | Multi-steps | 200             | 10.64    |\n|            | 2      | Multi-steps | 100             | 5.13     |\n|            | 2      | Adaptive    | 44              | 2.30     |\n\n> Q: Novelty analysis of generation results on ShapeNet.\n\nA: In our experiments, we perform nearest neighbor search of our generated samples on ShapeNet dataset, as depicted in Figure 12 of the main paper. The results demonstrate that our model is capable of generating novel geometry and textures rather than simply memorizing the training data.\n\n> Q: Is the digital avatar creation model conditioned on image?\n\nA: Yes. The digital avatar creation model is conditioned on a single portrait image.\n\n> Q: Is it possible to train an unconditional digital avatar creation model?\n\nA: Certainly. We can train an unconditional model on a digital avatar dataset using the same methodology as with the ShapeNet datasets. In fact, our image-conditioned digital avatar creation diffusion model utilizes classifier-free guidance, allowing us to conduct unconditional inference simply by dropping the image conditions. Please refer to Figure 2 of the attached PDF in the top-level comment for our unconditionally generated digital avatars.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YAm1OIzWzb", "reviewer": "Reviewer_u6Hp", "summary": "The paper introduces a novel explicit and structured 3D representation - GaussianCube - used in combination with U-Net diffusion models for 3D generation. While 3D Gaussians have a lot of advantages, they are not directly compatible with efficient architectures for generative modeling because of their unstructured nature. This paper tackles this problem by proposing two steps from 3D Gaussians to GaussianCube: 1) densification-constrained fitting to bound the number of Gaussians, and 2) structuralization via optimal transport to convert the underlying unstructured point cloud of 3D Gaussians to a structured voxel grid. \n\nUsing datasets in this format, the authors train diffusion models parameterized as 3D U-Nets for object, avatar, and text-to-3D generation.", "questions": "* Why do you use $x_0$-parameterization instead of $\\epsilon$ or v?\n* The resolution 32 is rather small and the appendix shows stronger fitting performance for larger numbers of Gaussians. Is there potential for latent diffusion?\n* Why is the reported performance of DiffTF worse than the one from the original paper?", "limitations": "The authors addressed limitations and broader (including potential negative societal) impact in the appendix.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper introduces a simple but effective idea of bringing structure into unstructured 3D Gaussians.\n  - This small change to the representation makes it directly compatible with standard architectures for diffusion models.\n  - The use of optimal transport for voxelization is novel and elegant.\n  - The representation might have potential to enable more efficient / optimized network architectures in other 3D tasks as well.\n* The paper is presented well:\n  - The main approach is clear.\n  - The paper is well-written and easy to follow.\n* The paper provides an extensive experimental evaluation validating their approach:\n  - Generation with GaussianCube is evaluated for 4 tasks on synthetic as well as real data: unconditional generation on ShapeNet, class-conditional generation on OmniObject3D, image-conditioned avatar generation on Synthetic Avatar, and text-to-3D generation on Objaverse.\n  - Quantitative and qualitative results show strong performance compared to baselines.\n  - The supplementary material includes a video with more convincing qualitative results.\n  - The ablation study validates the design choices.", "weaknesses": "* Lack of clarity:\n  - The paper does not include any details about the diffusion process on the properties of 3D Gaussians (except of normalization with dataset statistics in the appendix):\n    - 3D Gaussians have parameters with very different distributions making the diffusion process on them non-trivial.\n    - As one example, if rotations are represented as quaternions with a diffusion process adding simply Gaussian noise, there is no guarantee about the final generation being a valid rotation.\n    - Also the distribution of offsets from voxel centers would be insightful, especially across different objects/scenes with different level of detail.\n  - The fact that positions are replaced by offsets of the voxel centers became clear rather late in the paper (ll. 153 ff.) and should be explained earlier.\n  - The optimal transport illustration in Fig. 3 is not intuitive. Are there only two valid pixel corners on the diagonal?\n\n* Missing related work and baseline:\n  - The related work about diffusion models on 3D representations is missing [1] and [2].\n  - The state-of-the-art comparison on ShapeNet misses an important baseline [1] with strong performance.\n\n[1] Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction, ICCV 2023\n[2] Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation, CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions. We address the reviewer's concerns below:\n\n> Q: Parameter distribution of 3D Gaussians used for diffusion process.\n\nA: Although the parameters of 3D Gaussians have very different distributions, we observe that after applying data normalization, they mainly obey the normal distribution, i.e., $\\mathcal{N}(\\mathbf{0}, \\boldsymbol{I})$. Consequently, the data distribution does not present any significant divergence from that employed in the conventional diffusion process.\n\n> Q: How to guarantee the predicted properties are valid?\n\nA: We employ quaternion representations for rotations and normalize the predicted rotations to procure a valid unit quaternion following original Gaussian Splatting. We clamp the prediction of opacity $\\alpha$ to $[0, 1)$ and clamp the minimum value of predicted scaling $\\mathbf{s}$ to $0$ to ensure validity. No additional operations are applied to the positional and color feature predictions. Our findings suggest that these processes effectively validate the predicted Gaussians, yielding satisfactorily rendered images. We will incorporate these specifics into the revised manuscript.\n\n> Q: The distribution of the offsets from voxel centers.\n\nA: Thank you for your insightful suggestion. Please see Figure 3 in the attached PDF in the top-level comment. We visualize the offset distribution of 1K randomly selected GaussianCubes from each experimental dataset in Figure 3. We observe that most distributions exhibit a bell curve, similar to a normal distribution. However, the Digital Avatar dataset presents a more uniform distribution with multiple peaks. We believe these distributions offer valuable insights into how well the fitted 3D Gaussians align with voxel grid centers. Bell-shaped distributions akin to a normal distribution, such as in the ShapeNet Car and Chair datasets, suggest a strong initial alignment and lower complexity. On the other hand, broader distributions (e.g., the Digital Avatar dataset) indicate a higher level of detail (for instance, hair) and a greater need for adjustments during organization.\n\n> Q: The positions of Gaussians are replaced by offsets of voxel centers should be explained earlier.\n\nA: Thanks for your valuable comments. We will explain this operation earlier in the revised paper.\n\n> Q: Illustration of Fig. 3 of the main paper.\n\nA: We apologize for any confusion caused by the optimal transport illustration in Figure 3 of the main paper, where we provided a 2D toy example of two pixels. Actually, each pixel on the grid is valid rather than just two pixel corners on the diagonal. We appreciate your insightful suggestion and will revise this figure accordingly in the updated version.\n\n> Q: Missing related works of [1] and [2].\n\nA: Thanks for pointing out this. We will discuss these works in our revision.\n\n> Q: Missing comparison with [1] on ShapeNet.\n\nA: We have extended our analysis to include a comparison with SSDNeRF[1] on ShapeNet Car. By evaluating the official model checkpoint of SSDNeRF, we obtained the FID and KID scores between 50K generated renderings and 50K ground-truth renderings, aligning with the data in Table 3 of the main paper. The quantitative evaluation indicates that our method outperforms SSDNeRF in terms of both the FID and KID scores. The visual comparison in Figure 4 of the attached PDF in the top-level comment demonstrates that our model is able to generate high-fidelity results, surpassing SSDNeRF on the level of detail.\n\n| Method | FID-50K↓ | KID-50K(‰)↓ |\n| :--------- | :-----------: | :-------: |\n| SSDNeRF[1] | 23.10 | 11.68 |\n| Ours | **13.01** | **8.46** |\n\n> Q: Why do we use $x\\_0$-parameterization instead of $\\epsilon$ or $v$?\n\nA: Previous work [3] demonstrates that $x\\_0$, $\\epsilon$ and $v$ predictions only result in varied loss weights in diffusion training. We investigate both of them during earlier exploration and find $x\\_0$-parameterization achieves the best performance. Therefore, we employ $x\\_0$-parameterization in all subsequent experiments.\n\n> Q: Potential for latent diffusion?\n\nA: That's an insightful observation. Existing 2D latent diffusion models successfully achieve scale-up and exhibit amazing generative results. As such, 3D latent diffusion may potentially enhance the scalability of our model. However, this would necessitate the additional training of a Variational Autoencoder (VAE) that needs to be carefully designed to maintain the quality of reconstruction. We plan to explore this aspect further in our future research.\n\n> Q: Performance of DiffTF.\n\nA: We infer the official checkpoints of DiffTF and report the FID score between 50K generated and 50K ground-truth renderings rather than 50K generated and all ground-truth renderings used in the original paper. All reported FID and KID in Table 3 of the main paper are evaluated between 50K generated and 50K GT.\n\n[1] Chen H, Gu J, Chen A, et al. Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2023: 2416-2425.\n\n[2] Schröppel P, Wewer C, Lenssen J E, et al. Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 8785-8794.\n\n[3] Salimans T, Ho J. Progressive distillation for fast sampling of diffusion models[J]. arXiv preprint arXiv:2202.00512, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We express our sincere gratitude to all reviewers for their valuable feedback, which has immensely contributed to the enhancement of our paper. We are greatly encouraged by the reviewers' acknowledgment that our paper:
- propose a novel method to address the irregularity of 3D Gaussians for 3D generation (Reviewer hQzy, Reviewer E9Nw), which might inspire future research (Reviewer hQzy).
- is creative and elegant to use Optimal Transport for Gaussian structuralization (Reviewer Z6fD, Reviewer u6Hp).
- is mostly well-written and clear and provides convincing results with strong performance (Reviewer hQzy, Reviewer E9Nw, Reviewer Z6fD, Reviewer u6Hp).

We have individually addressed each reviewer's concerns and also provided a one-page PDF incorporating additional tables and figures. If our responses adequately address your concerns, we would be grateful if you considered increasing your score. For any further questions, we are available for more detailed discussions.

In the attached one-page PDF, we additionally provide:
- Fitting comparison with Voxels of size $32\times32\times32\times14, 128\times128\times128\times32$ and Triplane of size $3\times128\times128\times10, 3\times256\times256\times32$ on ShapeNet Car (Table 1) and Objaverse (Table 2).
- Fitting results on NeRF Synthetic dataset using different numbers of Gaussians (Figure 1).
- Unconditionally generated samples of digital avatars by our approach (Figure 2).
- Distributions of offsets from voxel centers on each experimental dataset (Figure 3).
- Visual comparison with SSDNeRF on ShapeNet Car (Figure 4).

Meta-review: The paper proposes a diffusion model for 3D Gaussians which is non-trivial due to the irregularity of 3D Gaussians. To use a standard 3D UNet for diffusion, the 3D Gaussians are confined into a fixed-size volume, and further optimized using Optimal Transport. The method outperforms the relevant baselines.

All reviewers are positive about the paper and the authors addressed some remaining concerns in their rebuttal. They point out that the presentation is good, the method appears technically sound and performs well. A shared concern is the scalability of the method, due to the need for 3D training data which the authors address in their rebuttal to some extent by reporting results for jointly fitting multiple datasets.

Overall, this is a solid paper and I recommend acceptance.
Decision: Accept (poster)