GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling

Authors: bowen  zhang
yiji  cheng
jiaolong  yang
chunyu  wang
feng  zhao
yansong  tang
dong  chen
baining  guo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 12:00:19

Abstract: We introduce a radiance representation that is both structured and fully explicit and thus greatly facilitates 3D generative modeling. Existing radiance representations either require an implicit feature decoder, which significantly degrades the modeling power of the representation, or are spatially unstructured, making them difficult to integrate with mainstream 3D diffusion methods. We derive GaussianCube by first using a novel densification-constrained Gaussian fitting algorithm, which yields high-accuracy fitting using a fixed number of free Gaussians, and then rearranging these Gaussians into a predefined voxel grid via Optimal Transport. Since GaussianCube is a structured grid representation, it allows us to use standard 3D U-Net as our backbone in diffusion modeling without elaborate designs. More importantly, the high-accuracy fitting of the Gaussians allows us to achieve a high-quality representation with orders of magnitude fewer parameters than previous structured representations for comparable quality, ranging from one to two orders of magnitude. The compactness of GaussianCube greatly eases the difficulty of 3D generative modeling. Extensive experiments conducted on unconditional and class-conditioned object generation, digital avatar creation, and text-to-3D synthesis all show that our model achieves state-of-the-art generation results both qualitatively and quantitatively, underscoring the potential of GaussianCube as a highly accurate and versatile radiance representation for 3D generative modeling.

Main paper: 
 introduction: 
 The field of 3D generation [59, 39, 5, 57, 49, 8, 19, 11, 61, 76] has witnessed remarkable growth, driven by advancements in generative modeling [25, 20, 41, 17, 72, 29] . Most of the prior works in this domain leverage variants of Neural Radiance Field (NeRF) [38, 8, 57, 40] as their underlying 3D representations, which typically consist of an explicit structured proxy representation and an implicit feature decoder. However, such hybrid NeRF variants exhibit degraded representation power, particularly when used for generative modeling where a single implicit feature decoder is shared across all objects. Additionally, the high computational complexity of volumetric rendering leads to both slow rendering speed and extensive memory costs.Recently, the emergence of 3D Gaussian Splatting (GS) [30] has enabled improved reconstruction quality and real-time rendering capabilities [69, 36, 63, 35] . The fully explicit nature of 3DGS eliminates the need for a shared implicit decoder, providing another key advantage over NeRFs. Although 3DGS has been widely studied in scene reconstruction tasks, its spatially unstructured nature presents a significant challenge when applied to mainstream generative modeling frameworks.Fully-explicit Real-time Rendering Rel. Parameters↓ Instant-NGP [38] ✗ ✗ ✗ 26.63× Neural Voxels [57] ✓ ✗ ✗ 145.9× Triplane [8] ✓ ✗ ✗ 13.7× Gaussian Splatting [30] ✗ ✓ ✓ 4.0× Our GaussianCube ✓ ✓ ✓ 1.0×Table 1 : Comparison with previous 3D representations with respect to spatial structure, explicitness, real-time rendering capability, and relative parameter count (Rel. Parameters) for representations of comparable quality.To overcome these barriers, we introduce GaussianCube -an innovative radiance representation that is both structured and fully explicit, with strong fitting capabilities (see Table 1 for comparisons with prior works). The proposed approach first ensures high-accuracy fitting with a predefined number of free Gaussians, and subsequently organizes these Gaussians into a structured voxel grid. Such an explicit grid-based structure permits the seamless application of standard 3D convolutional architectures, such as U-Net, thereby eliminating the need for complex, specialized network designs [77, 59] that are often necessary with unstructured or implicitly decoded representations.Structuring 3D Gaussians without sacrificing fitting quality is not a trivial task. A naive starting point would be obtaining a fixed number of Gaussians by omitting the densification and pruning steps in GS. However, such simplification fails to lead the Gaussians close to the object surfaces and results in significant quality degradation. In contrast, we propose a densification-constrained fitting strategy, which retains the original pruning process yet constrains the number of Gaussians that perform densification, ensuring the total does not exceed a predefined maximum N 3 . For the subsequent structuralization, we allocate the Gaussians across an N × N × N voxel grid using Optimal Transport (OT). Consequently, our fitted Gaussians are systematically arranged within the voxel grid, with each voxel containing the features of a Gaussian. The proposed OT-based structuralization achieves maximal spatial correspondence, characterized by minimal total transport distances, while preserving the expressive power of 3DGS.The structured nature of GaussianCube enables us to perform efficient 3D diffusion [25] modeling for the following three reasons: 1) It allows the use of standard 3D U-Net as our backbone for diffusion modeling without elaborate designs.2) The spatial coherence of GaussianCube permits the use of standard 3D convolutions to capture the correlations among neighboring Gaussians, facilitating efficient feature extraction. 3) GaussianCube enables high-quality fitting with orders of magnitude fewer parameters than prior structured representations of similar quality. Since recent works [32, 3] have demonstrated diffusion models' struggle in handling high-dimensional distributions, the compactness of GaussianCube significantly reduces the modeling difficulty of the generative framework.We conduct comprehensive experiments to verify the efficacy of our approach. The model's capability for unconditional and class-conditioned generation is evaluated on the ShapeNet [9] and OmniObject3D [64] datasets. Both the quantitative and qualitative comparisons indicate that our model surpasses all previous methods. We also perform digital avatar generation on a synthetic avatar dataset [62] . Our model is capable of producing high-fidelity 3D avatars conditioned on single portrait images, excelling beyond prior art in both identity preservation and detail creation. Additionally, we assess our model's capacity for the challenging text-to-3D creation task on Objaverse [14] . Our model demonstrates competitive performance both quantitatively and qualitatively, producing results consistent with the given text prompts in just 2.3 seconds. All experiments show the strong capabilities of our GaussianCube and suggest its potential as a powerful and versatile 3D representation for a variety of applications. Some generated samples of our method are presented in Figure 1 . 
related work: 
 Radiance field representation. Radiance fields model ray interactions with scene surfaces and can be in either implicit or explicit forms. Early works of neural radiance fields (NeRFs) [38, 74, 43, 1, 45] are often in an implicit form, which represents scenes without defining geometry. These works optimize a continuous scene representation using volumetric ray-marching that leads to extremely high computational costs. Recent works introduce the use of explicit proxy representation [8, 26, 18, 51, 40, 68] followed by an implicit feature decoder to enable faster rendering. Recently, the 3D Gaussian Splatting methods [30, 69, 63, 13, 31, 10, 71] utilize 3D Gaussians as their underlying representation and offer impressive reconstruction quality. The fully explicit representation also provides real-time rendering speed. However, the 3D Gaussians are unstructured representation, and Figure 1 : Our diffusion model is able to create diverse objects with complex geometry and rich texture details (top three rows). Our method also supports creating high-fidelity digital avatars (the forth row) conditioned on single portrait images (visualized in dashed boxes) and high-quality 3D assets given text prompts (the fifth row).require per-scene optimization to achieve photo-realistic quality. In contrast, our work proposes a structured representation termed GaussianCube for 3D generative tasks.3D generation. Previous works of SDS-based optimization [44, 55, 67, 52, 12, 53, 70, 56] distill 2D diffusion priors [47] to a 3D representation with the score functions, but these works are notably time-intensive, often taking several minutes to hours. While 3D-aware GANs [8, 19, 7, 21, 42, 16, 66] facilitate view-dependent image generation from single-image collections, they struggle to capture the complexity of diverse objects with intricate geometric variations [65] . Although recent works [59, 39, 22, 57, 49, 73] have utilized diffusion models with structured proxy representations for 3D generation, the use of a shared implicit feature decoder across different assets restricts expressiveness and the computational demands of NeRF hinder efficient training. In contrast, we introduce a structured and fully explicit radiance representation for 3D generative modeling, building upon 3DGS [30] . A concurrent work of [23] includes elaborate designs to form the Gaussians into volumetric representation during fitting, yet does not thoroughly address global correspondence. In contrast, our approach only restricts the total count of Gaussians while allowing freedom in their spatial distribution during the fitting. We then organize these Gaussians into a voxel grid using Optimal Transport, which yields a spatially coherent arrangement with minimal global offset cost, effectively easing the difficulty of generative modeling. 
method: 
 Following prior works, our framework comprises two primary stages as shown in Figure 2 : representation construction and diffusion modeling. In representation construction phase, we first apply a densification-constrained 3DGS fitting algorithm for each object to obtain a constant number of Gaussians. These Gaussians are then organized into the proposed spatially structured GaussianCube via Optimal Transport between the positions of Gaussians and centers of a predefined voxel grid. For diffusion modeling, we train a 3D diffusion model to learn the distribution of GaussianCubes. We will detail our designs for each stage subsequently.We build our GaussianCube upon 3DGS, an explicit representation that offers impressive fitting quality and real-time rendering speed. However, it fails to yield Gaussians of fixed length since the adaptive density control during GS fitting can lead to a varying number of Gaussians for different objects. Furthermore, the lack of a predetermined spatial ordering for Gaussians leads to a disorganized spatial structure. These aspects pose significant challenges to 3D generative modeling. To overcome these obstacles, we first introduce our densification-constrained fitting strategy to obtain a fixed number of free Gaussians. Then, we systematically arrange the resulting Gaussians within a predefined voxel grid via Optimal Transport, thereby achieving a structured and explicit radiance representation.Formally, a 3D asset is represented by a collection of 3D Gaussians as introduced in Gaussian Splatting [30] . The geometry of the i-th 3D Gaussian g i is given byEQUATIONwhere µ i ∈ R 3 is the center of the Gaussian and Σ i ∈ R 3×3 is the covariance matrix defining the shape and size, which can be decomposed into a quaternion q i ∈ R 4 and a vector s i ∈ R 3 for rotation and scaling, respectively. Moreover, each Gaussian g i have an opacity value α i ∈ R and a color feature c i ∈ R 3 for rendering. Combining them together, the C-channel feature vectorθ i = {µ i , s i , q i , α i , c i } ∈ R C fully characterizes the Gaussian g i .Densification-constrained fitting. Our approach begins with the aim of maintaining a constant number of Gaussians g ∈ R Nmax×C across different objects during the fitting. A simplistic approach might involve omitting the densification and pruning in the original GS. However, we argue that such simplifications significantly harm the fitting quality, with empirical evidence shown in Table 6 . Instead, we propose to retain the pruning process while imposing a new constraint on the densification phase as shown in Figure 3 (a). The fitting process encompasses several distinct stages: 1) Densification Detection: Assuming the current iteration includes N c Gaussians, we identify densification candidates by selecting those with view-space position gradient magnitudes exceeding a predefined threshold τ . We denote the number of candidates as N d . 2) Candidate sampling: To prevent exceeding the predefined maximum of N max Gaussians, we select min (N max -N c , N d ) Gaussians with the largest view-space positional gradients from the candidates for densification. 3) Densification: We modify the densification approach by alternating between cloning and splitting actions into separate steps. 4) Pruning Detection and Pruning: We identify and remove the Gaussians with α less than a small threshold ϵ. After completing the fitting process, we pad Gaussians with α = 0 to reach the target count of N max without affecting the rendering results. Benefiting from our proposed strategy, we attain a high-quality representation with orders of magnitude fewer parameters compared to existing works of similar quality, which significantly reduces the modeling difficulty for the diffusion models.Gaussian structuralization via Optimal Transport. To further organize the obtained Gaussians into a spatially structured representation for 3D generative modeling, we propose to map the Gaussians to a predefined structured voxel grid v ∈ R Nv×Nv×Nv×C where N v = 3 √ N max . Intuitively, we aim to "move" each Gaussian into a voxel while preserving their geometric relations as much as possible. While naive approaches such as nearest neighbor transport fall short in conserving these relations due to disregard for global arrangement with evidence shown in Figure 10 , we formulate this as an Optimal Transport (OT) problem [58, 4] between the Gaussians' spatial positions {µ i , i = 1, . . . , N max } and the voxel grid centers {x j , j = 1, . . . , N max }. Let D be a distance matrix with D ij being the moving distance between µ i and x j , i.e., D ij = ∥µ i -x j ∥ 2 . The transport plan is represented by a binary matrix T ∈ R Nmax×Nmax , and the optimal transport plan is given by:minimize T Nmax i=1 Nmax j=1 T ij D ij subject to Nmax j=1 T ij = 1 ∀i ∈ {1, . . . , N max } Nmax i=1 T ij = 1 ∀j ∈ {1, . . . , N max } T ij ∈ {0, 1} ∀(i, j) ∈ {1, . . . , N max } × {1, . . . , N max }.(EQUATIONThe solution is a bijective transport plan T * that minimizes the total transport distances. We employ the Jonker-Volgenant algorithm [27] to solve the OT problem. We provide a 2D illustration in Figure 3 (b). We organize the Gaussians according to the solutions, with the j-th voxel encapsulating the feature vector of the corresponding Gaussianθ k = {µ k -x j , s k , q k , α k , c k } ∈ R C ,where k is determined by the optimal transport plan (i.e., T * kj = 1). Note that we replace the original Gaussian positions with offsets of the current voxel center to reduce the solution space for diffusion models. As a result, our fitted Gaussians are systematically arranged within a voxel grid v and preserve the spatial correspondence of neighboring Gaussians, which further facilitates generative modeling.We now introduce our 3D diffusion model incorporated with the proposed expressive, efficient and spatially structured representation. After organizing the fitted Gaussians g into GaussianCube y for each object, we aim to model the distribution of GaussianCube, i.e., p(y).Formally, the generation procedure can be formulated into the inversion of a discrete-time Markov forward process. During the forward phase, we gradually add noise to y 0 ∼ p(y) and obtain a sequence of increasingly noisy samples {y t |t ∈ [0, T ]} according to y t := α t y 0 + σ t ϵ, where ϵ ∈ N (0, I) represents the added Gaussian noise, and α t , σ t constitute the noise schedule. As a result, y T will finally reach isotropic Gaussian noise after sufficient destruction steps. By reversing the above process, we are able to perform the generation process by gradually denoise the sample starting from pure Gaussian noise y T ∼ N (0, I) until reaching y 0 . Our diffusion model is trained to denoise y t into y 0 for each timestep t, facilitating both unconditional and conditional generation. Model architecture. Thanks to the spatially structured organization of the proposed GaussianCube, standard 3D convolution is sufficient to effectively extract and aggregate the features of neighboring Gaussians without elaborate designs. We leverage the standard U-Net network for diffusion [41, 17] and simply replace the original 2D operators including convolution, attention, upsampling and downsampling with their 3D counterparts.Conditioning mechanism. Our model supports a variety of condition signals to control the generation process. When performing class-conditioned diffusion modeling, we employ adaptive group normalization (AdaGN) [17] to inject the class labels into our model. For image-conditioned digital avatar creation, we leverage a pretrained vision transformer [6] to encode the conditional image into a sequence of feature tokens. We subsequently adopt cross-attention to make the model learn the correspondence between 3D activations and 2D image feature tokens following [5] . We also leverage cross-attention as our condition mechanism when creating 3D objects from text, similar to previous text-to-image diffusion models [47] .Training objective. In our 3D diffusion training, we parameterize our model ŷθ to predict the noise-free input y 0 using:EQUATIONwhere the condition signal c cls is only needed when training conditional diffusion models. We additionally impose image-level supervision to improve the rendering quality of generated GaussianCube, which has been demonstrated to effectively enhance the perceptual details in previous works [59, 39] . Specifically, we penalize the discrepancy between the rasterized images I t pred of the model prediction at timestep t and the ground-truth images I gt using:EQUATIONwhere Ψ l is the multi-resolution feature extracted using the pre-trained VGG [50] . Benefiting from the efficiency of both rendering speed and memory costs from GS [30] , we are able to perform fast training with high-resolution renderings. Our overall training loss can be formulated as:EQUATIONwhere λ is a balancing weight. Our model is capable of generating results of complex geometry with rich details.DiffTF [5] Ours 4 Experiments 
dataset and metrics: 
 To measure the expressiveness and efficiency of various 3D representations, we fit 100 objects in ShapeNet Car [9] using each representation and report the PSNR, LPIPS [75] and Structural Similarity Index Measure (SSIM) metrics when synthesizing novel views. Furthermore, we conduct experiments of single-category unconditional generation on ShapeNet [9] Car and Chair, and class-conditioned generation on real-world scanned dataset OmniObject3D [64] . We compute the FID [24] and KID [2] scores between 50K generated renderings and 50K ground-truth renderings. For image-conditioned digital avatar generation, we utilize the synthetic avatar dataset [62] , which comprises highly-detailed 3D avatars created by synthetic pipeline. We assess the generation quality of 5K rendering from 500 test avatars and additionally include cosine similarity of identity embedding [15] (CSIM) to measure the ID preservation. The experiments of text-to-3D generation are performed on the large-scale challenging Objaverse dataset [14] . We numerically evaluate the text alignment quality using CLIP score [46] of 300 test prompts. All images are rendered with 512 × 512 resolution. For more details of data, please refer to Appendix A.1. For GaussianCube construction, we set N max to 32,768 and C to 14 across all datasets. We perform the proposed densification-constrained fitting for 30K iterations, which requires approximately 2.67 minutes on a single V100 GPU for each object. After OT-based structuralization, we obtain 32 × 32 × 32 × 14 GaussianCube for each object. The OT-based structuralization takes around 2 minutes per object on an AMD EPYC 7763v CPU. For the 3D diffusion model, we adopt the ADM U-Net network [41, 17] . We perform full attention at the resolution of 8 3 and 4 3 within the network. The timesteps of diffusion models are set to 1, 000 and we train the models using the cosine noise schedule [41] 3D fitting. We first evaluate our representation capability of object fitting against previous NeRFbased representations including Voxels [57] and Triplane [8] , which are widely adopted in previous 3D generation works [8, 59, 5, 39, 57] . We set the representation size of Voxels and Triplane to 128 × 128 × 128 × 32 and 256 × 256 × 32 respectively for comparable fitting quality. We also include Instant-NGP [40] and original Gaussian Splatting [30] for reference despite their unsuitability for generative modeling due to their unstructured spatial nature. As shown in Table 2 , our GaussianCube outperforms all NeRF-based representations among all metrics. Figure 3 illustrates that GaussianCube can faithfully reconstruct geometry details and intricate textures. Moreover, we achieve such highquality fitting with orders of magnitude fewer parameters than previous structured representation due to the densification-constrained fitting, showcasing our compactness. Notably, the shared implicit feature decoder in the multi-object fitting of NeRF-based methods leads to significant decreases in quality compared to single-object fitting as evidenced in Table 2 . While the fully explicit nature of GS results in no quality gap between single and multiple object fitting.Single-category unconditional generation. For unconditional generation, we compare our method with the state-of-the-art 3D generation works including 3D-aware GANs [8, 19] and Triplane diffusion models [5] . As shown in Table 3 , our method surpasses all prior works in terms of both FID and KID scores and sets new records. We provide visual comparisons in Figure 5 , where EG3D and DiffTF tend to generate blurry results with poor geometry, and GET3D fails to provide satisfactory textures. In contrast, our method yields high-fidelity results with authentic geometry and sharp textures. DreamGaussian [53] VolumeDiffusion [57] Shap-E [28] LGM [54] Ours Large-vocabulary class-conditioned generation. We also compare class-conditioned generation with DiffTF [5] on more diverse and challenging OmniObject3D [64] dataset. We achieve significantly better FID and KID scores than DiffTF as shown in Table 3 . Visual comparisons in Figure 6 reveal that DiffTF often struggles to create intricate geometry and detailed textures, whereas our method is able to generate objects with complex geometry and realistic textures.Image-conditioned avatar generation. For 3D avatar generation conditioned on a single reference image, we compare our method with state-of-the-art Triplane diffusion models, Rodin [47] . Our model surpasses Rodin among all evaluated metrics as shown in Table 4 . Although Rodin utilizes a 2D refiner [60] to boost the visual quality of facial areas, which significantly compromises 3D consistency.Our model still outperforms it by direct real-3D generation. Results in Figure 7 demonstrate that our model faithfully preserves the identity, expression and accessories of the references with rich details, while Rodin struggles to provide satisfactory results even using 2D refinement.Text-to-3D generation. We compare text-to-3D generation with prior arts including diffusion models [28, 57] , optimization-based method [53] and feed-forward method [54] . Our model achieves competitive text-3D alignment results as shown in Table 5 . The visual comparison in Figure 8 shows that our model is able to create high-quality samples aligning with text prompts in just 2.3 seconds. DreamGaussian tends to create over-saturated results and suffers from Janus problem. VolumeDiffusion produces unsatisfactory textures with poor text alignment. Shap-E can produce semantically accurate results but struggles to generate complex geometry.LGM reconstructs 3D Gaussians from multi-view images generated by text-conditioned multi-view diffusion pipeline [48] , whereas the inconsistency [54] of the generated multi-views often results in inaccurate geometric reconstruction.We first examine the key factors in representation construction on ShapeNet Car. To spatially structure the Gaussians, a simplistic approach would be anchoring the positions of Gaussians to a predefined voxel grid while omitting densification and pruning, which leads to severe failure when fitting the objects as shown in Figure 9 . Even by introducing learnable offsets to the voxel grid, the results still lack details. We observe the offsets are typically too small to effectively lead the Gaussians close to the object surfaces, which indicates the importance of densification in the fitting process. Instead, GaussianCube can capture both complex geometry and intricate details as shown in Figure 9 . The numerical comparison in Table 6 also demonstrates the superior fitting quality of GaussianCube. Ground-truth Table 6 A. We also evaluate how the representation affects 3D generative modeling on ShapeNet Car as shown in Table 6 and Figure 10 . Limited by the poor fitting quality, performing diffusion modeling on voxel grid with learnable offsets leads to blurry generation results as shown in Figure 10 . To validate the importance of organizing Gaussians via Optimal Transport (OT), we compare with the organization based on nearest neighbor transport. We linearly map each Gaussian's corresponding coordinates of voxel to RGB color to visualize different organizations. As shown in Figure 10 (a), our proposed OT approach yields smooth color transitions, indicating that our method successfully preserves the spatial correspondence. However, nearest neighbor results in abrupt color transitions due to their disregard for global structure. Both the quantitative results in Table 6 and visual comparisons Figure 10 indicate that our globally structured arrangement facilitates generative modeling by alleviating its complexity, successfully leading to superior generation quality. 
conclusion: 
 We have presented GaussianCube, a structured and explicit radiance representation crafted for 3D generative models. We begin by fitting each 3D object with a constant number of Gaussians using our proposed densification-constrained fitting algorithm. We further organize the obtained Gaussians into a spatially structured representation by solving the Optimal Transport between the positions of Gaussians and the predefined voxel grid. The proposed GaussianCube is spatially structured, allowing to use standard 3D U-Net for diffusion modeling without elaborate designs. Moreover, GaussianCube can achieve high-quality fitting using much fewer parameters compared to prior works of similar quality, which further eases the difficulty of generative modeling. Our 3D diffusion models equipped with GaussianCube achieve state-of-the-art generation quality on the evaluated datasets, underscoring its potential of GaussianCube as a versatile and powerful radiance representation for 3D generation. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We include comprehensive training and test details in Appendix A.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The results are accompanied by statistical significance tests.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We include sufficient information on the computer resources needed to reproduce the experiments in Appendix A.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the broader impacts in Appendix A.5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: We also include the safeguards in Appendix A.5.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets are properly credited in this paper.The license and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Additional Implementation Details Dataset preparation. We conduct experiments on ShapeNet Car [9] , ShapeNet Chair [9] , OmniOb-ject3D [64] , Synthetic Avatar [62] and Objaverse [14] datasets. For each dataset, we report the total number of objects used for training, the number of views rendered per object for GaussianCube fitting and the distribution of camera poses used for rendering in Table 7 . For the Objaverse dataset, we excluded low-quality objects, such as those without textures or with defective reconstructions following [57] . We also report the object bounding box b in the world coordinate system of each dataset in Table 7 , which is used to construct the predefined voxel grid within [-b, b] 3 during OT-based Gaussian structuralization.Representation construction. We set N max to 32768 and C to 14 omitting the view-dependent spherical harmonics. This simplification appears to have a negligible impact on object fitting while concurrently reducing the data dimension, thereby alleviating the difficulty of diffusion modeling.During our densification-constrained fitting procedure, we primarily follow the hyper-parameters in original Gaussian Splatting [30] . For OT-based Gaussian structuralization, we adopt an approximate solution for the OT problem due to the O N 3 max time complexity of Jonker-Volgenant algorithm [27] . This is achieved by dividing the positions of the Gaussians and the voxel grid into four sorted segments and then applying the Jonker-Volgenant solver to each segment individually. We empirically find this approximation successfully strikes a balance between computational efficiency and spatial structure preservation. The proposed densification-constrained fitting takes around 2.67 minutes for each object of 30K iterations and the OT-based voxelization takes around 2 minutes which can be run on CPU in parallel.3D Diffusion. To train the 3D diffusion model, we initially compute the instance-wise statistics of mean μ ∈ R Nv×Nv×Nv×C and standard deviation σ ∈ R Nv×Nv×Nv×C , from the GaussianCubes of each training dataset respectively. These statistical measures are then utilized to normalize the training data. For our 3D diffusion model architecture, we adopt the ADM-UNet from [17] and replace the convolution, upsampling, downsampling and attention operations with 3D implementations. We train our model using AdamW optimizer [33] , and apply exponential moving average (EMA) with a rate of 0.9999 during training. We clamp the prediction of opacity α to [0, 1) and clamp the minimum value of predicted scaling s to 0 to ensure validity. For unconditional generation on ShapeNet, we train the model with a base learning rate 5e -5 for 850K iterations and then decay the learning rate to 5e -6 for another 150K iterations. For 3D digital avatar creation from a single portrait image, we adopt the pretrained DINO ViT-B/16 [6] to encode the 512 × 512 conditional images into 1025 × 768 conditional feature tokens. For text-to-3D creation, we take CLIP-L/14 [46] to encode the text prompts into 77 × 768 conditional feature tokens. We provide more detailed configurations of the model architectures, diffusion training and inference for each dataset in Table 8 . 10 (a) . For the i-th Gaussian, we obtain its corresponding voxel grid centers x k ∈ R 3 according to Optimal Transport plan T * (i.e., T * ik = 1) as illustrated in Section 3.1. To visualize the coordinates of x k , we map them to RGB color C k ∈ R 3 using:EQUATION)where b is the bounding box in the world coordinate system. The resultant point cloud like visualizations are shown in Figure 10 (a) , where smooth color transitions indicate coherent spatial correspondence preservation.Ablation of N max in densification-constrained fitting. We conduct experiments to evaluate how N max affects fitting on ShapeNet Car. The results in Table 9 indicate that there is a clear trend where increasing N max leads to improved fitting accuracy. However, a larger N max also incurs higher computational costs during diffusion training. Therefore, we set N max to 32,768 to strike a balance between high-quality fitting and computational efficiency. the-wild portraits with Rodin [59] in Figure 15 . We also include additional comparison conditioned on synthetic input from our test in Figure 16 . Our model can faithfully retain the identity of the reference portrait and is able to provide high-fidelity results with rich details, e.g. hair, glasses and clothing.Although utilizing a pretrained 2D super-resolution module which significantly compromises 3D consistency, Rodin struggles to follow the conditional images and fails to produce detailed textures in non-facial areas e.g. clothing and hair.We include additional qualitative comparison and generated samples of text-to-3D generation in Figure 17 and Figure 18 respectively. Our model yields samples with better visual quality, and is capable of handling challenging prompts. The results in Figure 19 show the generation diversity of our results given the same text prompt. Our model is also capable of performing text-guided editing of generated objects by leveraging SDEdit [37] as depicted in Figure 20 , demonstrating the promise of achieving controllable 3D generation.We provide more generated samples of unconditional and class-conditioned generation in Figure 21 , Figure 22 and Figure 23 . The additional results demonstrate the strong capability of our model to create high-quality 3D assets with complex geometry and intricate textures.Furthermore, we also provide an additional video in supplementary material, which intuitively illustrates our approach and visualizes the generated results.While GaussianCube represents a substantial step forward in developing an ideal representation for 3D content generation, it still has some limitations. Specifically, although the GaussianCube construction procedure is considerably more rapid than that of NeRF-based methods and can be executed in parallel, it still requires approximately 5 minutes to construct each object. This presents a challenge for scaling up training on extensive 3D datasets. In future work, we plan to investigate more time-efficient methods for GaussianCube construction. Additionally, akin to prior 2D diffusion models, our text-to-3D diffusion model encounters difficulties in presenting the specified number of objects within prompts as shown in Figure 13 . To address this, we will look into enhancing the precision and controllability of 3D generation in the future.The proposed GaussianCube enables high-quality 3D asset fitting with few parameters, which significantly simplifies the challenges of 3D generative modeling. Our diffusion model is capable of generating high-quality 3D assets of complex geometry and intricate textures while also accommodating a variety of conditional signals to steer the creating procedure. The strong capability of GaussianCube suggests its potential to serve as a versatile 3D representation for a variety of applications in future 3D research endeavors.Like all generative models, particular caution is required when dealing with sensitive tasks involving human representations. Our avatar creation model is trained exclusively on a synthetic dataset [62] composed of large-scale 3D digital avatars which are generated through a graphics pipeline. We conceptualize digital avatars as analogous to those created by specialized 3D artists, rather than photorealistic human images. This strategy in selecting training data mitigates privacy and copyright issues that might arise from utilizing real human photo collections. Nevertheless, it is crucial to acknowledge that avatars generated by our model from real-world imagery could still be misused for spreading disinformation. As such, we advocate implementing rigorous safeguards and promoting responsible use of our technology other related ones to mitigate such risks. Reference Rodin [59] Ours • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper proposes a diffusion model for 3D Gaussians which is non-trivial due to the irregularity of 3D Gaussians. To use a standard 3D UNet for diffusion, the 3D Gaussians are confined into a fixed-size volume, and further optimized using Optimal Transport. The method outperforms the relevant baselines.

All reviewers are positive about the paper and the authors addressed some remaining concerns in their rebuttal. They point out that the presentation is good, the method appears technically sound and performs well. A shared concern is the scalability of the method, due to the need for 3D training data which the authors address in their rebuttal to some extent by reporting results for jointly fitting multiple datasets.

Overall, this is a solid paper and I recommend acceptance.

Reviews: review: paper_summary: paper_summary: The paper proposes a new structured and explicit representation based on 3D Gaussian Splatting for 3D generation. The key idea is to properly organize the 3D Gaussians into a fixed-size volume, allowing for the use of the standard 3D UNet for diffusion. First, the paper uses a densification-constrained fitting algorithm to constrain the number of 3D Gaussians to be no more than a fixed value. Then, the paper proposes to use Optimal Transport to obtain a better spatial structure for the 3D Gaussians. The proposed representation can be easily used for both unconditional and conditional 3D generation. Experiments show that the proposed method outperforms baseline approaches.
strength_weakness: strength_weakness: There are some concerns that need to be clarified:

- In Table 2, how is the number of the Triplane parameters computed? Why is it larger than Voxels? In literature, people use Triplane because it is more efficient than voxels. In addition, L11-L13: when encoding a large dataset, do Triplane/voxels (with shared implicit decoder) still have more parameters than GaussianCube?

- Is Equation (4) used to train the diffusion model? How do you obtain I_pred? (Do you run multiple denoising steps to generate it?) How does the expectation depend on t here?
questions: questions: - The ShapeNet dataset is axis-aligned, but Objaverse is not. Objects in the Objaverse dataset might have different orientations. Is the method still able to generate meaningful results when the dataset size of Objaverse is small (e.g., only using ~5k cars from Objaverse)?
limitations: limitations: N/A
paper_summary: paper_summary: The paper proposes an approach to 3D generation at the object level with the help of diffusion models. The main challenge in using diffusion model to generate 3D is the choice of 3D representation that fits well with the denoising network. The paper proposes using fixed number of Gaussians lying on a regular 3D grid. Each cell of the 3D grid stores the parameters of the Gaussians along with an offset from the cell. However, the Gaussians trained using Gaussians Splatting can lie on arbitrary coordinates. In order to assign a Gaussian to the cell of the grid, the paper proposes using Optimal Transport algorithm. This datastructure is called GaussianCube and result in shape being represented with smaller number of parameters.

Based on this GaussianCube data structure, we can train a 3D diffusion model to generate 3D shapes via Gaussians. This network can also be used to predict Gaussians in conditional manner e.g. image, text etc.

The resultant generative model produces 3D consistent representation via Gaussian splatting, follows conditioning signal accurately, fast inference (5 sec) and lower number of parameters.
strength_weakness: strength_weakness: 1. Despite good results, my main concern is in the scalability of this approach. Given enough compute, we will be limited by the amount of 3D data. This is one of the reason of many efforts in 3D generative models based on images or videos only, as they are in abundance. I am skeptical can be used to generate assets for real applications. (Authors rebutting this should focus on how data scalability be achieved?)
2. How expressive and generalizable is the model in term of fitting many datasets (shapenet, objaverse etc) jointly? (Authors should not do experiments if they have not already tried training on all datasets.)
3. I find it quite surprising that the GaussianCube data structure works well even when trying to fit shapes with different scales and dimension. I expect that this representation will be hard to scale for complicated topology and intricate details where number of gaussians required to approximate the shape will be quite large. (Authors should discuss it in the rebuttal.)
4. Missing training time and compute requirement for training diffusion model. (Authors should provide this data in the paper.)
questions: questions: 1. In my opinion, Figure 3 is not required. The textual explanation is enough.
2. What are the failure cases of optimal transport? Have authors considered a learnable scattering of gaussians to cubic grid? (Authors should discuss this in the rebuttal.)
limitations: limitations: Limitations and societal impacts are described in the Supplementary material.

Though the scope of the paper is 3D asset generation and experiments are done accordingly, I would have appreciated some discussion on large scale scene generation also.

I suspect that there are failure cases of optimal transport. If so, they should be described.
paper_summary: paper_summary: The paper is about 3d object generative model. The generative objects are represented with gaussian splatting. Thus the main obstacle is the large point clouds. It is difficult to generate such large point clouds. The authors proposed a way to put point cloud to a structured grid using optimal transport. Some generative experiments are shown.
strength_weakness: strength_weakness: I would like to see some time and memory analysis about the method, including data preprocessing (object fitting and optimal transport), training resources, and sampling speed. These are important for readers to understand how difficult it is to reimplement the method.

Another thing might be useful is novelty analysis. Shapenet is a relatively small dataset, which can be easily memorized.
questions: questions: The digital avatar experiment is image-conditioned generation? Is it possible to train an unconditional generative model?

I do not have further questions for this paper.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper introduces a novel explicit and structured 3D representation - GaussianCube - used in combination with U-Net diffusion models for 3D generation. While 3D Gaussians have a lot of advantages, they are not directly compatible with efficient architectures for generative modeling because of their unstructured nature. This paper tackles this problem by proposing two steps from 3D Gaussians to GaussianCube: 1) densification-constrained fitting to bound the number of Gaussians, and 2) structuralization via optimal transport to convert the underlying unstructured point cloud of 3D Gaussians to a structured voxel grid. 

Using datasets in this format, the authors train diffusion models parameterized as 3D U-Nets for object, avatar, and text-to-3D generation.
strength_weakness: strength_weakness: * Lack of clarity:
  - The paper does not include any details about the diffusion process on the properties of 3D Gaussians (except of normalization with dataset statistics in the appendix):
    - 3D Gaussians have parameters with very different distributions making the diffusion process on them non-trivial.
    - As one example, if rotations are represented as quaternions with a diffusion process adding simply Gaussian noise, there is no guarantee about the final generation being a valid rotation.
    - Also the distribution of offsets from voxel centers would be insightful, especially across different objects/scenes with different level of detail.
  - The fact that positions are replaced by offsets of the voxel centers became clear rather late in the paper (ll. 153 ff.) and should be explained earlier.
  - The optimal transport illustration in Fig. 3 is not intuitive. Are there only two valid pixel corners on the diagonal?

* Missing related work and baseline:
  - The related work about diffusion models on 3D representations is missing [1] and [2].
  - The state-of-the-art comparison on ShapeNet misses an important baseline [1] with strong performance.

[1] Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction, ICCV 2023
[2] Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation, CVPR 2024
questions: questions: * Why do you use $x_0$-parameterization instead of $\epsilon$ or v?
* The resolution 32 is rather small and the appendix shows stronger fitting performance for larger numbers of Gaussians. Is there potential for latent diffusion?
* Why is the reported performance of DiffTF worse than the one from the original paper?
limitations: limitations: The authors addressed limitations and broader (including potential negative societal) impact in the appendix.

Comments: title: Post-Rebuttal
comment: Dear Reviewer,  

We have tried our best to address your questions as detailed in our top-level comment and the rebuttal above. Once again, we extend our gratitude for your time and effort in reviewing our manuscript, and we stand ready to provide any additional information that could be helpful.
Dear Reviewer,  

We have tried our best to address your questions as detailed in our top-level comment and the rebuttal above. Once again, we extend our gratitude for your time and effort in reviewing our manuscript, and we stand ready to provide any additional information that could be helpful.
Dear Reviewer,  

We have tried our best to address your questions as detailed in our top-level comment and the rebuttal above. Once again, we extend our gratitude for your time and effort in reviewing our manuscript, and we stand ready to provide any additional information that could be helpful.
Dear Reviewer,

We appreciate your thoughtful feedback and the increase in rating. We recognize that our current model demands high computation and we have only shown results of GaussianCube with a $32\times32\times32$ resolution. This is due to the inherent complexity of 3D generation. We're working hard to make our model more efficient. As we continue to learn and improve our techniques, we plan to apply our approach to more complex cases like 3D scene generation.

Thank you again for raising the score.
I thank the authors for their clarifications regarding my concerns and providing additional results. The rebuttal addresses the mentioned weaknesses and questions. After carefully considering all reviews and answers by the authors, I would like to increase the rating to 7: Accept (see edited score in review).

The following points prevent me from giving an even higher rating:
- Limited contribution: Application of optimal transport for voxelization
- Scalability concerns:
  - high computational demand for training, 
  - rather low resolution of 32x32x32, 
  - and therefore the limitation to objects only