Atlas Gaussians Diffusion for 3D Generation

## Abstract

Abstract Using the latent diffusion model has proven effective in developing novel 3D generation techniques. To harness the latent diffusion model, a key challenge is designing a high-fidelity and efficient representation that links the latent space and the 3D space.
In this paper, we introduce Atlas Gaussians, a novel representation for feed-forward native 3D generation. Atlas Gaussians represent a shape as the union of local patches, and each patch can decode 3D Gaussians. We parameterize a patch as a sequence of feature vectors and design a learnable function to decode 3D Gaussians from the feature vectors. In this process, we incorporate UV-based sampling, enabling the generation of a sufficiently large, and theoretically infinite, number of 3D Gaussian points. The large amount of 3D Gaussians enables the generation of high-quality details. Moreover, due to local awareness of the representation, the transformer-based decoding procedure operates on a patch level, ensuring efficiency.
We train a variational autoencoder to learn the Atlas Gaussians representation, and then apply a latent diffusion model on its latent space for learning 3D Generation.
Experiments show that our approach outperforms the prior arts of feed-forward native 3D generation.

## 1 Introduction

3D generation has become increasingly important in various domains, including virtual reality, gaming, and film production. Recent advances in diffusion models Ho et al. (); Song et al. () have improved the quality of 3D generation, offering superior performance over previous methods, such as variational autoencoders (VAEs) Kingma & Welling (); Mo et al. () and generative adversarial networks (GANs)Goodfellow et al. (); Gao et al. ().

Despite progress, the effectiveness of 3D generation still falls short compared to 2D generation models Ho & Salimans (); Ramesh et al. (); Saharia et al. (). The robust performance in 2D generation is largely due to the effective integration of VAEs with latent diffusion models (LDMs) Rombach et al. ().
A primary challenge hindering the complete success of this paradigm in 3D generation is the development of a high-fidelity 3D representation that can be efficiently embedded into a low-dimensional latent space. Pioneering efforts have applied diffusion models to various traditional 3D representations, such as point clouds Zeng et al. (); Zhou et al. (); Luo & Hu (), meshes Liu et al. (), occupancy fields Zheng et al. (); Zhang et al. () and signed distance functions Cheng et al. (); Li et al. (); Zhang et al. (). However, these approaches often focus solely on modeling geometry without considering the appearance attributes. More recently, a notable attempt Lan et al. () has designed a VAE that uses volume rendering techniques Mildenhall et al. () to incorporate appearance modeling. However, volume rendering presents inherent limitations, including slow rendering speeds and constrained rendering resolutions. To overcome these limitations, we have developed a VAE that leverages the latest 3D Gaussian representation Kerbl et al. (), which significantly improves both the quality and speed of rendering.

Designing a VAE based on 3D Gaussians presents considerable challenges. The first challenge lies in creating an efficient decoder capable of mapping low-dimensional latents to 3D Gaussians. Existing methods for 3D Gaussian decoding focus predominantly on reconstruction tasks and typically lack an information bottleneck design Tang et al. (); Yinghao et al. (), thus failing to provide low-dimensional latent directly. Alternatively, they often require multiple complex and interdependent components Xu et al. (); Zou et al. (). The second challenge involves generating a sufficiently large number of 3D Gaussians efficiently, since high-quality rendering necessitates an adequate quantity of these Gaussians. Some current methods Xu et al. (); Zou et al. () address this by employing additional complex point-upsampling networks to increase the number of 3D Gaussians, which inherently limits the number of Gaussians that can be generated. Other techniques Tang et al. (); Yinghao et al. () utilize image representations to generate a large number of 3D Gaussians. However, for all these methods, more network parameters are usually required as the number of 3D Gaussians increases.

To address the challenges of designing VAEs for 3D Gaussians, we propose Atlas Gaussians, a new representation for 3D generation. This representation is inspired by surface parameterization Floater & Hormann (), a foundation technique in many graphics applications where surface attributes are sampled and stored in a 2D texture map. Specifically, Atlas Gaussians model the shape as a union of local patches, with each patch decoding 3D Gaussians via UV-based sampling. By parameterizing 3D Gaussians in the UV space, we can easily generate a sufficiently large, and theoretically infinite, number of 3D Gaussians. Unlike traditional surface parameterization approaches, the UV mapping in Atlas Gaussians are learned end-to-end using a corner-encoding scheme. A significant advantage of Atlas Gaussians is that the sampling process does not require additional network parameters as the number of 3D Gaussians increases.

We design a transformer-based decoder to map low-dimensional latents to Atlas Gaussians. This decoder is specifically structured to disentangle geometry and appearance features, facilitating faster convergence and improved representation capabilities.
Using the local awareness of Atlas Gaussians, we also reduce computational complexity by decomposing the self-attention layers.
Finally, the latent space learned by our VAE can be applied to existing latent diffusion models efficiently.

Note that in contrast to the main approach in 3D generation that uses the multi-view representation Wang & Shi (); Long et al. (); Liu et al. (); Shi et al. (), our approach is inherently 3D-based. Therefore, a key advantage is that we do not need to address the challenging multi-view consistency issue associated the multi-view representation.
We take inspiration from the literature that 3D-based neural networks learn better 3D features for recognition and outperform multi-view based networks that leverage massive 2D labeled data.
Moreover, the rendering module of Atlas Gaussians allows representation learning from images.

In summary, we make the following contributions.

• We propose Atlas Gaussians, a new 3D representation that can efficiently decode a sufficiently large and theoretically infinite number of 3D Gaussians for high-quality 3D generation.

• We design a new transformer-based decoder to efficiently map low-dimensional latents to Atlas Gaussians, using separate branches to disentangle geometry and appearance features.

• We pioneer the integration of 3D Gaussians into the VAE + LDM paradigm, demonstrating superior performance on standard 3D generation benchmarks.

## 2 Related work

3D representation. 3D reconstruction and generation benefit from different 3D representations by leveraging their unique properties. These representations include explicit representations Wu et al. (); Mittal et al. (); Ren et al. (); Zeng et al. (); Zhou et al. (); Luo & Hu (); Sun et al. (); Xie et al. (); Yang et al. (); Sun et al. (); Achlioptas et al. (); Fan et al. (); Liu et al. (); Nash et al. (); Siddiqui et al. (); Groueix et al. (); Chen et al. (); Kerbl et al. (); Tang et al. (); Yinghao et al. (); Zou et al. (); Szymanowicz et al. () and implicit representations Park et al. (); Chen & Zhang (); Mescheder et al. (); Li et al. (); Zheng et al. (); Shue et al. (); Jiang et al. (); Hui et al. (); Yan et al. (); Mildenhall et al. (); Zhang et al. (); Chan et al. (); Gu et al. (); Chen et al. (); Cao et al. (); Müller et al. (); Watson et al. ().
In this paper, we focus on 3D Gaussians Kerbl et al. (); Tang et al. (); Yinghao et al. (); Zou et al. (); Szymanowicz et al. (); He et al. (), which possess high-quality rendering procedures that allow learning from image supervisions. However, existing results mainly focus on reconstructing 3D Gaussians. Our goal is to push the state-of-the-art in generative 3D Gaussians.

Our method is also related to the Atlas representation initially proposed by AtlasNet Groueix et al. () and its subsequent extensions Deprelle et al. (); Liu et al. (); Feng et al. (). AtlasNet models the shape as a union of independent MLPs, thus limiting the number of patches to a few dozen. In contrast, our proposed Atlas Gaussians model each patch using a patch center and patch features. This efficient encoding allows us to generate a significantly larger number of patches, providing stronger representation capabilities. Additionally, we use a transformer to learn this representation instead of an MLP, resulting in better scalability.

Diffusion models.
Diffusion models Sohl-Dickstein et al. (); Song et al. (; ) have been dominant for diverse generation tasks, including image Ho & Salimans (); Zhang et al. (); Podell et al. (); Rombach et al. (), video Ho et al. (; ); Blattmann et al. (), audio Huang et al. (); Kong et al. (); Liu et al. (), text Li et al. (); Gong et al. (); Lin et al. (), and motion Zhang et al. (); Tevet et al. (); Yuan et al. ().
The success of these models typically follows the VAE + LDM paradigm. Although pioneering efforts Zeng et al. (); Zhou et al. (); Luo & Hu (); Liu et al. (); Zheng et al. (); Zhang et al. (); Cheng et al. (); Li et al. (); Jun & Nichol (); Lan et al. () have attempted to apply this paradigm to 3D, the problem remains unsolved and has not achieved the same level of success. We argue that one of the main reasons is the need for an efficient VAE to represent high-quality 3D content, which is the key contribution of this paper.

3D generation.
3D generation methods can be classified into two genres. The first is optimization-based methods Jain et al. (); Sun et al. (); Wang et al. (); Lin et al. (); Wang et al. (); Chen et al. (), which are time-consuming due to per-shape optimization. For example, DreamField Jain et al. () uses CLIP guidance. DreamFusion Poole et al. () and SJC Wang et al. () introduce 2D diffusion priors in different formats. Magic3D Lin et al. () employs a coarse-to-fine pipeline to improve convergence speed. Prolificdreamer Wang et al. () uses variational score distillation to enhance generation fidelity. Fantasia3D Chen et al. () disentangles geometry and texture to achieve higher quality generation.

The second genre involves training a generalizable feed-forward network to output 3D content, which allows for fast 3D generation but often with less details. One direction is to apply existing generative modeling techniques (VAE, GAN, normalizing flow Rezende & Mohamed (), autoregressive model Bengio et al. (); Graves () and diffusion model) directly on various 3D representations. Another direction Xu et al. (); Liu et al. (; ; ); Shi et al. (); Hong et al. (); Tang et al. () uses 2D as intermediate representation, integrating 2D diffusion models Ho et al. (); Shi et al. (). For example, Instant3D Li et al. () trains a diffusion model to generate sparse multi-view images and uses a reconstruction model to derive the 3D shapes from the 2D images. VFusion3D Han et al. () and V3D Chen et al. () use video diffusion models to improve consistency between generated images. However, these methods rely on 2D representations and often suffer from multi-view consistency issues.
In this paper, we push the state-of-the-art in the first direction by developing a novel representation of Atlas Gaussians. Our representation is efficient and allows for learning 3D generative models from images.

## 3 Method

Figure: Figure 2: (Left) Atlas Gaussians $\mathcal{A}$ model the shape as a union of patches, where each patch can decode 3D Gaussians. (Right) Each patch $\boldsymbol{a}_{i}$ is parameterized by patch center $\boldsymbol{x}_{i}$ and patch features $\boldsymbol{f}_{i}$ and $\boldsymbol{h}_{i}$. The 3D Gaussians are decoded via the UV-based sampling.

In this section, we first introduce our Atlas Gaussians representation (Sec. 3.1). Then we introduce how we learn a VAE to connect the 3D space with the latent space (Sec. 3.2). Finally, we introduce how we learn the generative model using latent diffusion in the learned latent space (Sec. 3.3).

### 3.1 Atlas Gaussians Representation

In 3D Gaussian Splatting Kerbl et al. (), each 3D Gaussian $\boldsymbol{\mathrm{g}}$ can be parameterized with a center $\boldsymbol{\mathrm{\mu}}\in\mathbb{R}^{3}$, scale $\boldsymbol{\mathrm{s}}\in\mathbb{R}^{3}$, rotation quaternion $\boldsymbol{\mathrm{r}}\in\mathbb{R}^{4}$, opacity ${\mathrm{o}}\in\mathbb{R}$ and color $\boldsymbol{\mathrm{c}}\in\mathbb{R}^{3}$. To achieve high-quality rendering results, typically a sufficiently large number of 3D Gaussians are required.

The key idea of Atlas Gaussians is to represent the shape as a union of $M$ local patches $\mathcal{A}=\{\boldsymbol{a}_{i}\}_{i=1}^{M}$, where each patch $\boldsymbol{a}_{i}$ can decode 3D Gaussians through UV-based sampling. As shown in Figure 2, we parameterize each local patch $\boldsymbol{a}_{i}:=(\boldsymbol{x}_{i},\boldsymbol{f}_{i},\boldsymbol{h}_{i})$ with patch center $\boldsymbol{x}_{i}\in\mathbb{R}^{3}$, the geometry features $\boldsymbol{f}_{i}\in\mathbb{R}^{4\times d}$ and the appearance features $\boldsymbol{h}_{i}\in\mathbb{R}^{4\times d}$. More specifically, we parameterize the geometry and appearance features as the features at the four corners of the local patch in the UV space. We denote $\boldsymbol{f}_{i}=(\boldsymbol{f}_{i1},\boldsymbol{f}_{i2},\boldsymbol{f}_{i3
},\boldsymbol{f}_{i4})$ and $\boldsymbol{h}_{i}=(\boldsymbol{h}_{i1},\boldsymbol{h}_{i2},\boldsymbol{h}_{i3
},\boldsymbol{h}_{i4})$. This type of feature disentanglement can facilitate more effective learning Gao et al. (); Zou et al. (); Xu et al. (). We use the feature $\boldsymbol{f}_{i}$ to decode Gaussian positions while using $\boldsymbol{h}_{i}$ to decode the rest of Gaussian attributes. We also assign a 2D coordinate $\boldsymbol{u}_{ij}\in\mathbb{R}^{2}$ to each feature vector $\boldsymbol{f}_{ij}$ and $\boldsymbol{h}_{ij}$ as a positional embedding, where $\boldsymbol{u}_{i1}=(0,0)$, $\boldsymbol{u}_{i2}=(1,0)$, $\boldsymbol{u}_{i3}=(1,1)$, $\boldsymbol{u}_{i4}=(0,1)$, $\forall i$.

Note that this representation is motivated by the concept of UV-map, in which the four corners describe the corners of the rectangular parameter domain. As we shall discuss later, the features $\boldsymbol{f}_{i}$ and $\boldsymbol{h}_{i}$ are learned end-to-end with 3D Gaussians. This approach takes advantage of end-to-end learning, while the specific network design promotes learning better features.

For generation, we randomly sample query points in the predefined unit square UV space for each patch $\boldsymbol{a}_{i}$. Each point is then decoded into a 3D Gaussian.
Given a query point $\boldsymbol{q}_{ij}\in{[0,1]}^{2}$, we map the 2D coordinate $\boldsymbol{q}_{ij}$ to the center of 3D Gaussians as:

$$ $\boldsymbol{\mathrm{\mu}}_{ij}=\phi(\boldsymbol{q}_{ij},\boldsymbol{u}_{i}, \boldsymbol{f}_{i}),$ (1) $$

where $\phi$ is the mapping function, which takes the query point location $\boldsymbol{q}_{ij}$, the predefined location of patch feature vectors $\boldsymbol{u}_{i}$, and the geometry features $\boldsymbol{f}_{i}$ as inputs.
We implement the mapping function using interpolation in the 2D space:

$$ $\boldsymbol{\mathrm{\mu}}_{ij}=\textsc{MLP}(\sum_{k=1}^{4}w(\boldsymbol{q}_{ij },\boldsymbol{u}_{ik},\boldsymbol{f}_{ik})\cdot\boldsymbol{f}_{ik})+ \boldsymbol{x}_{i},$ (2) $$

where $w$ is the weight function of the four corners and we use an MLP to decode the residual between the Gaussian location and the patch center $\boldsymbol{x}_{i}$

One design choice for $w$ is the bilinear interpolation weight function, which has been widely used in feature decoding Müller et al. (). However, these linear weights purely based on coordinates have limited representation ability. Inspired by Zhang et al. (), we design a more powerful weight function defined on both coordinates and features:

$$ $w(\boldsymbol{q}_{ij},\boldsymbol{u}_{ik},\boldsymbol{f}_{ik})=\frac{ \widetilde{w}(\boldsymbol{q}_{ij},\boldsymbol{u}_{ik},\boldsymbol{f}_{ik})}{ \sum_{l=1}^{4}\widetilde{w}(\boldsymbol{q}_{ij},\boldsymbol{u}_{il}, \boldsymbol{f}_{il})},$ (3) $$

$$ $\text{and}\ \widetilde{w}(\boldsymbol{q}_{ij},\boldsymbol{u}_{il},\boldsymbol{ f}_{il})=e^{\omega_{2}(\boldsymbol{q}_{ij})^{T}(\boldsymbol{f}_{il}+\omega_{2} (\boldsymbol{u}_{il}))/\sqrt{d}},$ (4) $$

where $\omega_{2}:\mathbb{R}^{2}\mapsto\mathbb{R}^{d}$ is the sinusoidal positional encoding function with MLP projection.

Similarly, the remaining Gaussian attributes are decoded by the function $\psi$:

$$ $(\boldsymbol{\mathrm{s}}_{ij},\boldsymbol{\mathrm{r}}_{ij},{\mathrm{o}}_{ij}, \boldsymbol{\mathrm{c}}_{ij})=\psi(\boldsymbol{q}_{ij},\boldsymbol{u}_{i}, \boldsymbol{h}_{i})=\text{MLP}(\sum_{k=1}^{4}w(\boldsymbol{q}_{ij},\boldsymbol {u}_{ik},\boldsymbol{h}_{ik})\cdot\boldsymbol{h}_{ik}),$ (5) $$

The benefits of Atlas Gaussians representation are three-fold. First, through UV-based sampling in a unit square, Atlas Gaussians enable easy generation of a sufficiently large number of 3D Gaussians. They also possess the potential to generate a variable and theoretically infinite number of 3D Gaussians. Second, Atlas Gaussians utilize a non-linear learnable weight function based on the MLP projection of the positional encoding, which has a stronger representation ability than the existing linear interpolation weight. Third, Atlas Gaussians are computation efficient with low memory overhead. Another important property of Atlas Gaussians is that they do not require extra network parameters when scaling up the number of generated 3D Gaussians. We also provide an ablation study in Section 4.4 to validate the nice properties of Atlas Gaussians.

Figure: Figure 3: The proposed VAE architecture. CA denotes the cross-attention layer. For simplicity, the variational component of the VAE is omitted. The latent $\boldsymbol{z}_{0}$ is used for latent diffusion.

### 3.2 Stage 1: VAE

We design a VAE to link the low-dimensional latent space and the 3D space. The overall pipeline is shown in Figure 3.

Encoder.
The encoder takes shape information as input and outputs a latent set $\boldsymbol{z}_{0}\in\mathbb{R}^{n\times d_{0}}$, which follows the latent set representation in Zhang et al. (). The input shape information is represented as a set of features $\mathcal{S}$. We use transformers with cross attention to extract the latent features, as $\boldsymbol{z}=\text{CrossAttn}(\bar{\boldsymbol{z}},\mathcal{S}),$
where $\bar{\boldsymbol{z}}$ is the initialization of the latent features. In this notation, we omit the feed forward MLP of the transformers for simplicity. Intuitively, the query $\bar{\boldsymbol{z}}$ of the cross-attention is updated by aggregating information from key/value $\mathcal{S}$. We further refine the latent features $\boldsymbol{z}$ using transformers with self-attention.

To obtain shape information $\mathcal{S}$, we use point cloud $\mathcal{P}=\{\boldsymbol{p}_{i}\in\mathbb{R}^{3}\}$ and sparse view RGB images $\mathcal{I}=\{\boldsymbol{I}_{i}\in\mathbb{R}^{H\times W\times 3}\}$. We encode the location of points into positional embeddings, resulting in features in $\mathbb{R}^{|\mathcal{P}|\times d}$. Meanwhile, a ViT network Dosovitskiy et al. (); Oquab et al. () embeds each image into features of shape $\mathbb{R}^{h\times w\times d}$, resulting in the features of all images in $\mathbb{R}^{(|\boldsymbol{I}|\times h\times w)\times d}$. We use two transformers with cross-attention to aggregate information from them, respectively. We initialize the latent features $\bar{\boldsymbol{z}}\in\mathbb{R}^{n\times d}$ as the encoding of the points sampled using the farthest point sampling Zhang et al. (). After feature aggregation, we use an MLP to map the features to a lower-dimensional space $\mathbb{R}^{n\times d}\rightarrow\mathbb{R}^{n\times d_{0}}$, facilitating efficient latent diffusion.

Note that attention is invariant to the permutation of features.
We also avoid pose-dependent operations Jiang et al. (); Jun & Nichol () in our design to enhance generality and extendability.

Decoder.
The decoder recovers patch features $\mathcal{A}=\{(\boldsymbol{x}_{i},\boldsymbol{f}_{i},\boldsymbol{h}_{i})\}_{i=
1}^{M}$ from the latent code $\boldsymbol{z}_{0}$. In the decoder, we upsample the latent code, decode the patch centers $\{\boldsymbol{x}_{i}\}_{i=1}^{M}$, and then decode the patch geometry and appearance features $\{(\boldsymbol{f}_{i},\boldsymbol{h}_{i})\}_{i=1}^{M}$.

The upsampling module is designed to increase both the length and the feature dimension of the latent code, from $\boldsymbol{z}_{0}\in\mathbb{R}^{n\times d_{0}}$ to $\boldsymbol{z}_{l}\in\mathbb{R}^{M\times d}$. In detail, we first use a learnable query $\boldsymbol{y}$ to aggregate information from the latent code $\boldsymbol{z}_{0}$ using transformers with cross-attention. Then, we increase its channel dimension to $d$ using self-attention transformers, leading to the characteristic $\boldsymbol{z}_{1}\in\mathbb{R}^{n\times d}$.
We then use an MLP to increase the feature dimension of $\boldsymbol{z}_{1}$ into $\frac{M}{n}d$, leading to latent features in $\mathbb{R}^{n\times\frac{M}{n}d}$. We reshape the features into $\boldsymbol{z}_{2}\in\mathbb{R}^{M\times d}$, which is a pixel shuffle process Shi et al. (); Xu et al. (). We then use another set of transformers with self-attention to obtain the output latent features $\boldsymbol{z}_{l}$. This process is summarized as follows:

$$ $\boldsymbol{z}_{1}=\text{SelfAttn}(\text{CrossAttn}(\boldsymbol{y},\boldsymbol {z}_{0})),\quad\boldsymbol{z}_{2}=\text{PixelShuffle}(\text{MLP}(\boldsymbol{z }_{1})),\quad\boldsymbol{z}_{l}=\text{SelfAttn}(\boldsymbol{z}_{2}).$ (6) $$

Again, we omit the feed-forward MLP in the transformers for simplicity. We then decode the patch centers using transformers with self-attention:

$$ $\{\boldsymbol{x}_{i}\}_{i=1}^{M}=\text{SelfAttn}(\boldsymbol{z}_{l}).$ (7) $$

We use a two-branch module to decode geometry and appearance features $\{(\boldsymbol{f}_{i},\boldsymbol{h}_{i})\}_{i=1}^{M}$ from upsampled latent code $\boldsymbol{z}_{l}$. We demonstrate its architecture in Figure 4. Take the branch for decoding the geometry features $\{\boldsymbol{f}_{i}\}_{i=1}^{M}$ as an example. We use another upsampling module to map $\boldsymbol{z}_{l}\in\mathbb{R}^{M\times d}$ to $\boldsymbol{z}_{f}\in\mathbb{R}^{M\times\beta\times d}$, where $\beta=4$, corresponding to the geometry features of the four corners for each patch. We then use transformers with self-attention to refine the upsampled features to get $\boldsymbol{f}_{i}$.

Specifically, we apply computational decomposition to the self-attention layers, as naive self-attention leads to a $\mathcal{O}(\beta^{2}M^{2}d)$ complexity due to the long sequence length of $M\beta$. We apply local self-attention to the features that belong to each local patch, reducing the complexity to $\mathcal{O}(\beta^{2}Md)$. The features of different local patches are updated independently. This design ensures local awareness of Atlas Gaussians decoding. To further ensure global awareness, we repeat and add the global features to the local patch features.

Figure: Figure 4: In the patch feature decoder, global features are broadcast and added to the local features. SA denotes the self-attention layers.

After obtaining the Atlas Gaussians $\mathcal{A}=\{(\boldsymbol{x}_{i},\boldsymbol{f}_{i},\boldsymbol{h}_{i})\}_{i=
1}^{M}$ from the VAE, we can decode 3D Gaussians following Eq. 2 and Eq. 5.

Training. Similar to existing methods Lan et al. (); Xu et al. (); Zou et al. (), we utilize a 3D dataset for supervision. We first regularize the patch center to adhere to the 3D surface geometry:

$$ $\mathcal{L}_{\text{center}}=\mathcal{L}_{\text{CD}}(\{\boldsymbol{x}_{i}\}_{i= 1}^{M},\mathcal{P}_{\text{GT}})+\mathcal{L}_{\text{EMD}}(\{\boldsymbol{x}_{i} \}_{i=1}^{M},\mathcal{P}_{\text{GT}}),$ (8) $$

where $\mathcal{P}_{\text{GT}}$ is the ground truth surface point cloud, $\mathcal{L}_{\text{CD}}$ and $\mathcal{L}_{\text{EMD}}$ are Chamfer Distance (CD) and Earth Mover’s Distance (EMD), respectively.
Similarly, we also supervise the centers of all 3D Gaussians with

$$ $\mathcal{L}_{\boldsymbol{\mathrm{\mu}}}(S)=\mathcal{L}_{\text{CD}}(\{ \boldsymbol{\mathrm{\mu}}_{ij}\}_{i=1,j=1}^{M,S},\mathcal{P}_{\text{GT}})+ \mathcal{L}_{\text{EMD}}(\{\boldsymbol{\mathrm{\mu}}_{ij}\}_{i=1,j=1}^{M,S}, \mathcal{P}_{\text{GT}}),$ (9) $$

where $\{\boldsymbol{\mathrm{\mu}}_{ij}\}_{i=1,j=1}^{M,S}$ represents the generated point cloud by sampling $S$ points in each of the $M$ patches. When computing the loss for supervision, we can generate point clouds with different resolutions by varying $S$, thanks to Atlas Gaussians’ ability to dynamically generate a variable number of 3D Gaussians.
Note that patches may overlap, similar to AtlasNet. $\mathcal{L}_{\text{center}}$ and $\mathcal{L}_{\boldsymbol{\mathrm{\mu}}}(S)$ encourage the patches to distribute more uniformly across the surface, thereby making better use of the 3D Gaussians.

While surface points are independent samples of the continuous geometry, 3D Gaussians are interdependent because one 3D Gaussian can affect the attributes of its neighbors during alpha blending. To ensure consistency and achieve deterministic results, during rendering we employ a uniform $\alpha\times\alpha$ grid sampling in the UV space instead of random sampling, resulting in $N=\alpha^{2}M$ 3D Gaussians. We use the differentiable renderer from Kerbl et al. () to render $V$ views of RGB, alpha, and depth images, which is supervised by mean square error:

$$ $\mathcal{L}_{\text{render}}=\mathcal{L}_{\text{MSE}}(\hat{\boldsymbol{I}}_{ \text{rgb}},\boldsymbol{I}_{\text{rgb}})+\mathcal{L}_{\text{MSE}}(\hat{ \boldsymbol{I}}_{\text{alpha}},\boldsymbol{I}_{\text{alpha}})+\mathcal{L}_{ \text{MSE}}(\hat{\boldsymbol{I}}_{\text{depth}},\boldsymbol{I}_{\text{depth}}),$ (10) $$

where $\hat{\boldsymbol{I}}_{\text{rgb}}$, $\hat{\boldsymbol{I}}_{\text{alpha}}$, and $\hat{\boldsymbol{I}}_{\text{depth}}$ are the predictions, $\boldsymbol{I}_{\text{rgb}}$, $\boldsymbol{I}_{\text{alpha}}$, and $\boldsymbol{I}_{\text{depth}}$ are the ground truths.
To improve the visual fidelity, we also employ the LPIPS loss Zhang et al. (); Tang et al. () $\mathcal{L}_{\text{LPIPS}}(\hat{\boldsymbol{I}}_{\text{rgb}},\boldsymbol{I}_{
\text{rgb}})$. The total loss function for the VAE is:

$$ $\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{center}}+\sum_{S=\{1,4\}} \mathcal{L}_{\boldsymbol{\mathrm{\mu}}}(S)+\lambda_{r}\left(\mathcal{L}_{\text {render}}+\mathcal{L}_{\text{LPIPS}}\right)+\lambda_{KL}\mathcal{L}_{\text{KL} }(\boldsymbol{z}),$ (11) $$

where $\mathcal{L}_{\text{KL}}$ is the Kullback–Leibler divergence, $\lambda_{r}$ and $\lambda_{KL}$ are the loss weights.

### 3.3 Stage 2: latent diffusion model

The proposed VAE provides a low-dimensional latent code that can be mapped to Atlas Gaussians. We employ EDM Karras et al. () for latent diffusion. Our denoising network follows the same architecture as Zhang et al. (), which consists of a series of transformer blocks:

$$ $\boldsymbol{z}^{(i)}=\text{CrossAttn}\left(\text{SelfAttn}(\boldsymbol{z}^{(i- 1)}),\mathcal{C}\right),\quad i=1,\ldots,l$ (12) $$

where $\boldsymbol{z}^{(0)}=\boldsymbol{z}_{0}$ is the initial input to the network, $l$ is the number of blocks. In unconditional generation, we designate the condition $\mathcal{C}$ as a learnable parameter. Since the LDM network design is not our main contribution, we provide the implementation details in Appendix A.3.

## 4 Experiments

We first introduce our experimental setup, and then show the results for both unconditional and conditional generation.

### 4.1 Experimental setup

Following most existing methods Gao et al. (); Müller et al. (); Chen et al. (); Lan et al. (), we benchmark unconditional single-category 3D generation on ShapeNet Chang et al. (). We use the training split from SRN Sitzmann et al. (), which comprises 4612, 2151, and 3033 shapes in the categories Chair, Car, and Plane, respectively. We render 76 views for each shape using Blender Community () with the same intrinsic matrix as Cao et al. (); Lan et al. (). Fréchet Inception Distance (FID@50K) and Kernel Inception Distance (KID@50K) are used for evaluation.
In addition, we experiment with text-conditioned 3D generation on Objaverse Deitke et al. (). We use the renderings from G-buffer Objaverse Qiu et al. () and the captions from Cap3D Luo et al. (). Due to limited computational resources, we select a high-quality subset with around 18K 3D shapes. We use CLIP score Radford et al. (); Hessel et al. (), FID and KID for evaluation.
Please refer to Appendix A.1 for more implementation details.

### 4.2 Unconditional 3D generation

Table 1 presents the quantitative comparison between our method and baseline approaches. Our method outperforms all baseline approaches, including EG3D Chan et al. (), GET3D Gao et al. (), DiffRF Müller et al. (), RenderDiffusion Anciukevičius et al. (), SSDNeRF Chen et al. () and LN3Diff Lan et al. ().
We also include the qualitative comparison with LN3Diff in Figure 5.
Our results demonstrate significant improvements over LN3Diff, particularly in the ShapeNet Chair category, which features greater geometric, structural, and textural complexity. The results highlight the robustness and efficacy of our approach.

Figure: Figure 5: The comparison between LN3Diff and our method on three ShapeNet categories.

Table: Table 1: Evaluation of single-category unconditional generation on ShapeNet.

### 4.3 Conditional 3D generation

We evaluate our method and baseline approaches He et al. (); Lan et al. (); Tang et al. (); Jun & Nichol () on text-conditioned 3D generation using Objaverse. The qualitative results are presented in 6, where all text prompts are sourced from the original baseline papers. As shown in Figure 6, the image-based generalizable 3D reconstruction method Tang et al. () sometimes produces shapes with significant artifacts, primarily due to its reliance on a multi-view generation network, which frequently suffers from multi-view inconsistency. In contrast, our method learns directly from 3D data, enabling the generation of consistent novel views.
Additionally, our method produces higher-quality 3D shapes compared to Shap-E and LN3Diff, both of which use the VAE + LDM paradigm. However, these methods depend on volume rendering during training, which is typically restricted to low resolution. Our approach instead leverages the more efficient 3D Gaussian representation. Compared to GVGEN, our generated shapes exhibit finer details, owing to the proposed Atlas Gaussians representation, which is capable of decoding a significantly larger number of 3D Gaussians.
Quantitative results are provided in Table 2, where our method achieves the best performance across all metrics and has the shortest inference time. This improvement is due to our efficient decoder design, which effectively links the low-dimensional latent space with 3D space. Additional results can be found in Appendix B.

Figure: Figure 6: Comparison of text-conditioned 3D generation on Objaverse. From left to right: GVGEN He et al. (), LN3Diff Lan et al. (), LGM Tang et al. (), Shap-E Jun & Nichol (), and our method. All text prompts are sourced from the original baseline papers.

Figure: Figure 7: (Left) Our generated results demonstrate significant diversity. (Right) Our generated results align closely with the text prompts, allowing for strong controllability. In the second row of each group, we present the nearest neighbors (NN) from the training dataset.

In Figure 7, we provide a detailed analysis of our method. In Figure 7 (Left), we present text-conditioned generation results with different random seeds, demonstrating that our method produces highly diverse outputs. In Figure 7 (Right), we show that our model can be robustly controlled using different text prompts. The generated results are also significantly different from their nearest neighbors in the training dataset, highlighting the model’s ability to generate novel content.

Table: Table 2: Evaluation of text-conditioned 3D generation on Objaverse.

Table: Table 3: Ablation study on the number of 3DGS and the number of patches on Objaverse. (LPIPS$\downarrow$ / MEM$\downarrow$)

### 4.4 Ablation study

This section presents an ablation study on different components of our network.

Number of 3D Gaussians and number of patches.
In Table 4, we analyze the effects of both the number of patches and the total number of 3D Gaussians on LPIPS and memory usage (MEM, which represents the additional GPU memory required for each unit increase in batch size). The results indicate that increasing either the number of patches or the number of 3D Gaussians improves LPIPS. When the number of 3D Gaussians is fixed (32k or 100k), increasing the number of patches from 2,048 to 4,096 results in a 0.1-0.2 improvement in LPIPS. However, when the number of patches is fixed (2,048 or 4,096), increasing the number of 3D Gaussians from 32k to 100k leads to a more significant improvement in LPIPS while requiring less additional GPU memory. Importantly, the network parameters remain unchanged despite the increase in 3D Gaussians. This comparison demonstrates that increasing the number of 3D Gaussians through Atlas Gaussians decoding is more efficient and effective than increasing the number of patches. Notably, the Atlas Gaussians representation combines explicit patch generation with implicit interpolation to generate 3D Gaussians, offering a balanced trade-off between compact implicit and fast explicit representations.

Learned weights for feature decoding.
In Table 4, we ablate the VAE network design on the ShapeNet validation set using the PSNR metric. The table shows that replacing our learned weight function with bilinear interpolation weights leads to a performance drop for the VAE. This indicates that our learned nonlinear weights have stronger representation capabilities than linear weights.

Disentangle geometry and texture features.
Atlas Gaussians use separate branches to learn geometry and appearance features. To validate the effectiveness of this design, we experimented with an alternative network version where a single set of features is shared for both geometry and appearance. As shown in Table 4, performance decreases considerably with this shared feature approach. This shows that our design facilitates more effective learning. Note that both the geometry and appearance features are generated from the shared latent $\boldsymbol{z}_{0}$, so our design does not support sampling geometry or appearance independently.

Using global features.
As shown in Table 4, removing the global feature from $\boldsymbol{z}_{l}$ in the patch feature decoder leads to a performance drop. This outcome is expected, as the global features provide essential context that complements the local features.

Parameter decoding scheme.
In Gaussian Splatting, opacity, scale, and rotation also influence the geometry. We experimented with decoding opacity, scale, and rotation from the geometry features and found that the performance is nearly identical to when these attributes are decoded from the appearance features. This result indicates that the key factor is decoding the Gaussian centers and RGB colors using distinct, separate branches.

## 5 Conclusion

This paper introduces Atlas Gaussians, a new representation for feed-forward 3D generation. Atlas Gaussians enable the efficient generation of a sufficiently large and theoretically infinite number of 3D Gaussians, which is crucial for high-quality 3D generation. We also designed a transformer-based decoder to link the low-dimensional latent space with the 3D space. With these innovations, we pioneer the integration of 3D Gaussians into the VAE + LDM paradigm, achieving superior performance and producing high-fidelity 3D content.

#### Acknowledgments

We sincerely thank Yushi Lan for his invaluable assistance in the evaluation.

## Appendix A Implementation details

### A.1 Additional details of the dataset

We adapted the training setup from LN3Diff Lan et al. (), which initially used a subset of 35K shapes.(^1^11The authors recently scaled up their model with more data, and we use this updated version for comparison.) These shapes cover three general categories: Transportation, Furniture, and Animals, derived from the G-buffer Objaverse Qiu et al. (). Due to limited computational resources, we cleaned the dataset by filtering out duplicate and low-quality shapes (e.g., those with flat ground or poor geometry), resulting in a final subset of approximately 18K shapes for training. It is important to note that baselines like LN3Diff utilize additional data from other categories, which provides them an advantage, while our setting is comparatively disadvantaged.

We randomly selected 250 text prompts for evaluation, ensuring that each testing prompt differs from the training data. For each object, we rendered 24 views uniformly, following the G-buffer Objaverse protocol, resulting in a total of 6K images. FID and KID are computed based on the 2D image feature space.

### A.2 Additional details of VAE

The VAE training consists of two stages. In the first stage, $\lambda_{r}$ in Eq. 11 is set to 0. Rendering occurs only in the second stage with $\lambda_{r}$ set to 1. In both stages, $\lambda_{KL}$ maintains $1e^{-4}$. As shown in Eq. 11, all other loss weights are set to 1 for simplicity. Our method is robust to hyperparameters; doubling or halving $\lambda_{r}$ in Eq. 11 results in almost identical loss curves.

We utilize a sparse point cloud of 2048 points and 4 input views, each of size $224\times 224$. $M=2048$ patches are used in all experiments. For the latent, $n=512$, $d=512$. We set $d_{0}$ to 4 for ShapeNet and $d_{0}$ to 16 in Objaverse. In ShapeNet, $\alpha$ is set to 4, resulting in $N=32768$ 3D Gaussians. In Objaverse, $\alpha=7$. Due to the real-time rendering capabilities of the 3D Gaussians, we are able to produce an output image of size $512\times 512$. We use the EMD implementation from Liu et al. (), which supports fast and efficient computation even with up to 8192 points. All networks are trained on 8 Tesla V100 GPUs for 1000 epochs using the AdamW optimizer Loshchilov & Hutter () with the one-cycle policy. Our VAE is trained using mixed precision (fp16) and supports a batch size of 8 per GPU. For instance, when trained on the ShapeNet dataset, it requires approximately 22GB of memory per GPU, making it accessible to a wide range of laboratories. Notably, the model does not require the latest A100 GPUs for training. Furthermore, the training process is completed in less than 30 hours. For Objaverse, the VAE training takes about 6 days.

In terms of network parameters, our model increases the number of Gaussians without adding extra network parameters. Specifically, for Objaverse, our model requires only 142M network parameters to generate 100K Gaussians, while LGM Tang et al. () requires 415M parameters for generating 64K Gaussians. This demonstrates that our representation is more efficient than purely explicit representations like LGM, offering a nice trade-off between compact implicit and fast explicit representations.

### A.3 Additional details of LDM

We adopt the EDM Karras et al. () framework for latent diffusion. EDM aims to learn a denoising network $D_{\theta}(\boldsymbol{z};\sigma,\mathcal{C})$ to convert the Gaussian distribution to the empirical distribution $p_{\text{data}}$ defined by $\boldsymbol{z}$, where $\theta$ is the network parameters, $\sigma$ is the noise level sampled from a predefined distribution $p_{\text{train}}$, and $\mathcal{C}$ denotes the optional condition. $D_{\theta}(\boldsymbol{z};\sigma,\mathcal{C})$ is parameterized using a $\sigma$-dependent skip connection:

$$ $D_{\theta}(\boldsymbol{z};\sigma,\mathcal{C})=c_{\text{skip}}(\sigma) \boldsymbol{z}+c_{\text{out}}(\sigma)F_{\theta}\left(c_{\text{in}}(\sigma) \boldsymbol{z};c_{\text{noise}}(\sigma),\mathcal{C}\right),$ (13) $$

where $F_{\theta}$ is the network to be trained. The training objective is

$$ $\mathbb{E}_{\sigma,\boldsymbol{z},\boldsymbol{n}}\lambda(\sigma)||D_{\theta}( \boldsymbol{z};\sigma,\mathcal{C})-\boldsymbol{z}||^{2},$ (14) $$

where $\sigma\sim p_{\text{train}},\boldsymbol{z}\sim p_{\text{data}},\boldsymbol{n}
\sim\mathcal{N}(0,\sigma^{2}I)$. Readers interested in details on $c_{\text{skip}}(\sigma)$, $c_{\text{out}}(\sigma)$, $c_{\text{in}}(\sigma)$, $c_{\text{noise}}(\sigma)$, and parameterization of the weight function $\lambda(\sigma)$ may refer to EDM Karras et al. ().

For the latent diffusion model, we set $l=12$ for ShapeNet and $l=24$ for Objaverse. The final latents are obtained via 40 denoising steps. For text-conditioned 3D generation, we use CLIP to encode the input text prompts. In addition, we adopt classifier-free guidance. We randomly drop the conditioning signal with a probability of $10\%$ and set the guidance scale to 3.5 during sampling. Note that more advanced architectures Peebles & Xie () could also be employed for the denoising network.

## Appendix B Additional results

In Figure 8, we present additional results of text-conditioned 3D generation using our method.

Figure 9 shows the results of real-image-conditioned 3D generation. In this experiment, we simply replace the CLIP text encoder with the CLIP image encoder and train the latent diffusion model using the same dataset. Given a real image, we first apply an off-the-shelf tool to remove the background and then use the processed image as the conditional input for 3D generation. The results demonstrate that our method can still produce reasonable outcomes.

Figure: Figure 8: Additional results of text-conditioned 3D generation using our method.

Figure: Figure 9: Image-conditioned 3D generation using our method.

## Appendix C Limitations and Future Work

Our method uses the vanilla 3D Gaussian representation, which is known to be challenging for extracting highly accurate geometry. Incorporating recent advancements Huang et al. (); Yu et al. (); Guédon & Lepetit () in 3D Gaussian techniques can benefit both geometry and appearance.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - It is unclear how geometry and appearance features are disentangled in the latent space. While the loss function is designed to learn each patch’s center and appearance features through supervision, the process for learning geometric features remains vague.
soundness: 3
strengths: - The paper is well-written and organized; hence it is easy for readers to follow. 

- Addressing the current challenges of VAEs in extracting efficient latent representations for generative diffusion models is important. 

- The introduction of the Atlas Gaussians representation with disentangled learning of geometry and appearance features of 3D shapes is novel.
summary: This paper introduces Atlas Gaussians, a new latent representation learning framework designed to improve the efficiency of 3D object generation by addressing the current challenges faced by VAEs. By modeling 3D shapes as a union of local patches, the authors develop a transformer-based decoder to map the low-dimensional latent representations of each patch via UV-based sampling and promote time efficiency when generating large samples of 3D Gaussians. Additionally, the introduced Atlas Gaussian representation considers a disentangled learning of geometric and appearance features, resulting in faster convergence.
weaknesses: - The proposed approach includes both the Atlas Gaussian representation with disentangled geometry and appearance learning mechanism, as well as a transformer-based encoder. It is unclear which component contributes more significantly to the overall performance.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. L060 "However, these approaches often focus solely on modeling geometry without considering the appearance attributes."
Can you give a more detailed explanation?

2. What does $d$ represent in L189 and L135 (Equation 4)?

3. What does $n$ represent in L256 and L268?

4. What is the exact relationship between the shape information $\mathcal{S}$ (L263) and the point cloud $\mathcal{P}$ as well as the sparse-view RGB images $\mathcal{I}$? 
From my understanding, the feature dimension of $\mathcal{P}$ is $\mathbb{R}^{|\mathcal{P}| \times d}$, and the feature dimension of $\mathcal{I}$ is $\mathbb{R}^{(|\mathcal{I}| \times h \times w) \times d}$. 
How, precisely, are these features of $\mathcal{P}$ and $\mathcal{I}$, and the feature dimensions related to $\mathcal{S}$, particularly in mathematical terms?

5. How is the learnable query $y$ of the VAE decoder (L277, L284) initialized?
soundness: 3
strengths: 1. I appreciate how the introduction section clearly explains and flows through the limitations of recent works, outlining how the authors address these challenges with their proposed method.

2. I appreciate how each section of Section 2: Related Work clearly connects to the proposed method.

3. Sufficient experimental results support the contributions claimed.

4. I value the release of code which enhances transparency and reproducibility.
summary: The authors introduce Atlas Gaussian, a 3D generation approach grounded in diffusion models. 

First, they train a VAE to map 3D shapes into a compact latent space, which is then decoded through a two-branch structure into patch centers and features. These patch centers and features can subsequently be decoded into 3D Gaussian distributions.
Next, they train a Latent Diffusion Model (LDM) within the learned latent space to serve as the generation model. 

Atlas Gaussian offers three main advantages:
1. Using UV-based sampling within a unit square, it can generate a large quantity of 3D Gaussians efficiently.
2. It incorporates a nonlinear, learnable weight function through an MLP-projected positional encoding, enhancing representational power compared to traditional linear interpolation weights.
3. It is computationally efficient with minimal memory usage.

These benefits are substantiated by comprehensive experiments and ablation studies.
weaknesses: 1. I suspect the term “atlas” is intended in the same sense as in AtlasNet (L125~128). If so, adding an explanation would be helpful, as it is part of the method’s name but is not clarified in the text.

2. The method section is presented in a list-like format, making it difficult to read smoothly.

3. The clarity of the writing is lacking. Although the text contains many details, the explanations are somewhat vague, requiring readers to infer and interpret a great deal.

4. Some notations lack sufficient explanation. Although readers might deduce their meanings through context, clearer descriptions would be beneficial. (Also see questions below.)

5. Figure 3 lacks detail and does not clearly relate to the main text, particularly Section 3.2, Stage 1: VAE. For instance,

(1) The steps from $\bar{z} \rightarrow z_0$ are ambiguous. Is the process 

$z' = \mathnormal{CrossAttn}(\bar{z}, \mathcal{P})$

$z'' = \mathnormal{CrossAttn}(z', \mathcal{I})$

$z_0 = \mathnormal{MLP}(z'')$

($z'$ and $z''$ notations arbitrarily assigned as they are not explicitly defined)

correct? 

(2) My understanding is that $z_0$ of Figure 3 represents the end of the VAE encoder, while subsequent steps are part of the decoder. It would help to indicate these details explicitly.

6. The conditioning ($\mathcal{C}$ of Section 3.3) mechanism lacks detail; although some information is provided in the supplementary material, it remains insufficient.

7. An ablation study should be conducted to analyze the impact of the loss hyperparameters ($\lambda_r$ and $\lambda_{KL}$).

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: Do the authors plan to open source the model?

Can UV-based sampling potentially generate artifacts? (e.g. inconsistency or distortions)

How well does the proposed framework handle noisy or sparse data?
soundness: 3
strengths: The paper nicely integrates 3DGS with diffusion generation scheme and achieves state of the art results in 3D shape generation.

The use of Atlas Gaussians to represent 3D shapes as union of local patch makes it capable of capturing better details and higher scalability.

Decomposing the 3D shape generation into local Gaussian patches and using UV-based sampling to handle shapes of varying complexity.

The evaluation is thorough with reasonable qualitative and quantitative results and achieves SOTA results in 3D shape generation.
summary: The paper introduces a new framework called Atlas Gaussians Diffusion for 3D shape generation. It models the shape as a union of local patches as 3D Gaussians defined in UV space. This allows the generation process to generate theoretically unlimited number of points while capturing more underlying details of the 3D shape. Extensive quantitative and qualitative experiments demonstrate their results are better than the current SOTA.
weaknesses: While in 4.4 discuss about memory consumption and # of patches, the paper is lacking thorough comparison on inference times (esp. with the increase of # of 3DGS) to further justify the value of the framework.

How could the current approach address gap for even finer details of the shapes as Gaussian representation struggles in fine details and often results in blurry effect. Is it only a matter of the # of Gaussians or patches? 

The author mentioned "We take inspiration from the literature that 3D-based neural networks learn better 3D features for recognition and outperform multi-view based networks that leverage massive 2D labeled data." without any citation. There are arguments on both sides about overall performance with both approaches. This sentence seems to be subjective.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the Weaknesses section, where the primary questions have been outlined. Additional points are provided below -

1. How are the local patches parameterized as a sequence of feature vectors, and how is the learnable function designed to decode 3D Gaussians from these feature vectors?

2. Why is UV-based sampling used instead of directly generating an infinite number of 3D Gaussian points? Why is it necessary, and does it help in preserving the topology of the synthesized 3D objects?

3. How does a large number of 3D Gaussians contribute to achieving high-quality details?

4. How transformer-based decoder help to remove all the existing limitations posed by a vanilla decoder (other than the transformer decoder in LN3Diff)? Are there any strong quantitative and qualitative ablations?

5. What is the corner-encoding scheme?
soundness: 3
strengths: **S1.** Methodological development is clear and well presented.

**S2.** The paper is very well written and has strong motivation.

**S3.** Consideration of generation 3D Gaussians in latent diffusion model is unique and well-motivated.
summary: The authors proposed a representation model for generating 3D Gaussians from local patches, named Atlas Gaussians. They parameterized each patch as a sequence of feature vectors and designed a learnable function to decode 3D Gaussians, followed by the integration of UV-based sampling to generate large and theoretically infinite Gaussian points. To learn the Atlas Gaussians, the authors employed a latent diffusion model in the latent space of a VAE.
weaknesses: While this paper explores a unique perspective on 3D generation, it primarily lacks proper experimental support and validation. It is difficult to discern the specific novelties and contributions (considering the three contributions outlined in L102-L107) that this paper offers compared to recent SOTAs, e.g., LN3Diff and others. Although the current methodological development is generally clear, the contributions are still not well-supported by the qualitative findings and significantly improved quantitative results reported in the paper.


**W1. Latent Diffusion Novelty:** I believe the variational diffusion model is not a novelty in terms of technical development. Besides, the authors directly utilized the EDM algorithm as their latent diffusion. What makes the integration of the latent diffusion model a unique/novel contribution other than working with a different data type such as 3D Gaussians? Please highlight specific contributions with details in the revised version.


**W2. Experimental Validation:** 
*(i)* Going through the generated samples reported in the LN3Diff and this paper, it is difficult to understand the effectiveness of the proposed model. The qualitative comparison with the LN3Diff model in Fig. 5 raises concerns regarding whether the baselines have been trained to their optimal or not. Besides, for most of the pictorial depictions, I can guess that either the sampling or training has not been done properly for all the SOTA, specifically in Fig. 6. In the SOTA baselines papers, the structure of the synthesized images seems to be well preserved. Then why the authors could not achieve that? *(ii)* Did the authors run all the baselines, and try to reproduce their results? Or they just copied the FID/KID numbers from the papers? *(iii)* I would suggest reporting the mean along with std to see the robustness of the scores. *(iv)* Inference times in Tab. 2 are not comparable with GVGEN and LN3Diff as they carried out their experiments on different GPU setups. *(v)* As one of the contributions of this paper is using a transformer-based decoder, I would suggest the authors show a very brief ablation carrying out the experiments on a vanilla decoder. Also, compared to the LN3Diff which used a transformer-base decoder, what is the uniqueness of their proposed architecture?


**W3. Loss function and optimization:** According to the derived loss function, the optimization process seems to be quite complex to converge to the optimal stage as one needs control around 10 distinct losses, including the KL divergence term. I suggest the authors to shed some light onto the complete optimization process other than simply denote large number of losses with weighting parameters.


**Minor.** Please cite necessary papers that are relevant to the proposed methodology, other than just scrapping eight pages of references which includes papers that do not fall into the problem of this paper.


In its current format, I believe the paper falls below the threshold for acceptance. The score reflects an evaluation of the problem definition, motivation, and presentation. However, I am open to raising my score if the above concerns are properly addressed. Besides, I request the authors to specifically highlight the methodological contributions and novelties (compared to the current SOTA), beyond the focus on generating 3D Gaussians.

Meta-review: This paper proposes a novel framework, Atlas Gaussians Diffusion, for 3D shape generation. The method represents 3D shapes as a union of local patches defined as 3D Gaussians in UV space, enabling the generation of theoretically unlimited points while preserving finer details. The integration of the 3D Gaussian Splatting (3DGS) method with a diffusion generation scheme results in a scalable and detailed shape generation process. Extensive quantitative and qualitative experiments validate the framework's superiority, achieving state-of-the-art (SOTA) performance in 3D shape generation.

Reviewers unanimously praised the paper, highlighting the innovative integration of 3D Gaussian Splatting with diffusion-based generation.
The use of Atlas Gaussians to represent shapes can also effectively capturefine details and ensuring scalability.
Comprehensive evaluations with strong quantitative and qualitative results that establish the framework's SOTA performance.

The paper introduces a well-validated, innovative approach to 3D shape generation, combining methodological novelty with strong experimental results. The unanimous positive feedback supports its acceptance as a significant contribution to the field.

Decision: Accept (Spotlight)