Atlas Gaussians Diffusion for 3D Generation
Authors: Haitao Yang
Yuan Dong
Hanwen Jiang
Dejia Xu
Georgios Pavlakos
Qixing Huang
Venue: ICLR 2025 Spotlight (2025)
Keywords: 3D generation
diffusion
3D Gaussian Splatting
Primary area: generative models

Abstract: Using the latent diffusion model has proven effective in developing novel 3D generation techniques. To harness the latent diffusion model, a key challenge is designing a high-fidelity and efficient representation that links the latent space and the 3D space. In this paper, we introduce Atlas Gaussians, a novel representation for feed-forward native 3D generation. Atlas Gaussians represent a shape as the union of local patches, and each patch can decode 3D Gaussians. We parameterize a patch as a sequence of feature vectors and design a learnable function to decode 3D Gaussians from the feature vectors. In this process, we incorporate UV-based sampling, enabling the generation of a sufficiently large, and theoretically infinite, number of 3D Gaussian points. The large amount of 3D Gaussians enables the generation of high-quality details. Moreover, due to local awareness of the representation, the transformer-based decoding procedure operates on a patch level, ensuring efficiency. We train a variational autoencoder to learn the Atlas Gaussians representation, and then apply a latent diffusion model on its latent space for learning 3D Generation. Experiments show that our approach outperforms the prior arts of feed-forward native 3D generation. Project page: https://yanghtr.github.io/projects/atlas_gaussians.

Published as a conference paper at ICLR 2025
ATLAS GAUSSIANS DIFFUSION FOR 3D GENERATION
Haitao Yang Yuan Dong"
Hanwen Jiang' Dejia Xu' Georgios Pavlakos'
Qixing Huang
The University of Texas Austin
AAlibaba Group
100K Generated Points
old, wooden cart mossy elements
Star Wars Wing Fighter Starfighter
Figure 1: We propose the Atlas Gaussians representation for 3D generation. Our method supports both unconditional (left) and conditional (middle) generation with great diversity. With Atlas Gaussians, we can generate sufficiently large, and theoretically infinite, number of 3D Gaussian points. To demonstrate this, 100K Gaussian points generated by our method are shown (right). Additionally, we iteratively sample 32 patches of the shape, displaying each in one of four sub-figures (right).
ABSTRACT
Using the latent diffusion model has proven effective in developing novel 3D generation techniques. To harness the latent diffusson model, a key challenge is designing a high-fidelity and efficient representation that links the latent space and the 3D space. In this paper, introduce Atlas Gaussians, novel representation for feed-forward native 3D generation. Atlas Gaussians represent shape as the union of local patches, and each patch can decode 3D Gaussians. We parameter- ize patch as a sequence of feature vectors and design learnable function to decode 3D Gaussians from the feature vectors. In this process, we incorporate UV-based sampling, enabling the generation of sufficiently large, and theoreti- cally infinite, number of 3D Gaussian points. The large amount of 3D Gaussians enables the generation of high-quality details. Moreover, due to local awareness the representation, the transformer-based decoding procedure operates on a patch level, ensuring efficiency. We train variational autoencoder to learn the Atlas Gaussians representation, and then apply a latent diffusion model on its latent space for learning 3D Generation. Experiments show that approach outperforms the prior arts of feed-forward native 3D generation. Project page: Inttps://yanghtrr wwwpgghhtttttt.tt.....ee io/pro jectss at as gaussians
INTRODUCTION
3D generation has become increasingly important in various domains, including virtual reality, gaming, and film production. Recent advances in diffusion models (Ho et al. 2020 Song al. 2020) have improved the quality of 3D generation, offering superior performance over previous methods, such variational autoencoders (VAEs) (Kingma Welling 20133 Mo et al. 2019) and generative adversarial networks (GoNelGooo et al. 2014 Gao et al. 2022).
Equal Contribution

---

Published as a conference paper at ICLR 2025
Despite progress, the effectiveness of 3D generation still falls short compared to 2D generation models. The robust performance in 2D generation is largely due to the effective integration of VAEs with latent diffusion models (LDMs) (Rombach al. 2022). A primary challenge hindering the complete success of this paradigm in 31) generation is the development of a high-fidelity 31)
representation that can he efficiently embedded into low-dimensional latent space. Pioneering efforts
have applied diffusion models to various traditional 3D representations, such as point clouds (Zeng
al.
2022
Zhou et al. 202T] Luo & Hu 202TD, meshes (Liu al. (023c), occupancy fields (Zheng
et al.
2023
Zhang et al. (023a) and signed distance functions (Cheng et al. 2023 Li et al. 2023b]
Zhang et al. 2024). However, these approaches often focus solely on modeling geometry without
considering the appearance attributes. More recently, notable attempt (Lan et al.
(024) has designed
a VAE that uses volume rendering techniques (Mildenhall et al. 2020) to incorporate appearance modeling. However, volume rendering presents inherent limitations, including slow rendering speeds and constrained rendering resolutions. To overcome these limitations, we have developed a VAE that leverages the latest 3D Gaussian representation (Kerbl et al. 2023), which significantly improves both the quality and speed of rendering.
Designing a VAE based on 3D Gaussians presents considerable challenges. The first challenge lies in creating an efficient decoder capable of mapping oww-imensional latents to 3D Gaussians. Existing methods for 3D Gaussian decoding focus predominantly on reconstruction tasks and typically lack an information bottleneck design (Tang et al. [024] Yinghao et al., (024), thus failing to provide low- dimensional latent directly. Alternativeyy. they often require multiple complex and interdependent components (Xu et al. [024al Zou et al.) 2023). The second challenge involves generating a sufficiently large number of 3D Gaussians efficiently, since high-quality rendering necessitates an
adequate quantity of these Gaussians. Some current methods (Xu et al. 2024a Zou et al. 2023)
address this by employing additional complex voint-upsampling networks to increase the number of 3D Gaussians, which inherently limits the number of Gaussians that can be generated. Other techniques (Tang et al. 2024. Yinghao et al. 2024) utilize image representations to generate large number of 3D Gaussians. However, for all these methods, more network parameters are usually required as the number of 3D Gaussians increases.
To address the challenges of designing VAEs for 3D Gaussians, we propose Atlas Gaussians, new representation for 3D generation. This representation is inspired by surface parameterization (Floater & Hormann| (005). foundation technique in many graphics applications where surface attributes are sampled and stored in a 2D texture map. Specifically, Atlas Gaussians model the shape as a union of local patches, with each patch decoding 3D Gaussians via UV-based sampling. By parameterizing 3D Gaussians in the UV space, we can easily generate a sufficiently large, and theoretically infinite, number of 3D Gaussians. Unlike traditional surface parameterization approaches, the UV mapping in Atlas Gaussians is learned end-to-end. A significant advantage of Atlas Gaussians is that the sampling process does not require additional network parameters as the number of 3D Gaussians increases.
We design a transformer- based decoder to map low- di-eeniioiin latents to Atlas Gaussians. This decoder specifically structured to disentangle geometry and appearance features, facilitating faster convergence and improved representation capabilities. Using the local awareness of Atlas Gaussians, we also reduce computational complexity by decomposing the self-attention layers. Finally, the latent space learned by our VAE can be applied to existing latent diffusion models efficiently.
Note that in contrast to the main approach in 3D generation that uses the multi-view representa- tion (Wang& & Shi, 2023 Long et al. 2023 Liu et al.) 2023d Shi et al. (023b), our approach is inherently 3D-based. Therefore, a key advantage is that we do not need to address the challenging multi-view consistency issue associated with the multi-vies representation. Moreover, the rendering module of Atlas Gaussians allows representation learning from images.
In summary, we make the following contributions.
We propose Atlas Gaussians, a new 3D representation that can efficiently decode sufficiently large and theoretically infinite number of 3D Gaussians for high-quality 3D generation.
We design new transformer- bassed decoder efficiently map low-dimensional latents Atlas Gaussians, using separate branches to disentangle geometry and appearance features.
We pioneer the integration of 3D Gaussians into the VAE + LDM paradigm, demonstrating superior performance standard 3D generation benchmarks.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
3D representation. 3D reconstruction and generation benefit from different 3D representations by
leveraging their unique properties. These representations include explicit representations (Wu et al..
2016; Mittal et al.] 2022; Ren et al. 2024 Zeng et al. 2022] Zhou et al. 2021 Luo & Hu, [222] I
Sun et al. 2020a| Xie et al. 202 Yangg
Yang et al. 2019; Sun et al. 2020b Achlioptas et al. 2018]] Fan
et al.
2017
Liu et al. 2023e Nash et al. [2020] Siddiqui et al. [2033] Groueix et al. [018 Chen
ct al. al. 2024) and implicit representations (Park al.] 2019 Chen & /Zhang [019]] Mescheder et al.
2020
Kerbl et al. 2023; Tang et al.]
2024] Yinghao et al. [2024] [224|z00 Zou et |003] Szymanowicz
2019
Li et al. 2023b; /Zhng et al. 2022 Shue et al. 2023 Jiang et al. [2022]] Hui et al. [0022] Yan
et al.
2022 Mildenhall et al. |020 /Zhng et al.] [022] C Chan al. [022]] Gu et al.) [023 Chen
cC al.
2023a Cao et al.
2024
Müller et al. [2033] Watson et al. 2023). In this paper, we focus on 3D
Gaussians (Kerbl
2023
Tang et al. [024]] Yinghao al. 2024; Zou et al., 2023] Szymanowiiz
it al.
[024]] He et al. 2024),
which possess high-quality rendering procedures that allow learning
from image supervistons. However, existing results mainly focus on reconstructing 3D Gaussians.
Our goal is to push the state-of-the-art in generative 3D Gaussians.
Our method is also related to the Atlas representation initially proposed by AtlasNet (Groueix et al., 2018) and its subsequent extensions (Deprelle et al. 2019 Liu et al.] 2019, Feng et al.] 2022). AtlasNet models the shape as a union of independent MLPs, thus limiting the number of patches to few dozen. In contrast, our proposed Atlas Gaussians model each patch using patch center and patch features. This efficient encoding allows us to generate a significantly larger number of patches, providing stronger representation capabilities. Additionally, use a transformer to learn this representation instead of an MLP, resulting in better scalability.
Diffusion models. Diffusion models (Sohl-Dickstein
al., [005 [Son et al. 202 1 2020) have been dominant for diverse generation tasks, including image (Ho & Salimans 2022 Zhang et al.
2023b Podell et al.,
2023
Rombach al. 2022), video (Ho et al. 2022b a Blattmann et al. 2023)
audio Huang et al. 2023
Kong et al. 2020 Liu et al. 2023a) and text (Li et al. [002]] Gong et al.
2022
Lin et al. 2023b). The success of these models typically follows the VAE + LDM paradigm...
Although pioneering efforts (Zeng et al. 2022 Zhou et al. 2021: Luo & Hul 2021 Liu et al. 2023e
Zheng et al. 2023) Zhang et al. 2023a Cheng et al. 2023 3i al. [023b] Jun & Nichol (023)
have attempted to apply this paradigm to 3D, the problem remains unsolved and has not achieved
the same level of success. We argue that one of the main reasons the need for an efficient VAE to
represent high-quality 3D content, which is the key contribution of this paper.
3D generation. 3D generation methods can be classified into two genres. The first optimization- based methods (Jain et al. 2022] Sun et al.] 2023 Wang et al., 2023 Lin et al. 2023a| Wang et al. 2024; Chen et al. (023b), which are time-consuming due to per-shape optimization. For example, DreamField (Jain et al. 2022) uses CLIP guidance. DreamFusion (Poole et al., 2022) and
SJC (Wang et al. 2023) introduce 2D diffusion priors in different formats. Magic3D (Lin et al. 2023a) employs coarse-to-fine pipeline to improve convergence speed. Prolificdreamer (Wang et al. 2024) uses variational score distillation to enhance generation fidelity. Fantasia3D (Chen et al. 2023b) disentangles geometry and texture achieve higher quality generation.
The second genre involves training a generalizable feed-forward network to output 3D content, which allows for fast 3D generation but often with less details. One direction modeling techniques ( (VAE, GAN, normalizing flow (Rezende & Mohamed, 2013), autoregressive
apply existing generative
model (Bengio et al.] [2000] Graves 2013) and diffusion model) directly on various 3D representations. Another direction (Xu et al.) 2024b)
Liu Liu et al. 2024 2023b| [Shi et al. 2023a Hong et al.) 2024] 2024) uses 2D as intermediate representation, integrating 2D diffusion models (Ho et al.
Tang et al.
2022a Shi et al.) 2023b). For example, Instant3D (Li et al. 2023a) trains a diffusion model to
generate sparse multi-view images and uses a reconstruction model to derive the 3D shapes from the 2D images. VFusion3D (Han al. 2024) and V3D (Chen et al. 2024) use video diffusion models to improve consistency between generated images. However, these methods rely on 2D representations and often suffer from multi-view consistency issues. In this paper, we push the state-of-the -hrtartt in the first direction by developing a novel representation of Atlas Gaussians. Our representation is efficient and allows for learning 3D generative models from images.
3

---

Published as a conference paper at ICLR 2025
A
g
systu Og Su
4 %
Patíis level Dacoding
Gausscan Splatting
Query Point
&
Patches & Features
3D Gaussians
Rendered
Mar
(x, Fia
Single Gaussian
Images
Map
Patch a;
Figure 2: (Left) Atlas Gaussians A model the shape as union of patches, where each patch can decode 3D Gaussians. (Right) Each patch a, is parameterized by patch center æ; and patch features f, and h. The 3D Gaussians are decoded via the UV-based sampling.
METHOD
In this section, we first introduce our Atlas Gaussians representation (Sec. 3.1 1. Then we introduce how we learn VAE to connect the 3D space with the latent space (Sec. 3.2). Finally, we introduce how we learn generative model using latent diffusion learned latent space (See. 3.3)
3.1 ATLAS GAUSSIANS REPRESENTATION
In 3D Gaussian Splatting Kerbl et al. 2023). each 3D Gaussian g can be parameterized with center E R3, scale s E 3 rotation quaternion E RR, opacity R and color с E R To achieve high-quality rendering results, typically a sufficiently large number of 3D Gaussians are required.
The key idea of Atlas Gaussians is to represent the shape union of M local patches A {a: J.M where each patch a; can decode 3D Gaussians through UV-based sampling. As shown in Figure 2 we parameterize each local patch a; == (æ;, f, h;)) with patch center æ; R3, the geometry features f; E gaxd and the appearance features h; IR 4xd More specifically, we parameterize the geometry and appearance features the features at the four corners of the local patch in the UV space. We denote f; = (f f2 fa f(a) and h; = (ha,1 h,2, h,3, h,1). This type of feature disentanglement facilitate more effective learning (Gao al. 2022 Zou al. 2023 Xu al. (024a) We the feature f: to decode Gaussian positions while using h; to decode the rest of Gaussian attributes. We also assign 2D coordinate 4g E R² to each feature vector fij and hig positional embedding. where u, 0), u,2 0), u.3 = 11, u, (0, 1). Vi.
ii=1
Note that this representation motivated by the concept of UV-map, in which the four corners describe the corners of the rectangular parameter domain. As we shall discuss later, the features f; and h. are learned end-to-end with 3D Gaussians. This approach takes advantage of end-to-end learning, while the specific network design promotes learning better features.
For generation, we randomly sample query points in the predetined unit square UV space for each patch a, Each point is then decoded into a 3D Gaussian. Given query point qú E [0, 1) we map 2D coordinate gi the center Gaussians as:
Mi
<( Uss f(). f.).))
where Φ is the mapping function, which takes the query point location qıj. the predefined location of patch feature vectors u, and the geometry features f, as inputs. We implement the mapping function using interpolation in the 2D space:
My = MLP
w w(a uak ax) f(x) ) wis
where w is the weight function of the four corners and the Gaussian location and the patch center æ;
MLP to decode the residual between
One design choice for is the bilinear interpolation weight function, which has been widely used in feature decoding Muüller et al. 2022). However, these linear weights purely based on coordinates have limited representation ability. Inspired by (Zhang al. 2023a), design powerful
4

---

Published as a conference paper at ICLR 2025
Query Point 4i
Patch
Feature
Decoder
ViT
Patch Center
Lalent
CA
CA
Upsample
Decoder
A
g
Image
Figure 3: The proposed VAE architecture. CA denotes the cross-attention layer. For simplicity, the variational component of the VAE is omitted. The latent Zo is used for latent diffusion.
weight function defined on both coordinates and features:
w(qij- Uike fsk)
w u( Uik: f f() Ei= in Ua fa)
(3)
and w Oijs Mit, fa) = " (Futwa(ua))/a
(4)
where RR2 Rd is the sinusoidal positional encoding function with MLP projection.
Similarly, the remaining Gaussian attributes are decoded by the function to:
(Sij. Fej; Oij, Cij) (gij. Ui h.) MLP
Σ w w Wike hak) ha),
The benefits of Atlas Gaussians representation are three-fold. First, through UV-based sampling in a unit square, Atlas Gaussians enable easy generation of sufficiently large number of 3D Gaussians. They also possess the potential to generate a variable and theoretically infinite number of 3D Gaussians. Second, Atlas Gaussians utilize non-linear learnable weight function based on the MLP projection of the positional encoding, which has a stronger representation ability than the existing linear interpolation weight. Third, Atlas Gaussians computation efficient with low memory overhead. Another important property of Atlas Gaussians is that they do not require extra network parameters when scaling up the number of generated 3D Gaussians. We also provide an ablation study in Section 4.4 to validate the nice properties of Atlas Gaussians.
3.2
STAGE 1: VAE
We design a VAE link the latent space and the 3D space. The overall pipeline shown Figure
Encoder. Following the latent set representation in (Zhang et al. 2023a). the encoder takes shape information as input and outputs a latent set 2g E xx where
the size of the set in the latent set representation and do is the latent dimension. The shape information includes point cloud P = (p. E R3} and sparse view RGB images I = {I, E H x3). We encode the location of points into positional embeddings, resulting in features in iPIxd Meanwhile, a VIT network (Dosovitskiy et al., 2021 Oquab al.) (2024) embeds each image into features of shape Rhxarxd resulting in the features of all images in R x(xxxxxwxxd We then initialize the latent features e Raxd as the encoding of the points sampled using the farthest point sampling (Zhang al. 2023a) and use transformers with cross-attention to aggregate input shape information. After feature aggregation, we use an MI.P to map the features to dower-dimensional space Raxd Raaxdo facilitating efficient latent diffusion. The process is summarized as follows:
z' CrossAttnzz, P), " CrossAutn22' (I), z" SelfAum(z")) 20 MLP(="), (6)
In this notation, we omit the feed forward MLP of the transformers for simplicity. Intuitively, the query is updated by aggregating the input point features and image features.
Note that attention is invariant to the permutation of features. We also avoid pose-dependent opera- tions (Jang al. 2024 Jun & Nichol 2023) in our design enhance generality and extendability.
5

---

Published as a conference paper at ICLR 2025
Decoder. The decoder recovers patch features A { f h4)) from the latent code zu- In the decoder, we upsample the latent code, decode the patch centers ( t= and then decode the patch geometry and appearance features {(f h+
M
The upsampling module designed increase both the length and the feature dimension of the latent code, from E Ruxdo to 21 E I Mxd, In detail, we first use a learnable query y to aggregate information from the latent code zo using transformers with Pross-attention channel dimension to d using self-attention transformers, leading to the characteristic ž E Raxd
Then, we increase its
We then use an ML.P to increase the feature dimension of 21 into M d, leading latent features in n
Rax # We reshape the features into 22 E R xd which is pixel shuffle process (Shi et al. 2016 Xu et al. 2024a). We then use another set of transformers with self-attention to obtain the output latent features z1. This process is summarized as follows:
2 where y is initialized using a Gaussian distribution. Again, we omit the feed-forward MLP in the transformers for simplicity. We then decode the patch centers using transformers with self-attention:
zo)),
PixeIShuffe
= Selffttt((z)) (7)
a M = SelfAutn(z:).
We use two-branch module decode geometry and appearance features {(f h:)) from upsampled latent code Z₁. We demonstrate its architecture in Figure Take the branch for decoding the geometry features {fi) i=1 as an example. We use another upsampling module to map z E R
to 2 R Mx8xd, where 3 4, corresponding to the geometry features of the four corners for each patch. We then use transformers with self-attention to refine the upsampled features to get f+.
Specifically, apply computational decomposition to the self-attention layers, naive self-attention leads to a ( d) complexity due to the long sequence length of M3. We apply local self- attention to the features that belong to each local patch, reducing the complexity to O(8² Md). The features of different local patches are updated independently. This design ensures local awareness of Atlas Gaussians decoding. To further ensure global awareness, we repeat and add the global features to the local patch features.
After obtaining the Atlas Gaussians A (xx, f., h4) =1 from the VAF, we can decode 3D Gaussians following Eq. 2 and Eq.
x
Zz
Training. Similar to existing methods (Lan al. 2024 Xu al. 2024a Zou et al. 2023), we utilize a 3D dataset for supervision. We first regularize the patch center to adhere to the 3D surface geometry:
SA
Lvemer = (cv((x; MM Par) + )
(9)
SA
where Pat is the ground truth surface point cloud, LCD and LEMD are Chamfer Distance (CD) and Earth Mover's 's Distance (EMD), respec- tively. Similarly, we also supervise the centers of all 3D Gaussians with
Z
SA
M.S (S) = (co((,,,)) 1i Por) + LEND (1M MullEt== Par).
M.S
Zn
h
(10)
M.S represents the generated point cloud by sampling where [1g)]
S points in i supervision, we can generate point clouds with different resolutions by varying S, thanks to Atlas Gaussians' ability to dynamically generate a variable number of 3D Gaussians. Note that patches may overlap, similar to AtlasNet. Leenter and (S) encourage the patches to distribute more uniformly across the surface, thereby making better use of the 3D Gaussians.
of the M patches. When computing the loss for
Figure 4: In the patch feature decoder, global features are broadcast and added to the lo- cal features. SA denotes the self-attention layers.
While surface points independent samples of the continuous geometry. 3D Gaussians interde- pendent because one 3D Gaussian can affect the attributes of its neighbors during alpha blending. To ensure consistency and achieve deterministic results, during rendering we employ uniform grid sampling in the UV space instead of random sampling, resulting in N = a² M 3D Gaussians. We use the differentiable renderer from (Kerbl al. 2023) to render views of RGB, alpha, and depth images, which is supervised by mean square error:
Crender LMSE ( rihh righ) LASE alpha; alpha) LMSE ( deptts I depth),
(11)

---

Published as a conference paper at ICLR 2025
where Ligh. I aigha- and Lepph are the predictions. I Lggs I alpha- and I depth the ground truths. To improve the visual fidelity, we also employ the LPIPS loss (Zhang et al. 2018 Tang et al. 2024) LiPIPs Lig,, lgb)) The total loss function for the VAE is:
Grotal
Leenter
Σ (s(() + Ay (Crender + LLPIPS) + A S=(14)
(12)
where CKL is the Kullback-Leibler divergence, Ay and AKL are the loss weights.
3.3 STAGE 2: LATENT DIFRUSION MODEL.
The proposed VAE provides low- www-dimensional latent code that can be mapped to Atlas Gaussians. We employ EDM (Karras architecture as (Zhang et al. 2023a), which consists of series of transformer blocks:
2022) for latent diffusion. Our denoising network follows the same
(i) CrossAttn (SelfAutn( (1 ()). i i == I....,!
(13)
where 20)) Zo is the initial input the network, / the number of blocks. In unconditional generation, we designate C as a learnable parameter. For text-conditioned generation, C is set as the CLIP embedding (Radford et al., 2021) of the input text prompts. Since the LDM network design is not our main contribution, we provide the implementation details in Appendix A.3
4 EXPERIMENTS
We first introduce our experimental setup, and then show the results for both unconditional and conditional generation.
4.1 EXPERIMENTAL SETUP
Following most existing methods (Gao et al. 2022 Müller et al. 2023 Chen et al. 2023a Lan et al. 2024), we benchmark unconditional single-category 3D generation on ShapeNet (Chang et al. 2015). We use the training split from SRN (Sitzmann et al. 2019), which comprises 4612, 2151, and 3033 shapes in the categories Chair, Car, and Plane, respectively. We render 76 views for each shape using Blender (Community) 2018) with the same intrinsic matrix (Cao et al. 2024 Lan al.) 2024).
Fréchet Inception Distance (0K) and Kernel Inception Distance (KID @ 50K) are used for evaluation. In addition, we experiment with text-conditioned 3D generation on Objaverse (Deitke
et al. 2022). We use the renderings from G-buffer Objaverse (Qiu et al., (203) and the captions from Cap3D (Luo et al.) (2023). Due limited computational resources, we select a high-quality subset with around 18K 3D shapes. We use CLIP score (Radford et al. 2021 Hessel et al., 2021). FID and KID for evaluation. Please refer to Appendix A.1 for more implementation details.
4.2
UNCONDITIONAL. 3D GENERATION
Table
presents the quantitative comparison between method and baseline approaches. Our method outperforms all baseline approaches, including EG3D (Chan et al., 2022). GET3D (Gao et al.
2022), DiffRF (Müller et al.) 2023), RenderDiffusion (Anciukevičius al. 2023). SSDNeRF (Chen et al. 2023a) and LN3Diff (Lan et al.) 2024). We also include the qualitative comparison with LN3Diff in Figure Our results demonstrate significant improvements
LN3Diff, particularly in
the ShapeNet Chair category, which features greater geometric, structural, and textural complexity.
The results highlight the robustness and efficacy of our approach.
4.3 CONDITIONAL 3D GENERATION
We evaluate our method and baseline approaches (He al.) 2024 Lan et al. 2024 Tang al. 2024, Jun & Nichol 2023) on text-conditioned 3D generation using Objaverse. The qualitative results presented i 6. where all text prompts soureed from the original baseline papers. As shown in Figure 6 the image- based generalizable 3D reconstruction method (Tang et al.) 2024) sometimes produces shapes with significant artifacts, primarily due its reliance multi-view
7

---

Published as a conference paper at ICLR 2025
LN3D
Outrs
LN3DRR
Ous
Oue LN3DMI
Figure 5: The comparison between LN3Diff and our method on three ShapeNet categories.
Table 1: Evaluation of single-category unconditional generation ShapeNet.
Chair
Car
Plane
Method
50D
KID@ (0K(S)
FID@ 50K
KID&SOK(SS
FID@50K
KID@ 50K(S)
2022
GET3D G99 al 20/4
35.33
1.5
41.41
1.8
26.80
1.7
DiffRF
99.37
75.09
5.1
101.79
RenderDimusmn Aenekkekiinu et 2023
53.30
6.4
46.50
4.1
43.50
5.9
SSDNeRF Chen
3.0
47.72
2.8
21.01
LN3Diff
16.90
0.47
17.60
0.49
8.84
0.36
8.09
generation network, which frequently suffers from multi-view inconsistency. In contrast, our method learns directly from 3D data, enabling the generation of consistent novel views. Additionally, our method produces higher-quality 3D shapes compared to Shap-E and LN3Diff, both of which use the VAE LDM paradigm. However, these methods depend on volume rendering during training, which is typically restricted to low resolution. Our approach instead leverages the more efficient 3D Gaussian representation. Compared GVGEN, our generated shapes exhibit finer details, owing to the proposed Atlas Gaussians representation, which is capable of decoding significantly larger number of 3D Gaussians. Quantitative results are provided in Table where our method achieves the best performance across all metrics and has the shortest inference time. This improvement is due to our efficient decoder design, which effectively links the low-dimensional latent space with 3D space. Additional results can be found in Appendix
(Left), we present text- conditioned generation results with different random seeds, demonstrating that method produces highly diverse outputs. In Figure 7 (Right), we show that our model can be robustly controlled using different text prompts. The generated results also significantly different from their nearest. neighbors in the training dataset, highlighting the model's ability to generate novel content.
In Figure 7 we provide detailed analysis of our method. In Figure 7
Table 2: Evaluation of text-conditioned 3D generation Objaverse.
GVGEN He et 20224
LN3DMI LLn et al, 2024
LGM Tung et a
2024
Shup E Fum & Nichol 5023
Ours
CLIP Scoce (
27.33
27.21
29.02
30.22
30,66
FIDG66
132,4
123.8
117.0
114.5
109,5
KIDe>k
Inference (V100)
8

---

Published as a conference paper at ICLR 2025
GVGEN
LN3DIN'
Shap-E
GVGEN
LN3Dif
Shap-E
A AA
wooden stool with seat and legs.
ship.
A blue plastic chair.
Pixelated model of wooden ship with sails.
wooden worktable.
A wooden ship with most.
a wooden bedside side
White scooter model.
gray sectional sofa with pillows and wooden legs.
yellow and black forklift truck.
light purple teddy
A Yellow Chevrolet Comaro sports can.
goldfish.
red, white, and blue Lego airplane with wheels.
white and red Lego toy chicken.
Space War Tie Fighter.
Figure 6: Comparison of text-conditioned 3D generation on Objaverse. From left right: GV-
GEN (He et al. 20244, LN3Diff (Lan et al.) 2024). LGM (Tang al., 2024). Shap-E E (Jun & Nichol and our method. All text prompts are sourced from the original baseline papers.
2023
Generation
aoocin cheir. wozder wheels.
woodon child's with blus sheets
bed with red sheats.
childs with sheats.
child's with white shoets
with slack sheets.
Geseatio
modern upholstered thair.
tharacter resenbling hipp.
resembling cat.
resembling darkey.
resemblnng horse.
resembiiig coe
pixxleted mocel of with harm lood, resembling deer with hores
Figure 7: (Left) Our generated results demonstrate significant diversity. (Right) Our generated results align closely with the text prompts, allowing for strong controllability. In the second row of each group, we present the nearest neighbors (NN) from the training dataset.

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study on the number of 3DGS and the number of patches on Objaverse. (LPIPS). / MEM()
Table Ablation study on network design
evaluated on ShapeNet with PSNR.
#patches
#3DGS=8K
#3DGS=32K
#3DGS=<100K
Atlas Gaussians (Full)
26.56
512
0.068 / 1.2G
(a) linear weights
26.13
1024
0.060/ 1.3G
disentangle
25.77
2048
0.072/1.3G
0.063 1.4G
0.058 / 1.7G
no global feature
26.18
4096
0.06
0.057 2.3G
(d) decode s,r.o from f:
26.55
4.4
ABLATION STUDY
This section presents an ablation study on different components of our network.
Number of 3D Gaussians and number of patches. In Table 3] we analyze the effects of both the number of patches and the total number of 3D Gaussians on LPIPS and memory usage (MEM, which represents the additional GPU memory required for each unit increase in batch size). The results indicate that increasing either the number of patches the number of 3D Gaussians improves LPIPS. When the number of 3D Gaussians fixed (32k or 100k), increasing the number of patches from 2,048 to 4,096 results in 0.1-0.2 improvement in LPIPS. However, when the number of patches is fixed (2,048 or 4,096), increasing the number of 3D Gaussians from 32k to 100k leads to a more significant improvement in LPIPS while requiring less additional GPU memory. Importantly, the network parameters remain unchanged despite the increase in 3D Gaussians. This comparison demonstrates that increasing the number of 3D Gaussians through Atlas Gaussians decoding is more efficient and effective than increasing the number of patches. Notably, the Atlas Gaussians representation combines explicit patch generation with implicit interpolation generate 3D Gaussians, offering a balanced trade-off between compact implicit and fast explicit representations.
Learned weights for feature decoding. In Table we ablate the VAE network design on the ShapeNet validation set using the PSNR metric. The table shows that replacing our learned weight function with bilinear interpolation weights leads to a performance drop the VAE. This indicates that our learned nonlinear weights have stronger representation capabilities than linear weights.
Disentangle geometry and texture features. Atlas Gaussians use separate branches to learn geometry and appearance features. To validate the effectiveness of this design, experimented with an alternative network version where a single set of features is shared for both geometry and appearance. As shown in Table 4 performance decreases considerably with this shared feature approach. This shows that our design facilitates more effective learning. Note that both the geometry and appearance features generated from the shared latent žu. and disentanglement is performed during feature generation rather than in the latent space.
Using global features. As shown in Table 4 removing the global feature from ž₁ in the patch feature decoder leads to a performance drop. This outcome is expected, as the global features provide essential context that complements the local features.
Parameter decoding scheme. In Gaussian Splatting, opacity, scale, and rotation also influence the geometry. We experimented with decoding opacity, scale, and rotation from the geometry features and found that the performance is nearly identical to when these attributes are decoded from the appearance features. This result indicates that the key factor is decoding the Gaussian centers and RGB colors using distinct, separate branches.
CONCLUSION
This paper introduces Atlas Gaussians, a new representation for feed-forward 3D generation. Atlas Gaussians enable the efficient generation of sufficiently large and theoretically infinite number of 3D Gaussians, which crucial for high-quality 3D generation. We also designed a transformer- based decoder to link the oow-dimesiional latent space with the 3D space. With these innovations, we pioneer the integration of 3D Gaussians into the VAE + LDM paradigm, achieving superior performance and producing high-fidelity 3D content.
10

---

Published as a conference paper at ICLR 2025
Acknowledgment. We thank Yushi Lan for his invaluable assistance in the evaluation. Qixing Huang acknowledges the support from NSF IIS-2047677, IIS-2413161, and GIFTs from Adobe and Google.
REFERENCES
Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas J. Guibas. Learning repre- sentations and generative models for 3d point clouds. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssaon Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 40 49. PMLR, 2018.
Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J. Mitra, and Paul Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12608-12618, June 2023.
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. In T. Leen, T. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000.
Andreas Blattmann, Tim Dockhor, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231 2023.
Ziang Cao, Fangzhou Hong, Tong Wu, Liang Pan, and Ziwei Liu. Large-vocabulary 3d diffusion model with transformer. In The Twelfth International Conference on Learning Representations, 2024.
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J. Guibas, Jonathan Tremblay, Samch Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-awane 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16123-16133, June 2022.
Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, L. Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. ArXiv, abs/1512.03012, 2015.
Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single- stage diffusion nerf: A unified approach to 3d generation and reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2416-2425, October 2023a.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appear- ance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22246-22256, 2023b.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang, Bsp-net: Generating compact meshes via binary space partitioning. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Zilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang, and Huaping Liu. V3d: Video diffusion models are effective 3d generators. arXiv preprint arXiv:2403.06736 2024.
Yen-Chi Cheng, Hsin- Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang- Yan Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 4456-4465, 2023.
Blender Online Community. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL htRp:ttp: www. b weneeee..... .rg
11

---

Published as a conference paper at ICLR 2025
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint ar arXiv:2212.0805// 2022.
Theo Deprelle, Thibault Groueix, Matthew Fisher, Vladimir Kim, Bryan Russell, and Mathieu Aubry. Learning elementary structures for 3d shape generation and matching. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
Haoqiang Fan, Hao Su, and Leonidas J. Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Wanquan Feng, Jin Li, Hongrui Cai, Xiaonan Luo, and Juyong Zhang. Neural points: Point cloud representation with neural fields for arbitrary upsampling. 2022.
Michael S. Floater and Kai Hormann. Surface parameterization: a tutorial and survey. In Advances in Multiresolution for Geometric Modelling, 2005. URL
org/CorpusID: 9922896
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. GET3d: A generative model of high quality 3d textured shapes learned from images. In Advances in Neural Information Processing Systems, 2022.
Shansan Gong, Mukai Li, Jiangtao Feng. Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Alex Graves. Generating sequences with recurrent neural networks. ArXiv, abs/ 30///308..0 008.0850, 2013. URL htwp://app..ssmniiaici...a.c http:/oo......ooooooooso::::::::
Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. A papier-måché approach to learning 3d surface generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi. Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion. In International Conference on Machine Learning, pp. 11808-11826. PMLR, 2023.
Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. arXiv preprint arXiv:2311.1277 2023.
Junlin Han, Filippos Kokkinos, and Philip Torr. Viusion3d: Learning scalable 3d generative models from video diffuusin models. arXiv preprint arXiv:2403. 12034, 2024.
Xianglong He, Junyi Chen, Sida Peng, Di Huang, Yangguang Li, Xiaoshui Huang, Chun Yuan, Wanli Ouyang, and Tong He. Gvgen: Text-to-3d generation with volumetric representation, 2024.
Jack Hessel, Ari Holtzman. Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: reference-free evaluation metric for image captioning. In EMNLP, 2021.
Jonathan Ho and Tim Salimans. Classifier free diffusion guidance. arXiv preprint arXiv:2207.12594 2022.
12

---

Published as a conference paper at ICLR 2025
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arxiv:2006. 11239, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633 8646, 2022b.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large reconstruction model for single image to 3d. In The Twelfth International Conference on Learning Representations, 2024. URL
net ket/Eoruu Eow..oodndddddurreeee
Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. SIGGRAPH, 2024.
Rongjie Huang, Jiawei Huang, Dongchao Yang. Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audios Textioo-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pp. 13916-13932. PMLR, 2023.
Ka-Hei Hui, Ruihui Li, Jingyu Hu, and Chi-Wing Fu. Neural wavelet-domain diffusion for 3d shape generation. December 2022.
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 867-876, 2022.
Hanwen Jiang. Zhenyu Jiang, Kristen Grauman, and Yuke Zhu. Few-view object reconstruction with unknown categories and camera poses. arXiv preprint urXiv:2212.0444 04499,, 2022.
Hanwen Jiang. Zhenyu Jiang, Yue Zhao, and Qixing Huang. LEAP: Liberate pparse-viww 3d modeling from camera poses. In The Twelfth International Conference Learning Representations, 2024. URL /ettpss:/opeereveee
net forum? i dd=KPma BxEaFF]
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. ArXiv, abs/2305.02463. 2023.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL https:// /tt::///rppppoee sam inria www.:in.rr....irr..i.i wwwp://uggggrp//.aaaua.eese
Diederik P. Kingma and Max Welling, Auto-encodinng variational bayes. CORR, abs/1312.6114, 2013.
variattion URL httpttt/ppsp/ppp semanti www.nt.ccccoolrrrooooooo woccchoo......rrooorros........ 211078809
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint wxXi:2200909761,, 2020.
Yushi Lan, Fangzhou Hong, Shuai Yang, Shangchen Zhou, Xuyi Meng, Bo Dai, Xingang Pan, and Chen Change Loy. Ln3diff: Scalable latent neural fields diffusion for speedy 3d generation, 2024.
Yushi Lan, Shangchen Zhou, Zhaoyang Lyu, Fangzhou Hong, Shuai Yang, Bo Dai, Xingang Pan, and Chen Change Loy. Gaussiananything: Interactive point cloud latent diffusion for 3d generation. In ICLR, 2025.
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprint arXiv:2311.062/4 14, 2023a.
13

---

Published as a conference paper at ICLR 2025
Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: Text-to-shape via voxelized diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12642-12651, 2023b.
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. Advances in Neural Information Processing Systems, 35: 4328-4343 2022.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yi Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 300-309, 2023a.
Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. Text generation with diffusion language models: A pre-training approach with continuous paragraph denoise. In International Conference Machine Learning, pp. 21051-21064, PMLR, 2023b.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12500 2023a.
Minghua Liu, Lu Sheng, Sheng Yang, Jing Shao, and Shi-Min Hu, Morphing and sampling network for dense point cloud completion. arXiv preprint arXi::19/2.0028)) 2019.
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45+4: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint arXiv:2311.0788
2023b.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object, 2023c.
Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consisere images from a single-view image. arXiv preprint arXiv:2309.03455 033450 2023d.
Zhen Liu, Yao Feng, Michael J Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiffusion: Score-based generative 3d mesh modeling. arXiv preprint arXiv:2303.08/33 2023e.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15008 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL
(kkgRiicqY?
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2837-2845, 2021.
Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. arXiv preprint N3XY::2306.02279, 2023.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
14

---

Published as a conference paper at ICLR 2025
Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 306-315, 2022.
Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas Guibas. Structurenet: Hierarchical graph networks for 3d shape generation. ACM Transactions Graphies (TOG), Siggraph Asia 2019, Arttcce 242, 2019.
Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buld, Peter Kontschieder, and Matthias Nießner. Diffrf: gendering-guidee 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4328 4338,433, June 2023.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 11(4)::02:1-102:111 July 2022.
Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, and Peter W. Battaglia. Polygen: An autoregressive generative model of 3d meshes. ICML., 2020.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szaframiec, Vasil Khalidov, Pierre Fernandez. Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby. Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang-WWnn Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.097493 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint vrxiv:2307.01952 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14990 2022.
Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffuuson model for detail richness in text-to-3d. arXiv preprint arXiv:2311.16911 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 18-24 Jul 2021.
Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538, Lille, France, 07-099 Jul 2015. PMLR.
Barbara Roessle, Norman Müller, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, Angela Dai, and Matthias Nießner. L3dg: Latent 3d gaussian diffusion. In SIGGRAPH Asia 2024 Conference Papers, SA '24, New York, NY, USA, 2024.
15

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: single image consistent multi-view diffusion base model, 2023a.
Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023b.
J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 20875-20886, June 2023.
Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Nießner. Meshgpt: Generating triangle meshes with decoder-only transformers. arXiv preprini arXiv:2311.154477 2023.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous ad-structure-awarn neural scene representations. In Advances in Neural Information Processing Systems, 2019.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference Machine Learning. volume 37 of Proceedings of Machine Learning Research, pp. 2265-226662222222222222222222 Lille, France, 07-09 Jul 2015. PMLR. URL URLhttps://ppoceedi press/v37 wic.eeiin....tttt...........
mlr
Jiaming Song, Chenlin Meng. and Stefano Ermon.
Denoising diffusion implicit models. arXiv:2010.02502, October 2020. (RL/tttps://axxv wttp:///aa//......2.222
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations, 2021. URL httpattss//ppeeeeveew..... formm?
1d=PxTIG12RRiH
Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. arXiv preprint arXiv:2310. 16818. 2023.
Yongbin Sun, Yue Wang, Ziwei Liu, Joshua Siegel, and Sanjay Sarma. Pointgrow: Autoregressively learned point cloud generation with self-attention. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 61-70, 2020a.
Yongbin Sun, Yue Wang, Ziwei Liu, Joshua E Siegel, and Sanjay E Sarma. Pointgrow: Autoregres- sively learned point cloud generation with self-attention. In Winter Conference Applications of Computer Vision, 2020b.
Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In The IEEF/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2024.
Jaaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.00 05054, 2024.
16

---

Published as a conference paper at ICLR 2025
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12619-12629, 2023.
Peng Wang and Yichun Shi. Imagedream: mnage-promph multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02200 2023.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
Daniel Watson, William Chan, Ricardo Martin Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. In The Eleventh International Conference on Learning Representations, 2023. URL httttp:://openrevlle... net./ htt.:/eeettottteee forum?
id cdoHT300
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning proba- bilistic latent space of object shapes via 3d generative-adversriia modeling. Advances in neural information processing systems, 29, 2016.
Jianwen Xie, Yifei Xu, Zilong Zheng, Ruiqi Gao, Wenguan Wang. Zhu Song-Chun, and Ying Nian Wu. Generative pointnet: Deep energy-based learning on unordered point sets for 3d generation, reconstruction and classification. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Dejia Xu, Ye Yuan, Morteza Mardani, Sifei Liu, Jiaming Song, Zhangyang Wang, and Arash Vahdat. Agg: Amortized generative gaussians for single image to 3d. arXiv preprint 2401.04099 2024a.
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.077 0711 2024b.
Xingguang Yan, Liqiang Lin, Niloy Mitra, Dani Lischinski, Danny Cohen-Or, and Hui Huang. Shapeformer: Transformer-based shape completion via sparse representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Guandao Yang, Xun Huang. Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Xu Yinghao, Shi Zifan, Yifan Wang, Chen Hansheng, Yang Ceyuan, Peng Sida, Shen Yujun, and Wetzstein Gordon. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation, 2024.
Zehao Yu, Torsten Sattler, and Andreas Geiger. Gaussian opacity fields: Efficient high-quality compact surface reconstruction in unbounded scenes. arXiv preprint arXiv:2404.1077. 2024.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Biao Zhang, Matthias Nießner, and Peter Wonka. 3DILG: Irregular latent grids for 3d generative modeling. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL /ttp:////pe/eereeeeee
Biao Zhang, Jiapeng Tang, Matthias Nießner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Trans. Graph, 42(4). jul 2023a. ISSN 0730-0301. doi: 10.1145/3592442. URL 3592442
Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high-quality 3d assets. arXiv preprint arXiv:2406. 13897, 2024.
17

---

Published as a conference paper at ICLR 2025
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023b.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Xin- Yan-Yan Zheng, Hao Pan, Peng-Shuai Wang, Xin Tong, Yang Liu, and Heung- Yeung Shum. Locally attentional sdf diffusion for controllable 3d shape generation. ACM Transactions on Graphics (TOG), 113, 2023.
Xinyang Zheng. Yang Liu, Pengshuai Wang, and Xin Tong. Sdi-stylegan: Implicit sdf-based stylegan for 3d shape generation. In Computer Graphics Forum, volume 41, pp. 52-63, Wiley Online Library, 2022.
Junsheng Zhou, Weiqi Zhang, and Yu-Shen Liu, Diffgs: Functional gaussian splatting diffusion. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3826-5835, 2021.
Zi-Xin Zou, Zhipeng Yu,
Guo, Yangguang Li, Ding Liang. Yan-Pei Cao, and Song- Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint 13XX::2312.09142 2023.
A IMPLEMENTATION DETAILS
A.1 ADDITIONAL DETAILS OF THE DATASET
We adapted the training setup from LN3Diff Lan et al. 2024). which initially used a subset of 35K shapes] These shapes cover three general categories: Transportation, Furniture, and Animals, derived from the G-buffer Objaverse (Qiu et al.) 2023). Due to limited computational resources, we cleaned the dataset by filtering out duplicate and low-quality shapes (e.g., those with flat ground or poor geometry), resulting in a final subset of approximately 18K shapes for training. Given the relatively small size of our dataset, we manually align the 3D shapes establish consistent canonical orientation across each category, facilitating easier learning. It is important to note that baselines like LN3Diff utilize additional data from other categories, which provides them an advantage.
We randomly selected 250 text prompts for evaluation, ensuring that each testing prompt differs from the training data. For each object, we rendered 24 views uniformly, following the G-buffer Objaverse protocol, resulting in total of 6K images. FID and KID are computed based on the 2D image feature space.
A.2 ADDITIONAL DETAILS OF VAB
The VAE training consists of two stages. In the first stage, X, in Eq. 12 is set 0. Rendering occurs only in the second stage with X, set to 1. In both stages, λKL maintains 1e-4 As shown in Eq. 12 all other loss weights set to for simplicity. Our method is robust hyperparameters: doubling or halving Ap in Eq. 12 results in almost identical loss curves.
We utilize sparse point cloud of 2048 points and 4 input views, each of size 224 224. M 2048 patches are used in all experiments. For the latent, = 512, d = 512. We set da to 4 for ShapeNet and do 16 in Objaverse. In ShapeNet, a is set to 4, resulting in N 32768 3D Gaussians. In Objaverse,
7. Due to the real-time rendering capabilities of the 3D Gaussians, we are able to produce an output image of size 512 512. We use the EMD implementation from (Liu et al. 2019). which supports fast and efficient computation even with up to 8192 points. All networks are trained on Tesla V100 GPUs for 1000 epochs using the Adam optimizer (Loshchilov & Hutter 2019)
The authors recently scaled up their model with more data, and we use this updated version for comparison.
18

---

Published as a conference paper at ICLR 2025
with the one-cycle policy. Our VAE is trained using mixed precision (fp16) and supports batch size of per GPU. For instance, when trained on the ShapeNet dataset, it requires approximately 22GB of memory per GPU, making it accessible to wide range of laboratories. Notably, the model does not require the latest 100 GPUs for training. Furthermore, the training process is completed less than 30 hours. For Objaverse, the VAE training takes about 6 days.
In terms of network parameters, our model increases the number of Gaussians without adding extra network parameters. Specifically, for Objaverse, our model requires only 142M network parameters to generate 100K Gaussians, while LGM (Tang et al. 2024) requires 415M parameters for generating 64K Gaussians. This demonstrates that our representation more efficient than purely explicit representations like LGM, offering nice trade-off between compact implicit and fast explicit representations.
A.3 ADDITIONAL DETAILS OF LDM
We adopt the EDM Karras et al. 2022) framework for latent diffusion. EDM aims to learn denoising network (2; a, C) to convert the Gaussian distribution to the empirical distribution Pdata defined by 2, where is the network parameters, is the noise level sampled from predefined distribution Prain and denotes the optional condition. DØ(2; a, C) is parameterized using a a-dependent skip connection:
Da(z; a, C) = (skippo)== + Cour Cour( (o) (Cin (0) Cnoise (o),C),
(14)
where Fo is the network to be trained. The training objective is
E.. (o))
=1/3
(15)
where a ~ Pirain: ~ Pdatas n ~ (0, 1). Readers interested in details on Cakip()), Coot Conn(a), (ar), Cin (o)(o)
"noise()), and parameterization of the weight function X(a) may refer to EDM (Karras al. 2022
For the latent diffusion model, we set = 12 for ShapeNet and = 24 for Objaverse. The final latents obtained via 40 denoising steps. For text-conditioned 3D generation, input text prompts. addition, we adopt classifier-free guidance. We randomly drop the conditioning signal with probability of 10% and the guidance scale to 3.5 during sampling. Note that more advanced architectures (Peebles Xie 2022) could also be employed for the denoising network.
CLIP encode the
B ADDITIONAL RESULTS
In Figure we present additional results of text-conditioned 3D generation using our method.
Figure shows the results of real-ine real-image-onn mongge-onnditooed 3D generation. In this experiment, simply replace the CLIP text encoder with the CLIP image encoder and train the latent diffusion model using the dataset. Given real image, and then use the processed image as the conditional input for 3D generation. The results demonstrate that our method can still produce reasonable outcomes.
first apply off-the-shelf tool to remove the background
C DISCUSSION ON CONCURRENT WORK
We discuss several concurrent works that utilize the VAE + LDM paradigm for feed-forward 3D generation. GaussianAnything (Lan et al. 2025) was concurrently developed and introduces an explicit 3D latent space to enhance interactivity. L3DG (Roessle al. (024) and DifiGS (Zhou et al.) 2024) both design VAE that takes 3D Gaussians input and outputs 3D Gaussians. However, both require per-shape optimization before training the VAE, which limits scalability.
D LIMITATIONS AND FUTURE WORK
Our method uses the vanilla 3D Gaussian representation, which known to challenging for extracting highly accurate geometry. Incorporating recent advancements (Huang et al.) 2024 Yu (ct al.] 20024; Guédon & Lepetit 2023) in 3D Gaussian techniques can benefit both geometry and appearance.
19

---

Published as a conference paper at ICLR 2025
four wheeled armared vehicle.
a Yellow Vespo scooter.
An 18th century connon.
colorful fish with blue and white paint.
T-34-85 green
Totero
An UFO Space Aircraft.
white goat
small camouflaged helicopter.
raccoon
A blue toy airplane with propellers.
wooden desk with drowers.
NASA Space Shuttle Model
wooden bench with white, concrete, and marble top options.
Black and white gaming chair
wooden boat with cars.
Figure 8: Additional results of text-conditioned 3D generation using method.
Figure Image-conditioned 3D generation using our method.
20


Reviews:
[{"review_id": "KQiQLXO58Z", "reviewer": "Reviewer_3ymM", "summary": "This paper introduces Atlas Gaussians, a new latent representation learning framework designed to improve the efficiency of 3D object generation by addressing the current challenges faced by VAEs. By modeling 3D shapes as a union of local patches, the authors develop a transformer-based decoder to map the low-dimensional latent representations of each patch via UV-based sampling and promote time efficiency when generating large samples of 3D Gaussians. Additionally, the introduced Atlas Gaussian representation considers a disentangled learning of geometric and appearance features, resulting in faster convergence.", "questions": "- It is unclear how geometry and appearance features are disentangled in the latent space. While the loss function is designed to learn each patch’s center and appearance features through supervision, the process for learning geometric features remains vague.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and organized; hence it is easy for readers to follow. \n\n- Addressing the current challenges of VAEs in extracting efficient latent representations for generative diffusion models is important. \n\n- The introduction of the Atlas Gaussians representation with disentangled learning of geometry and appearance features of 3D shapes is novel.", "weaknesses": "- The proposed approach includes both the Atlas Gaussian representation with disentangled geometry and appearance learning mechanism, as well as a transformer-based encoder. It is unclear which component contributes more significantly to the overall performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dgCn7HnD7b", "reviewer": "Reviewer_pZYd", "summary": "The authors introduce Atlas Gaussian, a 3D generation approach grounded in diffusion models. \n\nFirst, they train a VAE to map 3D shapes into a compact latent space, which is then decoded through a two-branch structure into patch centers and features. These patch centers and features can subsequently be decoded into 3D Gaussian distributions.\nNext, they train a Latent Diffusion Model (LDM) within the learned latent space to serve as the generation model. \n\nAtlas Gaussian offers three main advantages:\n1. Using UV-based sampling within a unit square, it can generate a large quantity of 3D Gaussians efficiently.\n2. It incorporates a nonlinear, learnable weight function through an MLP-projected positional encoding, enhancing representational power compared to traditional linear interpolation weights.\n3. It is computationally efficient with minimal memory usage.\n\nThese benefits are substantiated by comprehensive experiments and ablation studies.", "questions": "1. L060 \"However, these approaches often focus solely on modeling geometry without considering the appearance attributes.\"\nCan you give a more detailed explanation?\n\n2. What does $d$ represent in L189 and L135 (Equation 4)?\n\n3. What does $n$ represent in L256 and L268?\n\n4. What is the exact relationship between the shape information $\\mathcal{S}$ (L263) and the point cloud $\\mathcal{P}$ as well as the sparse-view RGB images $\\mathcal{I}$? \nFrom my understanding, the feature dimension of $\\mathcal{P}$ is $\\mathbb{R}^{|\\mathcal{P}| \\times d}$, and the feature dimension of $\\mathcal{I}$ is $\\mathbb{R}^{(|\\mathcal{I}| \\times h \\times w) \\times d}$. \nHow, precisely, are these features of $\\mathcal{P}$ and $\\mathcal{I}$, and the feature dimensions related to $\\mathcal{S}$, particularly in mathematical terms?\n\n5. How is the learnable query $y$ of the VAE decoder (L277, L284) initialized?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. I appreciate how the introduction section clearly explains and flows through the limitations of recent works, outlining how the authors address these challenges with their proposed method.\n\n2. I appreciate how each section of Section 2: Related Work clearly connects to the proposed method.\n\n3. Sufficient experimental results support the contributions claimed.\n\n4. I value the release of code which enhances transparency and reproducibility.", "weaknesses": "1. I suspect the term “atlas” is intended in the same sense as in AtlasNet (L125~128). If so, adding an explanation would be helpful, as it is part of the method’s name but is not clarified in the text.\n\n2. The method section is presented in a list-like format, making it difficult to read smoothly.\n\n3. The clarity of the writing is lacking. Although the text contains many details, the explanations are somewhat vague, requiring readers to infer and interpret a great deal.\n\n4. Some notations lack sufficient explanation. Although readers might deduce their meanings through context, clearer descriptions would be beneficial. (Also see questions below.)\n\n5. Figure 3 lacks detail and does not clearly relate to the main text, particularly Section 3.2, Stage 1: VAE. For instance,\n\n(1) The steps from $\\bar{z} \\rightarrow z_0$ are ambiguous. Is the process \n\n$z' = \\mathnormal{CrossAttn}(\\bar{z}, \\mathcal{P})$\n\n$z'' = \\mathnormal{CrossAttn}(z', \\mathcal{I})$\n\n$z_0 = \\mathnormal{MLP}(z'')$\n\n($z'$ and $z''$ notations arbitrarily assigned as they are not explicitly defined)\n\ncorrect? \n\n(2) My understanding is that $z_0$ of Figure 3 represents the end of the VAE encoder, while subsequent steps are part of the decoder. It would help to indicate these details explicitly.\n\n6. The conditioning ($\\mathcal{C}$ of Section 3.3) mechanism lacks detail; although some information is provided in the supplementary material, it remains insufficient.\n\n7. An ablation study should be conducted to analyze the impact of the loss hyperparameters ($\\lambda_r$ and $\\lambda_{KL}$).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J7aNvnmgqZ", "reviewer": "Reviewer_uWac", "summary": "The paper introduces a new framework called Atlas Gaussians Diffusion for 3D shape generation. It models the shape as a union of local patches as 3D Gaussians defined in UV space. This allows the generation process to generate theoretically unlimited number of points while capturing more underlying details of the 3D shape. Extensive quantitative and qualitative experiments demonstrate their results are better than the current SOTA.", "questions": "Do the authors plan to open source the model?\n\nCan UV-based sampling potentially generate artifacts? (e.g. inconsistency or distortions)\n\nHow well does the proposed framework handle noisy or sparse data?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper nicely integrates 3DGS with diffusion generation scheme and achieves state of the art results in 3D shape generation.\n\nThe use of Atlas Gaussians to represent 3D shapes as union of local patch makes it capable of capturing better details and higher scalability.\n\nDecomposing the 3D shape generation into local Gaussian patches and using UV-based sampling to handle shapes of varying complexity.\n\nThe evaluation is thorough with reasonable qualitative and quantitative results and achieves SOTA results in 3D shape generation.", "weaknesses": "While in 4.4 discuss about memory consumption and # of patches, the paper is lacking thorough comparison on inference times (esp. with the increase of # of 3DGS) to further justify the value of the framework.\n\nHow could the current approach address gap for even finer details of the shapes as Gaussian representation struggles in fine details and often results in blurry effect. Is it only a matter of the # of Gaussians or patches? \n\nThe author mentioned \"We take inspiration from the literature that 3D-based neural networks learn better 3D features for recognition and outperform multi-view based networks that leverage massive 2D labeled data.\" without any citation. There are arguments on both sides about overall performance with both approaches. This sentence seems to be subjective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p3gEuWBTgf", "reviewer": "Reviewer_BYST", "summary": "The authors proposed a representation model for generating 3D Gaussians from local patches, named Atlas Gaussians. They parameterized each patch as a sequence of feature vectors and designed a learnable function to decode 3D Gaussians, followed by the integration of UV-based sampling to generate large and theoretically infinite Gaussian points. To learn the Atlas Gaussians, the authors employed a latent diffusion model in the latent space of a VAE.", "questions": "Please refer to the Weaknesses section, where the primary questions have been outlined. Additional points are provided below -\n\n1. How are the local patches parameterized as a sequence of feature vectors, and how is the learnable function designed to decode 3D Gaussians from these feature vectors?\n\n2. Why is UV-based sampling used instead of directly generating an infinite number of 3D Gaussian points? Why is it necessary, and does it help in preserving the topology of the synthesized 3D objects?\n\n3. How does a large number of 3D Gaussians contribute to achieving high-quality details?\n\n4. How transformer-based decoder help to remove all the existing limitations posed by a vanilla decoder (other than the transformer decoder in LN3Diff)? Are there any strong quantitative and qualitative ablations?\n\n5. What is the corner-encoding scheme?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**S1.** Methodological development is clear and well presented.\n\n**S2.** The paper is very well written and has strong motivation.\n\n**S3.** Consideration of generation 3D Gaussians in latent diffusion model is unique and well-motivated.", "weaknesses": "While this paper explores a unique perspective on 3D generation, it primarily lacks proper experimental support and validation. It is difficult to discern the specific novelties and contributions (considering the three contributions outlined in L102-L107) that this paper offers compared to recent SOTAs, e.g., LN3Diff and others. Although the current methodological development is generally clear, the contributions are still not well-supported by the qualitative findings and significantly improved quantitative results reported in the paper.\n\n\n**W1. Latent Diffusion Novelty:** I believe the variational diffusion model is not a novelty in terms of technical development. Besides, the authors directly utilized the EDM algorithm as their latent diffusion. What makes the integration of the latent diffusion model a unique/novel contribution other than working with a different data type such as 3D Gaussians? Please highlight specific contributions with details in the revised version.\n\n\n**W2. Experimental Validation:** \n*(i)* Going through the generated samples reported in the LN3Diff and this paper, it is difficult to understand the effectiveness of the proposed model. The qualitative comparison with the LN3Diff model in Fig. 5 raises concerns regarding whether the baselines have been trained to their optimal or not. Besides, for most of the pictorial depictions, I can guess that either the sampling or training has not been done properly for all the SOTA, specifically in Fig. 6. In the SOTA baselines papers, the structure of the synthesized images seems to be well preserved. Then why the authors could not achieve that? *(ii)* Did the authors run all the baselines, and try to reproduce their results? Or they just copied the FID/KID numbers from the papers? *(iii)* I would suggest reporting the mean along with std to see the robustness of the scores. *(iv)* Inference times in Tab. 2 are not comparable with GVGEN and LN3Diff as they carried out their experiments on different GPU setups. *(v)* As one of the contributions of this paper is using a transformer-based decoder, I would suggest the authors show a very brief ablation carrying out the experiments on a vanilla decoder. Also, compared to the LN3Diff which used a transformer-base decoder, what is the uniqueness of their proposed architecture?\n\n\n**W3. Loss function and optimization:** According to the derived loss function, the optimization process seems to be quite complex to converge to the optimal stage as one needs control around 10 distinct losses, including the KL divergence term. I suggest the authors to shed some light onto the complete optimization process other than simply denote large number of losses with weighting parameters.\n\n\n**Minor.** Please cite necessary papers that are relevant to the proposed methodology, other than just scrapping eight pages of references which includes papers that do not fall into the problem of this paper.\n\n\nIn its current format, I believe the paper falls below the threshold for acceptance. The score reflects an evaluation of the problem definition, motivation, and presentation. However, I am open to raising my score if the above concerns are properly addressed. Besides, I request the authors to specifically highlight the methodological contributions and novelties (compared to the current SOTA), beyond the focus on generating 3D Gaussians.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)