LaGeM: A Large Geometry Model for 3D Representation Learning and Diffusion
Authors: Biao Zhang
Peter Wonka
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion
geometry
generative model
3d
Primary area: generative models

Abstract: This paper introduces a novel hierarchical autoencoder that maps 3D models into a highly compressed latent space. The hierarchical autoencoder is specifically designed to tackle the challenges arising from large-scale datasets and generative modeling using diffusion. Different from previous approaches that only work on a regular image or volume grid, our hierarchical autoencoder operates on unordered sets of vectors. Each level of the autoencoder controls different geometric levels of detail. We show that the model can be used to represent a wide range of 3D models while faithfully representing high-resolution geometry details. The training of the new architecture takes 0.70x time and 0.58x memory compared to the baseline.
We also explore how the new representation can be used for generative modeling. Specifically, we propose a cascaded diffusion framework where each stage is conditioned on the previous stage. Our design extends existing cascaded designs for image and volume grids to vector sets.

Published as a conference paper at ICLR 2025
LAGEM A LARGE GEOMETRY MODEL FOR 3D REPRESENTATION LEARNING AND DIFFUSION
Biao Zhang, Peter Wonka
KAUST, Saudi Arabia
Doiao.zharge peter. wonkar (aaust) sa
ABSTRACT
This paper introduces novel hierarchical autoencoder that maps 3D models into a highly compressed latent space. The hierarchical autoencoder is specifically de- signed to tackle the challenges arising from large-scale datasets and generative modeling using diffusion. Different from previous approaches that only work on a regular image or volume grid, our hierarchical autoencoder operates on unordered sets of vectors. Each level of the autoencoder controls different geometric levels of detail. We show that the model can be used to represent a wide range of 3D models while faithfully representing high-resolution geometry details. The train- ing of the new architecture takes 0.70x time and 0.58x memory compared to the baseline. We also explore how the new representation can be used for generative modeling. Specifically, we propose a cascaded diffusion framework where each stage is conditioned the previous stage. Our design extends existing cascaded designs for image and volume grids to vector sets.
INTRODUCTION
Diffusion models are currently the best-performing models for image, video, and 3D object genera- tion. For 3D object generation, there are two main branches of research. The first branch, pioneered by Dreamfusion (Poole et al., 2022), aims to lift 2D diffusion models to 3D model generation. The advantage of this method is that it can benefit from the large-scale 2D datasets used for training 2D diffusion models and it sparked a lot of follow-up work (Poole et al., 2022; Wang et al., 2023; Lin et al., 2023; Chen et al., 2023; Wang al., 2024; Qian et al., 2023; Tang et al., 2023; Yi et al., 2023; Wang & Shi, 2023; Liu et al., 2024; Long et al., 2024; Zheng et al., 2024; Li et al., 2023a; Ho et al., 2022; Xu et al., 2023). The second branch tackles the training on 3D datasets directly. The advantage of this method is that it is more direct and leads to faster inference times (Mittal et al., 2022; Yan et al., 2022; Zhang et al., 2022; Zeng et al., 2022; Zheng et al., 2023; Hui et al., 2022; Zhang et al., 2023; Siddiqui et al., 2024; Chen et al., 2024a;b). Our work sets to contribute to this second branch of methods.
Among these 3D native generation methods, 3DShape2VecSer (Zhang al., 2023) (or VecSet for short) has been proven to be an effective method to encode 31) geometry. It proposed an autoencoder to find an efficient representation for 3D models as set of vectors. Because of the high reconstruc- tion quality and compactness of the latent space, the method alleviates the difficulty of training 3D generative models. Some other works (Zhao al., 2024; Cao al., 2024; Dong et al., 2024; Petrov et al., 2024: Zhang et al., 2024b; Zhang & Wonka, 2024; Lan et al., 2024b) follow the VecSet rep- resentation. We noticed that VecSet's expressiveness is limited by the number of latent vectors. is overfitting on smaller datasets like ShapeNet and is unable to scale to larger datasets. To improve the expressiveness, we need to scale up the latent size and the training dataset. The straightforward way is to employ hundreds of GPUs for training which is expensive (Zhang al., 2024b). Thus, our goal is to reduce the training cost in terms of time and memory consumption while achieving similar even better autoencoding quality.
In the image domain, NVAE (Vahdat & Kautz, 2020) extended the design of the variational autoen- coder (VAE) (Kingma, 2013) to a hierarchical VAE based on the design of the U-Net. The latent space of the NVAE is a multi-scale latent grid and the reconstruction quality of the images from the NVAE improves lot over the VAE. An illustration of the architectures can be found in Fig. 1. We

---

Published as a conference paper at ICLR 2025
AE
VAB
VECSET
FoLL
LioF
FuL
LioF
Encoder
Decoder
Encoder
Latent
Decoder
Encoder
Latent
Decoder
FooL
LioF
FuLL
LioF
Ftol.
LioF
Ftol.
LioF
U-NET
NVAE
LAGEM F
Figure I: Autoencoders. We show different autoencoder architectures here, including AE (AutoEn- coder), U-Net, VAE (Kingma, 2013), NVAE (Vahdat & Kautz, 2020), VecSet (Zhang et al., 2023) and the proposed LaGeM. VAE and NVAE are for image data, while VecSet and LaGeM are for geometry (distance function) data. In the top row, VAE and VecSet use a single-scale latent to rep- resent the data. Both NVAE and LaGeM use multi-scale latents to represent data. All the previous works VAE, NVAE, and VecSet apply KL divergence the bottleneck regularize the latent space, while in this work, we apply standardization in the bottleneck.
draw inspiration from the design of the NVAE and design multi- latent VecSet representation, called LaGeM. We train our architecture a large-scale geometry dataset Objaverse (Deitke et al., 2023) and improve training time by 0.7 and memory consumption by 0.58 compared VecSet.
Additionally, we also propose cascaded generative model for the hier- archical latent space. We generate the latent VecSet from the lower reso- lution level to the highest resolution level suge-by-stagee In each stage, we use the previously generated latents as conditioning information, As result, this enables control over the level of detail of the generated ge- ometry.
Latents
Controlling
Level 3
Main Structure
Level 2
Major Details
Level I
Minor Details
We summarize contributions follows:
We propose hierarchical autoencoder architecture with faster training time and low mem- ory consumption. The latent space is composed of several levels.
The model capable of training on large-scale datasets like objaverse.
We propose a cascaded diffusion model to generate 3D geometry in the hierarchical latent space. This enables control of level detail of the generated model.
RELATED WORKS
We show an overview of latent 3D generative models in Table 1, particularly focusing on the type of latent space used.
2.1 LEARNING METHODS
Usually, learning method required to convert 3D geometry to latent space. One way to do this is to convert 3d geometry to latent space with per-object optimization method, e.g. (Erkoç et al., 2023; Yariv et al., 2024). For larger datasets, this approach is very time-consuming, 2) Alternatively, auto-decoder, e.g., DeepSDF (Park et al., 2019), jointly optimize the latent space for all objects in the dataset. However, as there no encoder, new objects cannot be mapped to latent space easily. 3) Therefore, commonly used framework is the auto-encoder. The optimization is efficient because is performed jointly for all objects in the dataset, and new objects not in the training set can be quickly encoded using the encoder. Thus, also build this approach.
2

---

Published as a conference paper at ICLR 2025
Table 1: Geometric Latent Representation and Generation.
Method
Learning Method
Latent Rep
Hierarchies
ShapeFormer (Yan et al., 2022)
AutoEncoder
Sparse Volume
Single
3DILG (Zhang et al., 2022)
AutoEncoder
Irregular Grid
Single
LION (Zeng et al., 2022)
AutoEncoder
Latent Points
Multi
TriplaneDiffusion (Shue al., 2023)
AutoDeoeder
Planes
Single
SDFusion (Cheng et al., 2023)
AutoEncoder
Volume
Single
3DShape2VecSel (Zhang al., 2023)
AutoEncoder
VecSet
Single
HyperDiffusion (Erkoç al., 2023)
Per-Obje Per-Object Optimization
Network Weight
Single
XCube (Ren al., 2024)
AutoEncoder
Sparse Volume
Multi
Mosaic-SDF (Yariv et al., 2024)
Per-Object Optimization
Irregular Grid
Single
3LLhpia et al., 2024c)
Per-Object Optimization
Irregular Grid
Single
OctFusion (Xiong et al., 2024)
AutoEncoder
Sparse Volume
Multi
LN3Diff (Lan et al., 2024a)
AutoEncoder
Triplanes
Single
LaGeM (Ours)
AutoEncoder
VecSet
Multi
Diffusion
Diffusion
Ditfusion
Init
Level
Level 2
Level
Point Cloud
Encoder
Decoder
Output
Decoder $
STAGE AL AUTONCCOEER
STAGE 2: LATENT DIFFUSION
Figure 2: Pipeline. We proposed U-Net-style transformer for the autoencoding. In this way, obtain a hierarchical latent space, which contains several levels of latents. To train the generative diffusion models in the latent space, we propose the cascaded latent diffusion models.
2.2 LATENT REPRESENTATIONS
Early methods used regular grids (Yan et al., 2022; Cheng et al., 2023) as the latent representation because of their simple structure. We can easily convolutional layers process volume data. To represent high-quality geometric details, we need large-resolution volumes. This makes the training more difficult because of the O(n²) complexity. A way solve this problem is to introduce sparsity (Ren et al., 2024) to the representation like octrees (Xiong et al., 2024) or sparse irregular grids (Zhang al., 2022: Yariv al.. 2024). Both structures have the potential represent high- quality 3D models, but generating irregular structures explicitly is difficult for diffusion models. Different from the above-mentioned approaches, 3DShape2 VecSet (Zhang al., 2023) proposed to solve the reconstruction problem without using any sparse structures. The representation is easy to use. this paper, we investigate how to improve the VecSet representation. Compared to Zhang et al. (2023), our goal is to obtain an even higher-quality latent space by introducing Level of Latents (LoL).
2.3 CASCADED GENERATION
In the field of image generation, there are multiple cascaded diffusion models,e.g. (Ho et al., 2022; Saharia et al., 2022). In the 3D domain, some works (Zeng et al., 2022; Ren et al., 2024) also modeled geometries with hierarchical latents and proposed 3D generative models using cascaded diffusion models. Our work encodes 3D geometry into hierarchical VecSets. Thus, it is straightfor- ward to consider cascaded latent diffusion train generative models in our latent space.

---

Published as a conference paper at ICLR 2025
Pir
plapoi
CA - FtoL
LtoF
+1"
-
Encoder
Bottleneck
Decoder
Figure 3: Geometry Autoencoder. The design from VecSet (Zhang et al., 2023) can be seen
special case of the proposed LaGeM network with only one level.
PPS
Fiol.
2
LioF
X
CA
SAs
+ii
CA
FtoL
22
Ltol" +
CA
SAs
CA
x13 FtoL
Zg
LtoF
+X-
SAs
Encoder
Bottleneck
Decoder
Figure 4: LaGeM architecture. We show an illustration with levels of latents.
METHODOLOGY
3.1 BACKGROUND OF VECSET REPRESENTATIONS
The VecSet (Zhang et al., 2023) representation converts dense point cloud to latent vector set 2 = {Z,, ..,..., ZM} } with z E RD so that an occupancy/distance function (p)) can be recovered from the vector set. The simplified network is illustrated in Fig. 3.
Encoding. The process first downsamples the 3D input point cloud Plapot = {p:)i=i,..... with furthest point sampling (FPS), p = FPS(plaput, (plapptrr) where r is the down-sampling ratio, and P is low-resolution version of plnput Then plupur converted to an unordered set with cross-attention
CA(Q PE(P), PE(plnpu))
PE(plaput) X fx RC 1==
(I)
where PE is positional embedding function (Zhang et al., 2023) and ..(.....) -) is a cross-attention module. We also write CA(P, ploput) for short. Here, the positional embedding used to project a 3D coordinate p E R3 to the high dimensional space I is omitted for simplicity. To obtain highly compressed latent space, the vectors X' are further compressed lower-dimensional space R where D < C' (Feature to Latent, FtoL in short),
Ft) Z {z E R Ji=
This compression step is also regularized by KL divergence.
Decoding. Each latent vector in Z first converted back to feature space RC (Latent to Feature, or LtoF in short),
LtoF X' (x' E RC R
The features X' are fed into series of self-attention layers obtain final occupancy/dissance func- tion representations F.
SAs(x') = F= (f R
h=12....M
(4)
where SAs(-) is implemented using several self-attention layers. Now we can decode continuous function. For a continuous coordinate in the space IR3, we have
O(p) = FC (CA(p. E R.
(5)
See Table for more details on F.oL(()) and L.tF(().
3.2 HIERARCHICAL VECSET
The complexity of the self-attention layers in Eq. (4) is O(M)), ie.. quadratic in the number of input vectors. This severely affects the training time when M is large. However, to represent high- quality geometry details, usually need large M. This makes training a large VecSet network
4

---

Published as a conference paper at ICLR 2025
Table 2: Regularization in the Bottleneck. We compare the proposed regularization (NBAE) and
VAE. We do not need an explicit loss to regularize the latent space.
Features to Latents (Ftol.)
Latent Loss
Latents to Features (LtoF)
μ = (x)
VAE
FC (x)
p + a
④ e
KI. Divergence
= FCup(2)
Z E 2
NBAE
2 = FCoowa (x)
2
x = FCup(z 3 + g)
Var
more challenging (for example, M 2048 in CLAY (Zhang et al., 2024a)). Motivated by the de- sign of the U-Net and NVAE (Vahdat & Kautz, 2020), we propose a new network. Specifically, in the design of the U-Net (see an illustration in Fig. 1), image feature grids are downsampled to lower resolutions where some convolution blocks are applied, and then upsampled to the original resolution. In this way, we can avoid performing convolutional layers in high-resolution images (which can be time-consuming). We transferred this idea to the VecSet representations. Two neces- sary building blocks are operations to down-san wown-sample and up-sample a VecSet. Inspired by the design of 3DShape2 VecSet (Zhang et al., 2023) (an illustration can be found in Fig. 3), interpret the cross attention in the encoder part as a down-sampling operator. Similarly, we can also use it for up-sampling. The resulting network is shown in Fig. 4.
We have L levels in the U-Net-style transformer, where we number the levels from one (highest resolution) to L (lowest resolution). For notational convenience, denote the input point cloud as level O. In the i-th level, we first obtain a lower resolution of the point clouds in the (i - 1 P, where Po is the input point cloud. We cross attention to compress
level, FPS(P:-1,Pt- the feature set CA(P,, Pi-1) = X, Different from previous approaches, we propose a new to way regularize the latent space,
Fto Zero VanannnnnnnnnniiaooooCCCCCCC
= Zi,
where we normalize each vector in the set to have zero mean and unit variance (z E(z))/ / [z] (It is often called standardization in machine learning which is used to standardize the features present in the data in a fixed range.). The motivation behind this design is that diffusion starts with Gaussian noise which also has zero mean and unit variance. In this way, we enforce both our latent space and the initial Gaussian noise to have similar properties. To map the latents back to features, we first scale and shift latents back z C y + B (both y and B are learnable parameters like in Layer Normalization (Lei Ba et al., 2016)),
LtoF (2.) = FCup (ScaleAndShift((2)) (2,)) = X-
(7)
Thus we name it Normalized Bottleneck Autoencoder (NBAF). Unlike KL. divergence in a VAE, do not need an explicit loss term for the latent space. See Table for comparison between the proposed regularization and commonly used KL divergence in VAEs.
Inspired by the down-sampling usage of cross attention in Zhang al. (2023), we generalize it to resampling. Here we use it as upsampling for unordered set Fe. Before feeding the features to self-attention layers, we first upsample features Fi+1 from lower resolution levels,
CA >
Fx
(AA = Fi.
(8)
CA
-
FC
+ppp)
The query function in Eq. (5) is changed to (p) = FC ([CA(p, Fi)l ... (C)( RR
Fq
(9)
where |.I.1...|1 |-] is the symbol for concatenation. This means we are
using features from all levels to build the final (occupancy) function representation.
3.3 DIFFUSION
Cascaded Diffusion (Ho et al., 2022) proposed method for generating high-resolution images. The method is composed of several stages, where each stage is conditioned diffusion model. Moti- vated by this, we propose a cascaded latent diffusion model. In Cascaded Diffusion, images gener- ated from the previous stage used condition in the next stage. We build a cascaded latent
5

---

Published as a conference paper at ICLR 2025
Denoising Network
Denoising Network
Denoising Network
CA
CA
t
t,
t
Level
Level 2
Level
Figure 5: Cascaded Latent Diffusion.
Table 3: Running Statisties of LaGeM. When using small number (512) latent vectors, our model uses 0.87x time and 0.66x memory during training. For larger models (2k latent vectors), the advantage is even more significant (0.7x time and 0.58x memory).
VecSet
LaGeM
VecSet
LaGeM
VecSet
LaGeM
Batch Size
64
8
Self Attn Layers
24
8/8/8
24
8/8/8
24
8/8/8
Attn Channels
512
512/512/512
lk
lk/lk/k
1k
lk/lk/lk
# Parameters (M)
106.13
125.15
424.24
499.85
424.24
499.85
Latent Vectors
512
32/128/512
2k
128/512/2K
2k
128/512/2k
Latent Channels
8
32/16/8
64
64/32/16
64
64/32/16
Training Memory (M)
56,125
37,055
(0.66x)
OOM
53,791 (-)
54,543
31,662
(0.58x)
Training Iteration (sec)
0.6481
0.5658 (0.87x)
0.7714 (
0.6902
0.4853 (0.70x)
diffusion model based on Cascaded Diffusion. Formally, the optimization goal (for our three-level implementation) is as follows,
min
Ds(Zs(t),t,C
Zz
133
min
(2(22(0),4C,22 ) Z2
(10)
D2
()((, Zz. 2n) 21
min
D1
where D; is a denoising network, ? represents timestep or noise level, 2,(t) is the noised version (at timestep !) the latent, optional condition information (e.g., text, images, or categories). The network design is based on DiT (Peebles & Xie, 2022). To generate latents Z,, we need latents from previous stages Zyi- For diffusion-based image super-resolution methods, this often done by bilinearly interpolating small images and concatenating them with denoising networks' inputs. As shown the previous section, we use cross- aooss-attention for resampling (both down-sampling and upsampling). Here, also utilize cross-attention upsample latent set. Specifically, assuming are training denoising network for Z2, the input of the network is Zz)
(t) Za).
(11)
Similarly, for 21,
CA(CA(Ž Za), Z2).
(12)
In this way, we are gathering information from previous stages. See Fig. 5 for an illustration of the pipeline.
EXPERIMENTS
4.1 AUTOENCODING MODEL
The main autoencoding experiment is trained on Objaverse (Deitke et al., 2023). Models are zero- centered and normalized into the unit sphere. Since most 3D models in this dataset not watertight,

---

Published as a conference paper at ICLR 2025
Table 4: Evaluation on ShapeNet. We compare our results to VecSet (Zhang et al., 2023) trained on ShapeNet. If we train our model on ShapeNet and evaluate on ShapeNet our model is slightly better than VecSet. When our model is trained on Objaverse and evaluated on ShapeNet, we can see a very large improvement. Note that it is difficult to scale VecSet to Objaverse training.
Chamfer (x 100)
F-Score (x 100)
LaGeM(A)
LaGeM(A)
VS
VS
ShapeNet
Objaverse
ShapeNet
Objaverse
table
2.46
2.48
0.02
2.09
-0.37
99.94
99.97
0.02
99.96
0.02
car
5.99
5.89
-0.10
4.36
-1.63
89.85
90.31
0.46
92.15
2.30
chair
2.92
2.89
-0.03
2.01
-0.91
96.40
96.49
0.09
99.91
3.51
airplane
1.78
1.81
0.03
1.58
-0.21
99.50
99.48
-0.02
99.78
0.29
sofa
2.64
2.63
-0.01
2,25
-0.39
98.92
99.04
0.11
99.60
0.67
rifle
1.78
1.77
-0.01
1.44
-0.34
99.88
99.88
-0.01
99.94
0.06
lamp
4.36
4.44
0.08
2.37
-2.00
96.78
97.18
0.39
99.43
2.64
mean (selected)
3.13
3.13
0.00
2.30
-0.83
97.33
97.48
0.15
98.68
1.36
mean (all)
4.68
4.63
-0.04
2.42
-2.26
93.25
93.47
0.23
98.93
5,68
Table 5: Generalization on Various Datasets. Our trained model is capable of doing inference on several existing datasets, It can be applied on non-watertight datasets like ABO and pix3d even if the model is trained on watertight datasets. Note that models from ShapeNet were not originally watertight. We use the watertight version processed by (Zhang et al., 2022).
Chamfer + (x100) VS
F-Score f (x100) LaGeM(()
Dataset
# Meshes
Manifold
LaGeM(A)
VS
Thingi10k (Zhou & Jacobson, 2016)
10k
Yes
4.52
2.99
-1.53
92.75
97.19
4.44
ABO (Collins et al., 2022)
8k
No
4.91
3.66
-1.26
92.52
94.91
2.39
ShapeNet (Chang et al., 2015)-test
2k
Yes
3.25
2.33
-0.92
97.41
99.49
2.08
EGAD (Morrison et al., 2020)
2k
Yes
3.27
2.82
-0.45
99.02
99.76
0.74
GSO (Downs et al., 2022)
1k
Yes
3.78
2.35
-1.43
94.70
99.54
4.84
pix3d (Sun et al., 2018)
700
No
6.53
6.02
-0.50
87.25
87.96
0.71
FAUST (Bogo et al., 2014)
100
Yes
2.10
1.31
-0.79
99.58
99.90
0.32
we use ManifoldPlus (Huang al., 2020) to make all meshes watertight. Due to failures of modeling loading and conversion, we obtained around 600k watertight models for training. The three levels of latents are 128 x 64, 512 x2, and 2048 x 16 (where 64, 32, and 16 are channels of the latents). Some other hyperparameters of the network can also be found in Table 3. We name the model as LaGeM- Objaverse. We also apply the method to ShapeNet, where the train split is taken from (Zhang et al., choose Objaverse,
2022). Since ShapeNet is relatively small and easy dataset compared smaller latents which are 32 x 32, 128 x 16, and 512 x 8. The model named as LaGieM-Shapeeet, Both models are compared against VecSet (Zhang et al., 2023). We use Chamfer distance and F- score as the metrics. The results are shown in Table 4. Like (Zhang et al., 2023), we first compare the results on the largest categories (which have several thousand training samples) ShapeNet and then all categories. We can see that, LaGeM- M.GGeM-Shapeeet has almost the same number of parameters as VecSet, but with much shorter training time and less training memory. The quantitative results (averaged over all ShapeNet categories) are also better than VecSet's. While for LaGeM-Objavers there is large improvement in both training cost and quantitative results. The quantitative results show improvement of almost 50 percent averaged across the complete dataset in terms of the metric Chamfer. This demonstrates that LaGeM- Objjavere has good generalization ability. This can also be seen in Fig. 6. The results of LaGeM- --jjavereee good small categories of ShapeNet. In previous works (Zhang et al., 2023), this is nearly impossible because of limited training samples.
To further prove the generalization ability of LaGeM-Objavvrse, we also test the autoencoding on various datasets, including Thingi10k (Zhou & Jacobson, 2016), ABO (Collins al., 2022),

---

Published as a conference paper at ICLR 2025
1090
F-Score
Dataset Size
VecSet (Triined on ShepoNet, Tested on ShapeNett
Pruposed (Trained on Objaverse, Tested ShapeNet)
rable clait
airplans
rillo
vostoneell
bust
bulinns
colpphon
flowerpo
baptrg bookshelf Knife Inin
bin mooobii
max
bolker
bag didhoaseer
binllone dicoephae Aemito Seyborr usp
cuppous
Figure 6: Generalization on ShapeNet. Our results are better than VecSet in all categories. On
small categories, the results of VecSet are not stable because of limited training samples. In contrast,
our trained model also performs well in these categories.
GT
W.Laiim
VecSet
LaGeM
Figure 7: Qualitative Results on ShapeNet. We show autoencoding results on ShapeNet. We use VecSet as the baseline. Our model capable reconstructing detailed geometry, especially thin structures.
GT
VecSet
LaGeM
Figure 8: Qualitative Results on Thingi10k. 10k. Our model can even preserve highly detailed geome- try in CAD models.
8

---

Published as a conference paper at ICLR 2025
FAUST
GSO
GT
VecSet
LaGeM
Figure 9: Qualitative Results on FAUST and GSO. Results of VeeSet are over-smoothed, while method can preserve sharp details.
Figure 10:
Generative Results ShapeNet.
EGAD (Morrison et al., 2020), GSO (Downs et al., 2022), pix3d (Sun al., 2018) and FAUST (Bogo et al., 2014). The objects from these datasets vary from daily objects, CAD models, human models, and synthetic objects. The quantitative results can be found in Table 5. Again, we use VecSet's model as the baseline. From the metrics, we can see that LaGeM-Objaverse is able to represent different kinds of objects with highly detailed geometry and sharp features. Note that, even for non- watertight meshes, the model is still able to do reconstruction. Visual results of the method can be found in Fig. 7, Fig. 8, Fig.
4.2 GENERATIVE MODEL
We conducted two generative experiments; one was ShapeNet with categories as the condition, and the other one was an unconditional generation on Objaverse- 10k. For ShapeNet, the denoising networks of the 3 levels have 12 self-attention blocks with 768 channels. We trained the model for around 200 hours with 4 A100 GPUs The results are shown in Fig. 10. For Objaverse- 10k, due select subset of 10k models from Objaverse and train the to limited training GPU resources, unconditional generative model. There are 24 self-attention blocks with 768 channels in all stages of the latents. The model is trained 16 A100 GPUs for around 100 hours. See Fig. 11 for some unconditional generation results.
Controllability of the Latents. We verify that different levels of latents control different levels of detail of the generated samples. During generation, we first generate higher-level latents Z,, which determine the main structures of the 3D models. Then which adds major details to the models. In the end, we generate Z, conditioned on both Z, and Zz. This final step adds some minor details the samples. A visual illustration can be found in Fig. 12.
Z3 condition generate Zz,

---

Published as a conference paper at ICLR 2025
Figure 11: Unconditional Generative Results on Objaverse-10k.
Level
Levd
Latent Levels. Each small 4 block shares the same level 3 latents Zz. 3D models in the same block have similar structures. In each block, every 1 x line shares the same level latents Zz. In each line of block, 3D models look almost the same except for some minor details. Thus, we argue that 33 controls the structure, Z₂ affects the major details and ZN is responsible for minor details.
Figure 12:
5 CONCLUSION
We proposed LaGeM (Large Geometry Model), an architecture for encoding 3D geometry. Dif- ferent from previous approaches, the latent space is modeled as a hierarchical latent VecSets. To make this work, our model employs a U-Net-style design and new regularization technique for the bottleneck. We showed that this model can be trained much faster with much lower GPU memory costs, especially for larger networks and datasets. This enables scaling of the network for large-scale datasets. We release our model trained on 600k geometry dataset. Additionally, we proposed cascaded diffusion model to show some preliminary generative results with the hierarchical latent space.
Limitation. Since the latent space is divided into multiple levels, training diffusion model all levels still takes a lot of time. Our method does not solve the high training cost problem of diffusion itself.
ACKNOWLEDGMENTS
The work is supported by funding from KAUST Center of Excellence for Generative AI, under award number 5940, and the NTGC-AI program.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Federica Bogo, Javier Romero, Matthew Loper, and Michael J Black. Faust: Dataset and evaluation for 3d mesh registration. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3794-3801, 2014.
Wei Cao, Chang Luo, Biao Zhang, Matthias Nießner, and Jiapeng Tang. Motion2vecsets: 4d la- tent vector set diffusion for non-rigid shape reconstruction and tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20496-20500 2024.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, al. Shapenet: An information-res 3d model repository. arXiv preprint 15/2.030//0011111 2015.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF inter- national conference computer vision, pp. 22246-22256. 2023.
Sijin Chen, Xin Chen, Anqi Pang, Xianfang Zeng, Wei Cheng, Yijun Fu, Fukun Yin, Yanru Wang, Zhibin Wang, Chi Zhang, et al. Meshxl: Neural coordinate field for generative 3d foundation models. arXiv preprint arXiv:2405. 208533, 2024a.
Yiwen Chen, Tong He, Di Huang, Weicai Ye, Sijin Chen, Jiaxiang Tang, Xin Chen, Zhongang Cai, Lei Yang, Gang Yu, et al. Meshanything: Artist-created mesh generation with autoregressive transformers. arXiv preprint arXiv:2406. 10163, 2024b.
Zhaoxi Chen, Jiaxiang Tang, Yuhao Dong, Ziang Cao, Fangzhou Hong, Yushi Lan, Tengfei Wang, Haozhe Xie, Tong Wu, Shunsuke Saito, et al. 3dtopia-xl: Scaling high-quality 3d asset generation via primitive diffusion. arXiv preprint arXiv:2409. /2957, 2024c.
Yen-Chi Cheng, Hsin- Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang- Yan Gui. Sd- fusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 4456-4465, 2023.
Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. Abo: Dataset and benchmarks for real-world 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 21 11126-21136 1336, 2022.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11142-1355,, 2023.
Yuan Dong, Qi Zuo, Xiaodong Gu, Weihao Yuan, Zhengyi Zhao Zilong Dong, Liefeng Bo, and Qixing Huang. Gpld3d: Latent diffusion of 3d shape generative models by enforcing geometric and physical priors. In Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 56-66, 2024.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pp. 1553-2560 JEEE, 2022.
Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nicßner, and Angela Dai. Hyperdiffusion: Generat- ing implicit neural fields with weight-space diffuuion. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp. 14300, 2023.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(4)) 2022.
Jingwei Huang, Yichao Zhou, and Leonidas Guibas. Manifoldplus: A robust and scalable watertight manifold surface generation method for triangle soups. arXiv preprint arXiv:2005. 1621, 2020.
11

---

Published as a conference paper at ICLR 2025
Ka-Hei Hui, Ruihui Li, Jingyu Hu, and Chi-Wing Fu. Neural wavelet-domain diffusion for 3d shape generation. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-9, 2022.
Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565 35:26565-26577,, 2022.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint 131.v:1112.61144 2013.
Yushi Lan, Fangzhou Hong. Shuai Yang, Shangchen Zhou, Xuyi Meng, Bo Dai, Xingang Pan, and Chen Change Loy. Ln3diff: Scalable latent neural fields diffusion for speedy 3d generation. In European Conference on Computer Vision, pp. 112-130. Springer, 2024a.
Yushi Lan, Shangchen Zhou, Zhaoyang Lyu, Fangzhou Hong, Shuai Yang, Bo Dai, Xingang Pan, and Chen Change Loy. Gaussiananything: Interactive point cloud latent diffusion for 3d genera- tion. arXiv preprint arXiv:2411.0800. 2024b.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. ArXiv e-prints, pp- arXiv-1607, 2016.
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view gen- eration and large reconstruction model. arXiv preprint arXiv:23 2tt::23/11.02/1// 04, 2023a.
Yuhan Li, Yishun Dou, Xuanhong Chen, Binghing Ni, Yilin Sun, Yutian Liu, and Fuzhen Wang- 3dqd: Generalized deep 3d shape prior via part-discretized diffusion process. arXiv preprint arXiv:2303.10400 2023b.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d con- tent creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 300-309, 2023.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu. and Hao Su. One- 2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffuusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9970-9980, 2024.
Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 306-315, 2022.
Douglas Morrison, Peter Corke, and Jürgen Leitner. Egad! an evolved grasping analysis dataset for diversity and reproducibility in robotic manipulation. IEEE Robotics and Automation Letters, (3):4368 4375, 2020.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 165-174, 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
Dmitry Petrov. Pradyumn Goyal, Vikas Thamizharasan, Vladimir Kim, Matheus Gadelha, Melinos Averkiou, Siddhartha Chaudhuri, and Evangelos Kalogerakis. Gem3d: Generative medial ab- stractions for 3d shape synthesis. ACM SIGGRAPH 2024 Conference Pupers, pp. 1-11, 2024.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint 2209 14988, 2022.
12

---

Published as a conference paper at ICLR 2025
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov. et al. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306. 17843, 2023.
Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4209-4219, 2024.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-364944, 2022.
J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20875 2085660 2023.
Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Nießnnr. Meshgpt: Generating triangle meshes with decoder- only transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19615-196222 19625, 2024.
Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang. Chengkai Zhang, Tianfan Xue, Joshua Tenenbaum, and William T Freeman. Pix3d: Dataset and methods for single-image 3d shape modeling. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2974-2983, 2018.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint WxXiv:2399166533 2023.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 33:1 196779999999, 2020.
Haochen Wang. Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jaco- bian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12619-12629, 2023.
Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation.
arXiv preprint arXiv: 2023.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
Bojun Xiong, Si-Tong Wei, Xin-Yang Zheng, Yan-Pei Cao, Zhouhui Lian, and Peng-Shuai Wang. Octfusion: Octree-based diffusion models for 3d shape generation. arXiv preprint arXiv:2408. 147332 2024.
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. arXiv preprint arXiv:231 1.09211..111.,1.111111 2023.
Xingguang Yan, Liqiang Lin, Niloy Mitra, Dani Lischinski. Daniel Cohen-Or, and Hui Huang. Shapeformer: Transformer-based shape completion via sparse representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pp. 6239-62499 2022.
Lior Yariv, Omri Puny, Oran Gafni, and Yaron Lipman. Mosaic-sdf for 3d generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4639-4639, 2024.
Taoran Yi, Jiemin Fang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors. arXiv preprint 1r00v22310.08559, 2023.
13

---

Published as a conference paper at ICLR 2025
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint
1oXY::2210.06978, 2022.
Biao Zhang and Peter Wonka. Functional diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4723-4732, 2024.
Biao Zhang, Matthias Nießner, and Peter Wonka. 3dilg: Irregular latent grids for 3d generative
modeling. Advances in Neural Information Processing Systems, 35:21871-21885, 2022.
Biao Zhang, Jiapeng Tang, Matthias Nießner, and Peter Wonka. 3dshape2vecset: A 3d shape repre- sentation for neural fields and generative diffusion models. ACM Trans. Graph., 42(4), July 2023. ISSN 0730-0301. doi: 10.1 10.1145/3992442 URL https:ttpp...mm /doi www.:o..ooooleeeee 1145 44/3592442.
Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang. Lan Xu, and Jingyi Yu. Clay: controllable large-scale generative model for creating high-quality 3d assets. ACM Trans. Graph., 43(4), July 2024a. ISSN 0730-0301. doi: 10.1145/3658 146. URL https://dot orgg 10 1145 1145/3658144
Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high-quality 3d assets. ACM Transactions Graphics (TOG), 43(4):1-20. 2024b.
Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng, Rui Wang. Pei Cheng, Bin Fu, Tao Chen, Gang Yu, and Shenghua Gao. Michelangelo: Conditional 3d shape generation based on shape-image-tert aligned latent representation. Advances in Neural Information Processing Systems, 36, 2024.
Xin-Y Kin-Yang Zheng, Hao Pan, Peng-Shuai Wang, Xin Tong, Yang Liu, and Heung- Yeung Shum. Lo- cally attentional sdf diffusion for controllable 3d shape generation. ACM Transactions Graph- ics (ToG), 1234) 2023.
Xin- Yang Zheng, Hao Pan, Yu-Xiao Guo, Xin Tong, and Yang Liu. Mvd' 2: Efficient multiview 3d reconstruction for multiview diffusion. ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
Qingnan Zhou and Alec Jacobson. ThingilOk: preprint -XXv:1605.00497 2016.
A dataset of 10,000 3d 3d-printing models. arXiv
A
DATA PREPROCESSING
The data preprocessing is based on (Zhang et al., 2022).
A. VOLUME POINTS SAMPLING.
We sample volume points uniformly in the bounding sphere.
vol 250000
vol col__oints np. random. ranon (N_vcl,
vol_ _oo_ppintts__ vol_points / np.linalg. norm (vol_points, axis=1) [:, None! np.sqrt (3)
vol points vol points np.power (np. rwn.oa.aoo...mm rand (N vol), 1./3) [:, None
A.2 NEAR POINTS SAMPLING
The near-surface points are obtained by sampling mobsssann-iittered surface points.
N_near 125000
surface_pas pacaaccpppintsssseeee N_near x
hear_points
surface points np. random. nor.al (scale=0. 000, size= near, 3)),
surface points surface_points np. random. normal (scale=0. 05, size= (N near, 3)1,
@ear_points coccattatee (near_peints)
14

---

Published as a conference paper at ICLR 2025
B DATA AUGMENTATIONS
Random axis scaling. The augmentation is from (Zhang et al., 2022). We randomly sample scaling factor for each axis which ranges from [0.75, 1.25].
Unit sphere normalization. the point clouds is 1.
We normalize each mesh to a unit sphere, i.e., the max point norm of
1 vertices V (v. (w.max
v.min (axis=0)) /
distances = np. mp..inalg.nomme norm (v, axis=1)
scale = 1 / mp.max (distances)
scale
Random rotations.
We apply random rotations during the training of the autoencoder,
coos a
sin a
0
cos 3
sin 9
cOs 7 sin 7 [
0
R(a, B,,)
sin
0
0
0
(13)
0
sin
COS
sin
cos 7
where 3, and are yaw, pitch, and roll, respectively. Our meshes are firstly normalized into unit sphere. Thus after random rotations, the models will still inside of unit sphere.
C REGULARIZATION
The proposed regularization (see Table is implemented with layer normalization (PyTorch code).
network definition
self.ftl. proj nn Linear (x dim, z_dim)
self.ftl_nore norm = nn. nn. LayerNorr (dims, elementwise_
eps=le-6)
network forward
self.ftl_non (sclf (self. fto_projj )x))
D TRAINING TIME QUERY POINTS SAMPLING
the previous work (Zhang et al., 2022), the sampling strategy is uniformly sampling 1024 points in the bounding volume during training. We found this is not working on Objaverse. Since lots of meshes have very thin structures, this strategy will cause no inside points to be sampled during training. This heavily imbalanced data classification severely affects the occupancy loss.
We propose the following solution. each iteration, we make sure half of the points have positive labels and the other half have negative labels.
E TRAINING LOSS
The loss is binary cross entropy as in previous work (Zhang et al., 2022). Formally, we have
[BCE (o(p).
L EpErs
(14)
In practice, we use the empirical loss
Epegue |BE (0lp). + 0.1 Epe Quaa
[CC (O)p).
(15)
Here, Quo is the set of volume query points, and Quear is the set of near-surface query points.
F DIFFUSION
We use the formulation EDM (Karras et al., 2022) for the diffusion models. The inference/sampling algorithm is also taken from this paper.
15

---

Published as a conference paper at ICLR 2025
23, 22, 21
23, 32, 21
23, 22, 21
E3, 22, 21
Figure 13:
Latent with red color Z means it is replaced by Gaussian noise. Latent with blue color Z means it is generated with the diffusion models.
G LATENTS ANALYSIS
We analyze how latents are affecting the final reconstruction. The latents partially replaced by standard Gaussian noise (this is because our latents are also zero mean and unit variance). We show the visual results in Fig. 13.
16

---

Published as a conference paper at ICLR 2025
G00E
NOIT
Figure 14: Results from 3DQD (Li et al., 2023b) and LION (Zeng al., 2022).
Table 6: Generative result comparison.
chair
3DILG
VecSet
Ours
table
3DILG
VecSet
Ours
surface-FPD
0.96
0.76
0.64
surface- PFrace-FPPD
2.10
1.19
1.12
surface-KPD) (x 1x
1.21
0.70
0.57
surface-KPP (x10*)
3.84
1.87
1.72
H MORE COMPARISONS
We show additional visual comparisons with 3DQD and LION in Fig. 14.
I ADDITIONAL GENERATIVE METRICS
We present results in Table 6 for the "chair" and "table" categories for diffusion model using the generative metrics proposed in VecSet. Both models are trained on ShapeNet and have similar numbers of parameters, as detailed in Table 3 of the main paper.
17

---

Published as a conference paper at ICLR 2025
Figure 15: Our generated results. Compared to Fig. 14, we can generate clean, sharp, and detailed shapes.
Table 7: Results on different regularizers.
Loss
16 epochs
32 epochs
48 epochs
60 epochs
72 epochs
KI.
0.1438
0.0444 0.0409
0.0276
0.0217
0.0183
Ours
0.1320
0.0271
0.0217
0.0183
J ABLATION STUDY FOR THE NEW REGULARIZAR
We can also show that our regularizer has similar reconstruction behavior in Table 7. To evaluate the new regularizer, we conducted ablation study using the VecSet codebase. We compared the volume reconstruction loss of both methods and observed that the performance is nearly identical, with the new regularizer even showing advantage during the initial epochs. This demonstrates that the proposed regularizer can achieve results comparable to the commonly used KL divergence.
18


Reviews:
[{"review_id": "cZth2ENmnS", "reviewer": "Reviewer_g1aY", "summary": "This paper introduces a novel hierarchical autoencoder that compresses 3D models into a highly compact latent space, designed to handle large-scale datasets and support generative modeling using diffusion. Unlike previous approaches, the hierarchical autoencoder works on unordered sets of vectors, with each level controlling different geometric details. The model effectively represents a wide range of 3D models while preserving high-resolution details, and it reduces training time and memory usage compared to the baseline. Additionally, the authors propose a cascaded diffusion framework for generative modeling in the hierarchical latent space, allowing control over the level of detail in generated 3D models.", "questions": "For the rebuttal, please refer to the Weaknesses section. Additionally, I have a few questions and suggestions:\n\n- For the diffusion transformer, conducting cross-attention only once may not be sufficient.\n\n- In Figure 1, there is no explanation for 'FtoL'.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method extends prior work VecSet to a hierarchical architecture, which improves generalization ability.\n\n- The hierarchical autoencoder encodes the 3D shape into different levels of latent representations, with each level controlling different geometric details. This feature is highly beneficial for 3D generation.\n\n- The writing in this paper is clean and easy to follow. The comparison with previous work (Table 1) is a valuable addition.", "weaknesses": "- The improvement in training time (by 0.7×) and memory consumption (by 0.58×) does not seem significant. In this case, having three levels of latent representations might be too heavy.\n\n- The experiments presented in the paper involve up to 2K latent representations, which is not a substantial sequence length for Transformers with Flash Attention. Recent work [a] has scaled up to 64K latents.\n\n- When using three levels of latent representations, we need three levels of diffusion models, which may lead to additional error accumulation. It would be worthwhile to mention how this issue is addressed for the proposed diffusion model.\n\n- For the proposed regularization, it seems to force the datasets to share the same mean and standard deviation (Eq. 7), which could negatively impact model performance. In contrast, for KLD, very small values are typically used to avoid harming reconstruction. One suggestion is to try the method from [b] without applying any regularization.\n\n[a] Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scale. https://openreview.net/forum?id=mhzDv7UAMu\n\n[b] AutoDecoding Latent 3D Diffusion Models. NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SAxBKfIBkN", "reviewer": "Reviewer_uqmz", "summary": "The paper combines the idea of hierarchical VAE (Vahdat & Kautz, 2020) with VecSet (Zhang et al., 2023), leading to a 3D shape auto-encoder with hierarchical latent space. The proposed mode takes point clouds as input, and produces an occupancy field, which follows VecSet. It is more efficient than a single-level VecSet autoencoder, while being scaleable to large 3D datasets. The proposed auto-encoder outperforms VecSet in terms of reconstruction quality, while being more efficient. When coupled with a cascaded generative model (e.g. a cascaded diffusion model), this also enables control over the individual level of detail of the generated shapes.", "questions": "For the experiments presented in Table 4, Table 5 and Figure 8, I wonder if both LaGeM and VecSet are using the same latent size and type of latent regularization? If not, the comparison could be unfair, as these differences could dominate the performance difference rather than the hierarchical architecture itself.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The proposed autoencoder significantly outperforms VecSet, the previous work, in terms of reconstruction quality and generalization capability.\n* The paper demonstrates the controllability of individual level-of-detail, which is not possible with previous works due to having only a single level of latent vectors.", "weaknesses": "* It is well-known that having KL divergence on the latent trades reconstruction quality for a smoother and more compact latent space. Such a well-behaved latent space could be helpful for the performance of the upstream generative model. I thus won't consider removing the KL loss \"new regularization technique\" (L477-478).\n* The idea of controllability over different level-of-detail is interesting. However, according to Figure 13, its effect is very subtle and not particularly useful. This does increase the complexity and training cost of the diffusion model however, as stated in the Limitation section.\n* The effect of using different latent regularizations (Table 2) is heavily discussed in Sec. 3.2 but not ablated. \n* Table 3: It would be clearer if each column had a header explaining their differences and giving the setting a name.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4PjWT95Zjc", "reviewer": "Reviewer_Uog4", "summary": "This paper introduces a novel 3D autoencoder called LAGEM, which maps 3D models into a highly compressed latent space. The key contribution of this paper is the hierarchical autoencoder architecture, which takes 0.70x the time and 0.58x the memory compared to the baseline. Experiments on Objaverse and ShapeNet demonstrate promising results.", "questions": "Please refer to the weaknesses above.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel hierarchical 3D autoencoder with faster training time and low memory consumption, addressing the expressiveness of SoTA method 3DShape2VecSet which is unable to scale to larger datasets.\n2. Extensive experiments demonstrate the method's efficacy, outperforming previous state-of-the-art methods on key datasets.\n3. The paper is generally well-written and easy to follow. The figures are helpful in illustrating hierarchical architecture.", "weaknesses": "1. My main concern is whether having a better 3D autoencoder will lead to a better 3D generative model. On one hand, a better 3D autoencoder implies a higher upper bounder on the quality of the generated results. On the other hand, it requires that the latent space be smoother and easier to learn. Therefore, it would be even better if quantitative metrics for the 3D generated results could be provided.\n2. Training a diffusion model on multi-levels takes a lot of training time. So is it possible to only train a single diffusion model to generate latent codes. And train a feed-forward network which take the latent code from the previous level as input to predict the latent code for the next level.\n3. Is it possible to generalize the LaGeM to scene-level datasets like Matterport3D or the Replica dataset?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uymose7tH6", "reviewer": "Reviewer_J6bH", "summary": "This paper introduces a novel hierarchical autoencoder that maps 3D models into a highly compressed latent space. The hierarchical autoencoder is specifically designed to tackle the challenges arising from large-scale datasets and generative modeling using diffusion. The paper also proposes a cascaded diffusion framework where each stage is conditioned on the previous stage based on the above hierarchical autoencoder.", "questions": "The authors claim that it is difficult to scale VecSet to Objaverse training (#L328) and why?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and effectively expresses the motivation and role of hierarchical learning. \n\n2. The reporting of experimental hyperparameters is very detailed, enhancing the technical solidity of the paper. \n\n3. The model has been tested on a large number of datasets, validating the robustness of the proposed method.\n\n4. Additionally, I enjoy Fig. 13.", "weaknesses": "The main weakness of this paper lies in the lack of comparison experiments. \n\n1. Regarding reconstruction accuracy, this work is only compared with one baseline (VecSet). In reality, there are more comparative options for reconstruction, including autoSDF[1], LION[2], and 3DQD[3]. \n\n2. Additionally, this work does not compare the quality of generated results with other baselines. The authors have listed several potential comparison baselines in Table 1. From the perspective of visualizing generated quality, the quality of the generated results does not show a significant advantage over many baselines listed in Table 1. \n\n3. There is also a lack of ablation experiments, such as the rationale behind using 3 levels of latent features.\n\n[1] AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation\n\n[2] LION: Latent Point Diffusion Models for 3D Shape Generation\n\n[3] 3DQD: Generalized Deep 3D Shape Prior via Part-Discretized Diffusion Process", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)