LaGeM: A Large Geometry Model for 3D Representation Learning and Diffusion

Authors: Biao Zhang, Peter Wonka
Primary area: generative models
Keywords: diffusion, geometry, generative model, 3d

Abstract:
This paper introduces a novel hierarchical autoencoder that maps 3D models into a highly compressed latent space. The hierarchical autoencoder is specifically designed to tackle the challenges arising from large-scale datasets and generative modeling using diffusion. Different from previous approaches that only work on a regular image or volume grid, our hierarchical autoencoder operates on unordered sets of vectors. Each level of the autoencoder controls different geometric levels of detail. We show that the model can be used to represent a wide range of 3D models while faithfully representing high-resolution geometry details. The training of the new architecture takes 0.70x time and 0.58x memory compared to the baseline.
We also explore how the new representation can be used for generative modeling. Specifically, we propose a cascaded diffusion framework where each stage is conditioned on the previous stage. Our design extends existing cascaded designs for image and volume grids to vector sets.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1729569171117

thread_type:
official_review

<thread_object>
creator: reviewer_j6bh

<content>
summary:
This paper introduces a novel hierarchical autoencoder that maps 3D models into a highly compressed latent space. The hierarchical autoencoder is specifically designed to tackle the challenges arising from large-scale datasets and generative modeling using diffusion. The paper also proposes a cascaded diffusion framework where each stage is conditioned on the previous stage based on the above hierarchical autoencoder.

strengths:
1. The paper is clearly written and effectively expresses the motivation and role of hierarchical learning. 

2. The reporting of experimental hyperparameters is very detailed, enhancing the technical solidity of the paper. 

3. The model has been tested on a large number of datasets, validating the robustness of the proposed method.

4. Additionally, I enjoy Fig. 13.

weaknesses:
The main weakness of this paper lies in the lack of comparison experiments. 

1. Regarding reconstruction accuracy, this work is only compared with one baseline (VecSet). In reality, there are more comparative options for reconstruction, including autoSDF[1], LION[2], and 3DQD[3]. 

2. Additionally, this work does not compare the quality of generated results with other baselines. The authors have listed several potential comparison baselines in Table 1. From the perspective of visualizing generated quality, the quality of the generated results does not show a significant advantage over many baselines listed in Table 1. 

3. There is also a lack of ablation experiments, such as the rationale behind using 3 levels of latent features.

[1] AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation

[2] LION: Latent Point Diffusion Models for 3D Shape Generation

[3] 3DQD: Generalized Deep 3D Shape Prior via Part-Discretized Diffusion Process

questions:
The authors claim that it is difficult to scale VecSet to Objaverse training (#L328) and why?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730615483206

thread_type:
official_review

<thread_object>
creator: reviewer_uog4

<content>
summary:
This paper introduces a novel 3D autoencoder called LAGEM, which maps 3D models into a highly compressed latent space. The key contribution of this paper is the hierarchical autoencoder architecture, which takes 0.70x the time and 0.58x the memory compared to the baseline. Experiments on Objaverse and ShapeNet demonstrate promising results.

strengths:
1. The paper introduces a novel hierarchical 3D autoencoder with faster training time and low memory consumption, addressing the expressiveness of SoTA method 3DShape2VecSet which is unable to scale to larger datasets.
2. Extensive experiments demonstrate the method's efficacy, outperforming previous state-of-the-art methods on key datasets.
3. The paper is generally well-written and easy to follow. The figures are helpful in illustrating hierarchical architecture.

weaknesses:
1. My main concern is whether having a better 3D autoencoder will lead to a better 3D generative model. On one hand, a better 3D autoencoder implies a higher upper bounder on the quality of the generated results. On the other hand, it requires that the latent space be smoother and easier to learn. Therefore, it would be even better if quantitative metrics for the 3D generated results could be provided.
2. Training a diffusion model on multi-levels takes a lot of training time. So is it possible to only train a single diffusion model to generate latent codes. And train a feed-forward network which take the latent code from the previous level as input to predict the latent code for the next level.
3. Is it possible to generalize the LaGeM to scene-level datasets like Matterport3D or the Replica dataset?

questions:
Please refer to the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720220984

thread_type:
official_review

<thread_object>
creator: reviewer_uqmz

<content>
summary:
The paper combines the idea of hierarchical VAE (Vahdat & Kautz, 2020) with VecSet (Zhang et al., 2023), leading to a 3D shape auto-encoder with hierarchical latent space. The proposed mode takes point clouds as input, and produces an occupancy field, which follows VecSet. It is more efficient than a single-level VecSet autoencoder, while being scaleable to large 3D datasets. The proposed auto-encoder outperforms VecSet in terms of reconstruction quality, while being more efficient. When coupled with a cascaded generative model (e.g. a cascaded diffusion model), this also enables control over the individual level of detail of the generated shapes.

strengths:
* The proposed autoencoder significantly outperforms VecSet, the previous work, in terms of reconstruction quality and generalization capability.
* The paper demonstrates the controllability of individual level-of-detail, which is not possible with previous works due to having only a single level of latent vectors.

weaknesses:
* It is well-known that having KL divergence on the latent trades reconstruction quality for a smoother and more compact latent space. Such a well-behaved latent space could be helpful for the performance of the upstream generative model. I thus won't consider removing the KL loss "new regularization technique" (L477-478).
* The idea of controllability over different level-of-detail is interesting. However, according to Figure 13, its effect is very subtle and not particularly useful. This does increase the complexity and training cost of the diffusion model however, as stated in the Limitation section.
* The effect of using different latent regularizations (Table 2) is heavily discussed in Sec. 3.2 but not ablated. 
* Table 3: It would be clearer if each column had a header explaining their differences and giving the setting a name.

questions:
For the experiments presented in Table 4, Table 5 and Figure 8, I wonder if both LaGeM and VecSet are using the same latent size and type of latent regularization? If not, the comparison could be unfair, as these differences could dominate the performance difference rather than the hierarchical architecture itself.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730746762615

thread_type:
official_review

<thread_object>
creator: reviewer_g1ay

<content>
summary:
This paper introduces a novel hierarchical autoencoder that compresses 3D models into a highly compact latent space, designed to handle large-scale datasets and support generative modeling using diffusion. Unlike previous approaches, the hierarchical autoencoder works on unordered sets of vectors, with each level controlling different geometric details. The model effectively represents a wide range of 3D models while preserving high-resolution details, and it reduces training time and memory usage compared to the baseline. Additionally, the authors propose a cascaded diffusion framework for generative modeling in the hierarchical latent space, allowing control over the level of detail in generated 3D models.

strengths:
- The proposed method extends prior work VecSet to a hierarchical architecture, which improves generalization ability.

- The hierarchical autoencoder encodes the 3D shape into different levels of latent representations, with each level controlling different geometric details. This feature is highly beneficial for 3D generation.

- The writing in this paper is clean and easy to follow. The comparison with previous work (Table 1) is a valuable addition.

weaknesses:
- The improvement in training time (by 0.7×) and memory consumption (by 0.58×) does not seem significant. In this case, having three levels of latent representations might be too heavy.

- The experiments presented in the paper involve up to 2K latent representations, which is not a substantial sequence length for Transformers with Flash Attention. Recent work [a] has scaled up to 64K latents.

- When using three levels of latent representations, we need three levels of diffusion models, which may lead to additional error accumulation. It would be worthwhile to mention how this issue is addressed for the proposed diffusion model.

- For the proposed regularization, it seems to force the datasets to share the same mean and standard deviation (Eq. 7), which could negatively impact model performance. In contrast, for KLD, very small values are typically used to avoid harming reconstruction. One suggestion is to try the method from [b] without applying any regularization.

[a] Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scale. https://openreview.net/forum?id=mhzDv7UAMu

[b] AutoDecoding Latent 3D Diffusion Models. NeurIPS 2023.

questions:
For the rebuttal, please refer to the Weaknesses section. Additionally, I have a few questions and suggestions:

- For the diffusion transformer, conducting cross-attention only once may not be sufficient.

- In Figure 1, there is no explanation for 'FtoL'.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a hierarchical autoencoder for 3D shapes that allows to train diffusion models in for generation in latent space.
The paper was well-received by all reviewers, converging to positive scores, recommending acceptance. The reviewers highlighted the good writing and extensive experiments, clearly showing benefits over previous methods. After rebuttal, remaining concerns regarding ablations and missing metrics have been resolved.

I agree with the reviewers and follow their suggestion with an accept recommendation.