Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a hierarchical architecture for point cloud generation, which is composed of two diffusion models for shape latents and latent points. 
The major difference between PVD and this paper is the hierarchical architecture and its ability to reconstruct corresponding meshes even though it comes from an off-the-shelf method SAP. 
Both qualitative and quantitative results are performed to show the effectiveness.

strengths_and_weaknesses: Strengths:
- Extensive results and ablations are given to show the effecitieness of the proposed method (in both the paper and supp). 

- The paper is well written and easy to understand. It adequately discusses related works. 

- In terms of generation quality, it truely beats several SOTA methods. 

Weaknesses: 

- The proposed pipeline is not particularly novel. It is more like a combination of PVCNN and the diffusion model. The mesh reconstruction method is also off-the-shelf. The main contribution might be the hierarchical generation architecture. 

- The paper misses insights for choosing specific backbone for feature extraction. There should be a comparison between different backbones. 

- SVR should be a common application for generative networks. However, It does not show any single-view recontruction results. 
questions: - Why choose PVCNN as the featue extration module? Author should include more insights and perform experimental results to show the reason. 

- For the shape interpolation application, the author should include results of other baselines. 

- Most generation methods can do single-view reconstruction. The paper may miss a comaprison on SVR from rgb or rgbd data like PVD does. 

- As the proposed method can recontruct surface, I think it is possible to compare it with SOTA explicit/implicit surface-based methods like AutoSDF and Deep marching tetrahedra. 

- In 'Single-Class Unconditional Generation', I can not see a clear improvement in quantitative results. What is the reason? 
limitations: I only saw one 'slow sampling' limitation in the paper. However, this limitation comes from the generative model they use. So I do not think they fully showed their method's limitations. I did not see any negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes the hierarchical Latent Point Diffusion Model (LION) for 3D shape generation. Different from previous works where denoising diffusion models (DDMs) operate on the point clouds directly, LION is based on a VAE structure with a hierarchical latent space that includes a global shape latent space and a local point-structured latent space. Meanwhile, two hierarchical DDMs operate on such two latent spaces instead. Exhaustive experiments demonstrate LION outperforms several state-of-the-art models on multiple ShapeNet benchmarks. Besides, owing to the adopted VAE structure, it is convenient to transfer the LION model to other relevant tasks, which only requires fine-tuning the encoder. Combining with a surface reconstruction model like SAP enables the generation of smooth 3D meshes.
strengths_and_weaknesses: Pros:
1. This paper proposes to employ denoising diffusion models (DDMs) on a hierarchical latent space, instead of raw point clouds. It's a pretty good idea and the effectiveness is supported by the impressive generation results and ablation study contained in the supp. material.
2. The adopted VAE structure makes the LION model flexible to transfer to different relevant tasks without re-training the computationally intensive DDMs. Besides, the experiments on shape denoising and voxel-guided synthesis also demonstrate the power of the hierarchical latent space.
3. LION model achieves state-of-the-art results on both single-class 3D generation and many-class 3D generation tasks on widely used ShapeNet benchmarks.
4. Combining with a surface reconstruction model like SAP enables the generation of smooth 3D meshes.

Cons:
1. The idea of employing DDMs on latent space and incorporating VAE structure is not new. Actually, several relevant papers [1,2] are adopting similar ideas in other domain. However, these papers are not cited and discussed in this manuscript.
2. The authors just select the most compelling results (on purpose) and put them in the main paper. As we all know, there are a lot of metrics for 3D shape generation evaluation. But the main paper only contains one metric, 1-NNA. Actually, the LION model doesn't always outperform baselines in terms of other metrics like MMD and COV according to Table 13 & 14 results in the supp. material.

[1] DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents. Kushagra Pandey, et al.
[2] Diffusion Autoencoders: Toward a Meaningful and Decodable Representation. Konpat Preechakul, et al.
questions: 1. For the shape interpolation, do you interpolate on both latent spaces, or just one?
limitations: 1. A necessary discussion with some closely relevant papers is missing.
2. The explanation of the modest performance on MMD and COV metrics should be provided.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper targets 3D shape generation and manipulation. Similar to previous works PVD and DPM [46,47] it uses a denoising diffusion model on point clouds. My understanding is that the novelty compared to these works is:
N1: using a global shape and a point feature (a.k.a. hierarchical latent space)
N2: combining the network with a shape as point [67] network to obtain a surface from the output point cloud.
The other claimed contribution are essentially results, that go beyond [46,47] in terms of performances (clearly better numbers on ShapeNet dataset) and applications (shape interpolation, guided generation)
strengths_and_weaknesses: This paper seems to clearly improve state of the art shape generation (note this is not my area of expertise, so I have to trust the authors the metrics and datasets are meaningful) the evaluation seems exhaustive and the exposition of the method is clear.

What is less clear to me is what the novelty actually is compared to [46,47] and what explains the performance gap. To me, the ablation tables 10 and 11 from the supplementary material would be the most interesting to understand this, but are still unclear:
- I think they should be moved to the main paper
- I couldn't match the results from these two tables to any in the main paper: to which should they be compared? is there a change in the method? or is this the effect of the randomness of training? (if so, some variance really seems necessary)
- having additional dimensions for the latent points seem to provide only a very small boost, I wonder if this isn't just noise that leads to this selection. Moreover, the fact that the selected value is D_h=1 (and using D_h=0 performs almost as well) is important for understanding why the method work and should be emphasized in the main paper
- from table 10, using shape latents also provides only a small boost, this together with the previous point really makes me wonder what part of the performance improvement is really due to the proposed hierarchical architecture v.s. other changes
- putting the two previous points together, I would really want to see numbers comparable to [46,47] in the ablation, to better understand the origin of the performance improvement, which is more unclear to me the more I look at the results.

To conclude, I am unsure about this paper, that falls slightly out of my expertise (not in diffusion models or generation). At first look it seems to really boost state of the art, but looking at the results and paper in more details, I couldn't figure out any clear, strong and well justified novelty (using shape as point is different but very low novelty, the hierarchical architecture seems different if not very original but in the end I am unsure which part of the improvement is really due to it) and I do not really understand where the gain in performance comes from (except the very large 300k hours of V100)
questions: see weaknesses (clearer ablation compared with SoA)
limitations: yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Authors present a generative model of 3D point clouds. This is a latent-space diffusion model, with variational encoder and decoder mapping between point cloud and latent representation; the latent space itself is split into a 'global' shape embedding, and 'local' embeddings of details, the latter itself structured as a sparser, feature-valued point-cloud. The model is first trained as a 'simple' VAE with gaussian prior/posterior, and then the diffusion model is fitted post-hoc to the aggregate posterior. The model is demonstrated on various tasks using ShapeNet data: unconditional generation; sampling 'variations' on a coarse model; interpolating shapes; and sampling meshes by integration with a differentiable Poisson solver. It achieves qualitative-reasonable results on all tasks, and quantitatively out-performs various point-cloud-based baselines on generation tasks.
strengths_and_weaknesses: Strengths:

- the model is novel, and the approach sensibly motivated

- quantitative results show significantly better performance than state-of-the-art baselines on unconditional (per-class) generation tasks

- qualitative results are impressive – particularly the displayed meshes

- there are very helpful and detailed descriptions of components (e.g. how interpolation is performed) in the supplementary; this mitigates the fact that the overall model is rather complex

- there is an ablation study in the supplementary that provides some additional motivation for various design decisions

- the writing is clear and easy-to-read throughout; overall I found the paper enjoyable and informative to read

Weaknesses:

- the technical contribution of the model itself is fairly small – DDMs on latent space have become fairly common, and even DDMs on point-clouds is not a new idea (though authors show that the proposed method outperforms other such approaches)

- most experiments use only three shapenet classes; it would be nice to see more diversity, or discussion of why this is not possible (e.g. if there are too few shapes in some classes to train per-class models successfully -- this should be stated as a limitation of the method if so)

- the mapping from point-clouds to meshes is responsible for the best-looking results in the paper (the meshes are significantly cleaner than the point-clouds); however this is apparently trained as a post-hoc stage rather than jointly with the generative model. It would be more elegant to train this end-to-end (the same could also be said for training the diffusion jointly with the encoder/decoder, however I appreciate that this is challenging)

questions: Exactly how computationally expensive is the model, e.g. to generate a single chair or train all parts of the corresponding model? This doesn't seem to be specified anywhere, only the total compute for the entire project.

Discussing only discrete-time diffusion in the paper (sec. 2) in spite of the model using exclusively continuous-time (for which full details are only in the supplementary) seems a little strange. It might be better to pull in a couple of paragraphs from the supplementary into the main text.

limitations: There is very minimal discussion of limitations; this should be expanded. There is sufficient discussion of broader impact (which is negligible).
review_title: null

Meta-review: metareview: This paper proposes a latent point diffusion model, LION, for 3D shape generation. The model builds two denoising diffusion models in the latent spaces of a variational autoencoder. The latent spaces combine a global shape latent representation with a point-structured latent space.  Comprehensive experiments are conducted to evaluate the performance of the proposed method.  The authors address the major concerns of the reviewers and strengthen the paper by providing additional empirical results. After the rebuttal, all four reviewers reach an agreement on accepting the paper because of the novelty and the state-of-the-art performance. The AC agrees with the reviewers and recommends accepting the paper. 
confidence: Certain
Decision: Accept