Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a geometry representation based on irregular grids for generative shape modeling.
strengths_and_weaknesses: Strengths:
- It is an interesting idea, especially when combined with transformers and vector quantization.
- Some good results on reconstructed and generated shapes, although I am not convinced with the comparisons.

Weaknesses:
- The advantage over point cloud-based generative modeling is not well established, both conceptually and experimentally.
- It is a bit of a stretch to call this a grid-based algorithm and only consider grid-based alternatives.
- The reconstructions are only compared against other networks, there are no comparisons to surface reconstruction methods e.g. Poisson or edge-preserving MLS.
- The proposed generative tasks are only on shape-net, no real images or point clouds.
- Memory and runtimes are not clear.

questions: My suggestions for improvement are:
- Having examples with non-shape-net data to illustrate robustness and utility in practice.
- Having comparisons with surface reconstruction techniques other than only neural grids. The point clouds look clean and seem to sample the shapes well.
- If no good results with real data for generative tasks, I would drop that claim and focus on the new representation's ability for reconstruction, efficiency, spatial adaptivity, etc.
limitations: Please see above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method for detailed shape reconstruction from point clouds and images, as well as shape generation. The method utilizes FPS and KNN together with a PointNet to create patch embeddings of train shapes, and then it encodes a shape indicator function using a transformer architecture. The paper demonstrates impressive shape reconstruction and generation results for all of the aforementioned tasks.
strengths_and_weaknesses: Strengths
- The paper presents a novel method for encoding local shape information using irregular latent spaces and using transformer-based autoregressive models for shape reconstruction from point clouds and images.
- The paper presents impressive - accurate and detailed, shape generation results from high-res and blurred images, using compact shape latent spaces.
- The presentation is very clear.
- The paper presents a comprehensive set of experimental results illustrating the advantages of the proposed method and comparing it to current SOTA methods.

Weaknesses:
- A possible weakness of the proposed approach may be its sensitivity to shape sampling. E.g., how would the method work if the input point clouds were not all sampled at N = 2048 points? How were these points obtained? Would irregular sampling density affect the results of the proposed method? This possible limitation needs to be addressed in the paper, with some experimental results and a discussion.
Parts of the method's description which require further classification.
- In Figure 4, what’s the output of the model? How the point cloud, or another shape representation, is decoded from it?
- Figure 5 is inconsistent with Figure 4. Show a block diagram of the proposed architecture, as you do in Figure 4.
- In Section 4, how are x_{i,k} obtained for different i-s? What is the difference between x_{i,k} and x_{j,k} for different i, j? Please explain.
- In Section 5, further comparison with IF-NET is needed: the proposed method slightly improves over IF-NET. How does it compare to IF-NET (and other methods) in other respects - resource consumption, train and inference time, etc.?
- Explain the purpose of Eq.(8) and Eq.(9).
Comments regarding supplementary material.
- In A.4., how do the methods compare in terms of memory and runtime?
- In A.6., ll.62-63 - “Bad” in what sense? Non-realistic generations by the proposed method should be highlighted as well.
Other comments.
- In l.130, explicitly state which positional encodings are used, for completeness of the presentation.
- Introduction section needs to be revised to more clearly describe the main aspects of the proposed approach.
- A related work which should be cited: Y. Li, S. Pirk, H. Su, C. R. Qi, L. J. Guibas, FPNN: Field Probing Neural Networks for 3D Data, Neural Information Processing Systems (NIPS 2016).
- In Section 6.1, l.214, provide reference for GPT.
- Nit: In Table 1, consider writing all numbers in the same manner such that it will be clear which parameters are identical for different methods.
questions: Please see and address the questions raised in the previous section.
limitations: The authors need to address further a potential limitation of the method, namely, its sensitivity to changes in sampling of the input point clouds.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a new 3D shape representation for 3D generative modeling. The proposed shape representation is formed as irregular grids of latent vectors in the 3D space that roughly forms the shape, which can be encoded with Transformer-based networks and decoded via autoregressive generation, similar to VQ-GAN. Experiments on 3D shape generation conditioned on different context is performed on ShapeNet and shows promising results.

strengths_and_weaknesses: Strengths:
- The authors proposed a very interesting idea of bringing recent advances on generative image modeling into 3D shapes, specifically VQ-GAN. It nicely brings in the sparse nature of 3D shapes, and bridges together commonly accepted differentiable point cloud processing operations with autoregressive generation.
- The experimental results are pretty promising. I especially like the fact that, conditioned on a single input image, the model is capable of generating different shape modalities, which is something previous deterministic shape reconstruction methods cannot achieve.
- The paper is nicely written and the presentation is clear.

Weaknesses:
- Maybe I missed it, but it was not clear how the autoregressive generation ordering was determined (L156-158). I think the authors should also provide some discussions on whether (or how) the model is / is not invariant to the permutation of such ordering; for completeness, results comparing different input permutations may also give a better understanding.
- For the "image-conditioned generation" experiments comparing single-image shape reconstruction, I would like to understand more where the actual performance gain comes from. Besides the new irregular latent grid representation, the entire backbone architecture (GPT) is also much more powerful than OccNet (ResNet-18); even the pretraining data is entirely different. Therefore, I am not totally convinced that all the benefit comes from the new representation. I think a critical missing experiment is to ablate this factor; specifically, a baseline that uses GPT but predicting occupancy like OccNet should be included to verify that the irregular latent grids do have its merits.
- For the image-conditioned experiments (following the above concern), it was not clear whether there are still input point clouds being conditioned. If yes, then the comparison against OccNet becomes yet more unfair; if not, it is not very clear what the input pipeline would look like. My guess is that a full shape reconstruction network (Fig 3) is trained first, then only the Transformer part is taken for the experiment (with the PointNet discarded); it would be good if the authors could clarify.

questions: It would be great if the authors could address all the concerns listed above.
limitations: The limitations and potential societal impact have been adequately addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This submission proposes to irregularly distribute local SDF latent codes in 3D, to adapt to shapes and not be restricted to a 3D grid with fixed resolution.
In addition, authors present ways to quantize their representation, and use it in the generative setting with uni- and bi-directional transformers.
strengths_and_weaknesses: Strengths:
- excellent technical presentation. The use of colors in the text and equations to refer to different pipeline parts is a great idea.
- generating multiple plausible outputs given 1 single conditioning signal (eg. Fig. 10) is a very good use of probabilistic models
- results look qualitatively good, and the proposed pipeline is properly compared against recent models (IF-Net, ConvOccNet) that distribute latent codes on fixed grids.


Weaknesses:
- Quantizing 3D positions of local latents sounds very similar to using a grid. What is the difference here?
- Some very related papers such as "Local Deep Implicit Functions for 3D Shape" [CVPR 2020] are not mentioned. In this one, "the global implicit function is decomposed into the sum of N local implicit functions.", not on a grid...
- The intro+related Sections mention shape representation and generation, while the method section starts with an input point cloud.
questions: l. 33 : Why is fixed length representation a benefit? Wouldn't an adaptive representation be more useful, with a tiny representation for simple shapes, and long representation for complex ones?

l.141 What is the motivation behind the non linear interpolation between local latent codes? Could it be ablated?
limitations: Compared to autodecoders, this probabilistic pipeline can no longer represent a shape with a latent code.
review_title: null

Meta-review: metareview: All reviewers agree to accept this work, which presents a creative new shape representation for 3D generative modeling.  The negative aspects raised by the reviewers are fairly minor, and most were addressed during the rebuttal phase (please be sure to incorporate all comments/additional results into the final camera-ready version).  During the post-rebuttal discussion, reviewers suggested nominating for a spotlight given the new results on realistic data.
confidence: Certain
Decision: Accept