Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a new efficient 3D shape representation to reduce computation cost while also being able to preserve geometric details. The authors propose to represent 3D shapes using octree structures such that the hierarchies focus on modeling the surface geometry, where the leaf nodes additionally predicts implicit functions that returns the occupancy for each query 3D point. Both the hierarchical inference and prediction are designed to be differentiable such that the octree structures can be discovered and learned from data. Experiments show that the proposed representation achieves the state of the art on the ShapeNet reconstruction with applications of shape interpolation and completion.
main_review: Strengths:
+ This work combines a neat idea of blending octrees (often used for efficient voxelized shape inference/prediction) and implicit surfaces (for modeling dense and granular surface structures). It saves unnecessary computational costs by terminating early in monotonic occupancy/freespace regions to focus more on modeling the surfaces. I like how the hierarchy can be handled with MLPs, which would be much easier to parallelize while being adaptive to different numbers of input nodes, unlike sparse 3D convolution which requires specialized designs.
+ The use of additional variables $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$ for predicting recursive octree structures is nice, whose differentiability allows end-to-end training.
+ The proposed method achieves the state of the art performance on the ShapeNet reconstruction benchmark, and the new representation shows good results on scene reconstruction of indoor room scales as well.

Weaknesses:
- It is unclear why $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$ are necessary in the encoder; maintaining architectural symmetry does not seem important here. Furthermore, it is not clear why the hierarchical encoder is important. One can preprocess the input shapes into different 3D representations (meshes, point clouds, naive voxels) and use other encoders to get the latent codes, and still use the hierarchical decoder for shape reconstruction. The effect of the proposed encoder architecture is not very related to 3D shape predictions as presented in the experiments. To validate that the new encoder indeed learns a better latent space, the authors should run another experiment on comparing the learned latent space, e.g. by fitting a linear classifier on the trained latent codes as a measure of discrimination and generalization.
- The VAE reparametrization seems like an orthogonal component to the paper and only allows the application random shape generation. I would suggest removing it, as the experiments are more focused on shape reconstruction.
- Are the implicit occupancy decoders (at the leaf nodes) shared? I am assuming yes (like the encoders), but it is not mentioned anywhere in the paper. Also, are the intermediate MLPs shared as well? If so, would it be possible to apply the MLPs (in the decoder part) for an arbitrary number of times until the desired level of details is reached, adaptive to the threshold $\tau$?
- An ablation on the $\tau$ value should be included to better understand its effectiveness. The expectation is that when $\tau=1$, the proposed decoder would become OccNet [34]; if $\tau=0$, it would become OGN [49].
- In Sec 4.1, was a single network trained on multiple shape categories together, or was it category-specific?
- I'm confused by L280-282. Why was SDF computed where (binary) occupancy prediction is the target goal?
- It would be helpful to also compare the ShapeNet reconstruction against OGN [49], as it shares a very similar octree generating architecture but for voxels.
- The setting in the scene reconstruction experiment is unclear. A brief overview of the 3D-Front dataset should be provided (e.g. how many train/test examples). It would also be helpful to provide quantitative results and more visual comparisons for this experiment.

Other minor problems:
- It would be helpful to enlarge Fig. 4, as it is difficult to compare in detail. A page-width figure would be recommended. It would also be helpful to visualize important parts where the proposed method is significantly superior to the baseline methods (e.g. the bars of the chairs).
- L129: DeepSDF should not be considered as concurrent work, which is a bit misleading. It would also be helpful to elaborate how this work is different from ACORN [32], as both works share a large similarity in hierarchical modeling of signals (e.g. 3D shapes).
- L28-29: I don't think voxels are explicit representations since it encodes occupancy at 3D locations, which is different from meshes or point clouds.
- L247: the notations $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$ are overloaded with that from the encoder. It is suggested to use a slightly different notation to discriminate, e.g. $\boldsymbol{\hat{\alpha}}$.
- An important reference is missing: OctNet [A], which was the first to use an octree structure for inferring 3D shapes.

[A] Riegler et al. "OctNet: Learning Deep 3D Representations at High Resolutions." CVPR 2017
limitations_and_societal_impact: Adequately addressed
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper describes a learned shape representation that is based on hierarchical oct trees that store local implicit functions at the voxels. The implicit functions represent the geometry. It offers higher fidelity and smaller memory footprint than using local implicits without oct trees (LIG, [26]). In comparison to other oct-tree based methods, the proposed approach offers the ability to learn oct-tree hierarchy. 
main_review: The proposed method is technically sound and achieves good results and improvements over the state of the art. Ability to learn (encode and decode) the hierarchical structure of the octtree, seems like an interesting contribution that can be used in other spatial representations, beyond storing implicit functions. Quantitatively, the proposed method also offers improvements over previous work. 

I suggest authors revise their related work section to discuss prior learning techniques that used OctTrees in more details. Right now  only OpenVDB [39] and NGLOD [48] are discussed, while lots of highly relevant techniques [52, 54, 53, 49] are only casually mentioned on line 91. I think they need to be discussed more, especially [54] which is further used as a baseline.  Paper "Hierarchical surface prediction for 3d object reconstruction" [24] is only mentioned in the context of data pre-processing, but seems highly related. 
limitations_and_societal_impact: Acceptable
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposed a learnable hierarchical implicit representation for 3D learning, combined with a network architecture that can probabilistically predict hierarchical structure of 3D shape. Experiments on showed some improvements over baselines (although I have some questions, see below)
main_review: Strength:
1. The paper is tackling an important and interesting problem for 3D shape modelling as the memory consumption can increase dramatically if the resolution goes up, and the idea proposed in the paper is intersting as it  can differentially generate hierarchical structure (octree) of implicit functions, which can potentially for other tasks.
2. The paper had a wide range of experiments to demonstrate the performance including shape reconstruction, generation, interpolation, completion & scene reconstruction.
3. The paper is well-written and easy to understand

Weakness:
1. The most related work for this paper is NGLOD [48], the paper cited and discussed in the related work section (Line 109-116), however, it is very hard to claim the paper is concurrent work, as NGLOD  is already accepted to CVPR, and released almost half a year before neuritis ddl, and the paper has already released the code. The paper also argued the advantages of making the occ-tree generation to be differentiable comparing with NGLOD[48], which used a fixed oct-tree, however, in the main experiments, there's no comparisons with NGLOD that can support this argument, I'd recommend the author to add this comparison.

2. Novelty. Predicting hierarchical structure of 3D shape has been studied in many papers in computer vision literatures, as in the cited papers [49, 52, 53, 54], but they're operating on the voxels, the paper extends the ideas from these papers into the implicit function by adding an implicit decoder. Besides, considering many similar works (NeuralLoD[48], ACORN[32]), the contribution dosen't reach the high bar of neurips.

3. Minor problems: a) in table 2, using different level of both LIG and the method in the paper would have different performance, could the paper also provide the preformance of both methods at different level? otherwise it's hard to say which one is better, b) in the scene recconstruction experiments, what's the input? what exactly this experiment is doing? c) in the shape completion experiments, could the author provide quantitative comparison? looking at only two shape would have bias.

Typo: Line 314: Table 2
limitations_and_societal_impact: There's no discussion on the social impact in the main paper
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a hierarchical implicit grid representation for modeling implicit shape-spaces. While concurrent research has focused either on hierarchical representations in overfitting mode, or on fixed grids in shape-space mode, the paper proposes a technique that simultaneously satisfies these two objectives. The method is tested on a few classes from the synthetic SDF'ied shapenet against a few methods, and has seemingly good reconstruction performance.
main_review: The ability of modeling adaptive/learnable hierarchical representations is a very exciting research direction, and this paper is one the few that has been looking at it.
However, the paper makes the mistake to focus the entire analysis on reconstruction accuracy, rather than showing the capabilities to interpolate between trees with drastically different topologies (and all the ablations / explanations that justify why a VAE model make this possible).
This, together with the low'ish quality of the writing (see the many, too many, questions and suggestions below), is what makes me rate the paper as a weak reject.
**I might raise my score to a weak accept, as a way to be friendly to the fact that NGLOD partially scooped the work, as far as my questions are properly answered.** (RAISED)


1. Q: what happens if you disable the VAE loss? I was amazed not to see this ablation, nor a visualization of how the topology of the representation changes during interpolation tasks.
1. Q: it is very difficult to understand the quality of a latent space from screenshots... do you have a video? Further, in this video a side-by-side comparison to the IM-Net VAE shape-space (to understand the differential gain caused by the hierarchical model) is essential?
1. Q: can you justify why you only use 5 ShapeNet classes? (this is usually a red flag that results were cherry picked, unless you provide a very strong reasoning)
1. Q: while CD is commonly used, EMD is rarely seen, while mIOU (for solids) and F1 are much more common. Can you justify why were they not reported?
1. Q: I am quite concerned about Table2, where I was expecting a **significantly** bigger advantage in terms of memory vs. LIG (main competitor with a fixed grid). You argue the gains are more evident as you consider deeper trees, but provide no rationale on why subdivisions at 5,6,7 levels were not reported? (note L7 is the standard Poisson reconstruction subdiv level). Further, it would have been good to see the CD scores in this analysis as well... Why were they omitted?
1. Q: Discontinuities solved "a-la LIG" is not super elegant, increases memory usage (overlap) and causes visible artefacts (top-right of Fig.4). Why is the NSVF/NGLOD solution of interpolate latent codes not feasible in your setting?
1. Q: how can one (me) be sure that the generative model has not just retrieved training samples? (show closest model in the training set w.r.t. CD from the generated result to certify this)

Finally, while ACORN / NGLOD are contemporary, I was quite disappointed not to see some qualitative comparisons (would have not affected the rating).


## Further questions (Q) and suggestions to the authors

1. Q: what were the improvements that CVPR/TVCG asked, and that you applied? I was expecting to see this in the supplementary
1. Q: why did you not discuss "Neural Sparse Voxel Fields" from NeurIPS'20? While this is radiance rather than SDF based, the underlying hierarchical structure is very similar 
1. There is a disturbing amount of grammatical mistakes that a careful proofread by a native speaker would be able to fix. e.g. at lines L{88,89,149,193,207,214,228,235,247,278}, and this list is not exhaustive
1. please sort citations in `\cite` environments! [3,18,26] vs. your [26,3,18], it makes it faster to look up what you are referring to
1. I can see why you discuss generative modeling in the related works after reading the technical/results sections, but it was not clear at all when reading the introduction. The reasoning is that you are proposing a hierarchical shape-space, while ACORD/NGLOD are mostly overfitting. As such, I would **strongly** recommend you move the discussion of concurrent work to the intro, and mention there the focus on generative modeling? Currently the structure of intro/related sections confused me a little.
1. Q: I am quite confused about the notation surrounding (1)... why do you need two aggregation operation? If $\mathcal{V}$ is variance, then it contains an expectation, but then you also have $\mathbb{E}$. Yet, the metric is the one of a patch, so I would expect only one expectation in this expression...?
1. Q: Why is (1) more meaningful that a more classical the divergence of the gradients within the patch (which is then related to curvature)? This is what Poisson surface reconstruction uses, coupled with a density estimator (like what you do for weighting the computation of the loss)
1. Fig.2, you'd mention that the encoders are shared, also, if they are... why bother with giving them indices? 
1. Q: I think you made a mistake in preparing the figure, as the VAE constraints should affect both mean and variance, while you only show it on the variance branch? why? 
1. L199: undefined $\mathbf{E}$ in Fig.2?
1. L204 and L204: overloaded CNN $\mathcal{V}$ and variance operator $\mathcal{V}$?
1. Q: why is the VAE prior only applied to the **global** latent code? You have a hierarchy, how do you justify not having it at every level?
1. L243: a bit more precision than `classifiers` would be good... is it just a linear layer with a two-class softmax?
1. L260: the importance of pre-training is enormous (supplementary), and should be quickly mentioned in the main text.
1. L273: summarizing the set of experiments in the supplementary is a good practice?
1. Q: in Sec 4.2, I would recommend to call this is auto-encoding (reconstructing shape collections) rather than reconstruction (reconstructing a single shape/overfitting).
1. L304: comparisons in "c"?
1. L316: you are referencing the wrong table
1. Q: can you justify the weird failure case of LIG in Fig.4? I would expect LIG to just have slightly smoother results than your method. If there is a mistake in how the comparison is performed, the metric in Table.1 (for LIG) could be completely different.
1. Table2: very unclear whether this is an amortized results (average) or whether it's on a single object/scene?
1. Q: Sec 4.4: while I am pretty sure in 4.2 you intend auto-encoding, I am similarly pretty sure in this section you are referring to overfitting. If so, why not compare to methods like SIREN?
1. Q: Sec 4.5: why are you only showing qualitative results? If no quantitative is possible, a clear rationale should be provided! 
limitations_and_societal_impact: No, and it's fine.
review_title: null

Meta-review: Post rebuttal, the paper was the subject of extensive discussion both between the authors and reviewers and between the reviewers themselves. The reviewers were overall generally in agreement with many facts about the paper, but had good faith disagreements in terms of where to draw boundaries for novelty and contribution. The AC examined the reviews, rebuttals, and discussion and is inclined to agree with the reviewers advocating for acceptance. The AC is persuaded by overall perspective of reviewers 2BPG and Nodw that the paper contributes valuable insights to the field (presented in committee discussions). Moreover, if the reviewers are satisfied that their clarity concerns can be addressed in a revision, then the AC is as well.

The AC would make the following requests:
- The authors should carefully read the comments of the reviewers and be sure each commented is addressed in the final version of the paper. The rebuttal and a promise of revision was persuasive in this case. 
- The authors should add a stronger discussion of the social impact in the paper.
Decision: Accept (Poster)