Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes an efficient implicit representation to encode large scale datasets in real time: implicit recursive octree auto-decoder iROAD. The idea is directly learning latent code for each object in a hierarchy manner. During the inference, given the initial latent code $\mathbf{z}_0$, we can call iROAD recursively to get latent code for different level of details (LoD). To speed up training, this paper introduces a curriculum procedure to first train low LoD and gradually jump to higher LoD until the desired/final LoD reached. Experiments on synthetic datasets (ShapeNet and Thingi32) show that the proposed method outperforms existing approaches (e.g., DeepSDF, NGLOD) with a small memory footprint.
strengths_and_weaknesses: **Strengths:**

[S1] The paper is well motivated - compact and accurate representation of 3D shapes are always critial for robotics tasks. However, only a few approaches are proposed (NGLOD) to handle this problem. The idea of iROAD is simple yet very effective -> better than more complicated NGLOD.

[S2] The performance is great - iROAD outperforms to methods training a single network per model significantly. This inspires us that jointly training will encode more data priors and achieve superior performance. I like the generalization experiment which shows that the latent space is able to generalize on unseen objects (seen class).

**Weakness:**

[W1] Training time comparison for iROAD and other baselines (DeepSDF, NGLOD) for encoding the whole dataset. I am concerned that training auto-decoder in hierarchy way would significantly increase the computation load. More concise comparison with baselines especially NGLOD would be highly appreciated.

[W2] Quanlitative results?  It would be great to have more visual comparisons with existing approaches and see the difference in recovered surface (vs DeepSDF or NGLOD).  That would make the results more convincing.

[W3] More experiments on the robustness of iROAD would be more interesting. Currently, all results are conducted on synthetic datasets with dense oriented points. In practice, the points might be sparse and contain noise, it would be very interesting to see whether the joint learning can help overcome the noise and sparisty issue with priors from other objects.

**Others:** 
- why not reporting the storage for other LoDs in Table 1. Could we show the results of NGLOD with different LoD level?
- In Table 3, it would be great to be more explicit - showing the performance of NGLOD and iROAD for each level (low, medium and high) and the inference time.
- Good to have: Training time vs number  of objects
- Include more implementation details for baselines 
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The overall idea for this paper is simple but the performance is great. It outperform existing SOTA approach NGLOD and also achieves a large margin than some methods training a single network per model. The inference can be conducted in real time and the model storage is very efficient.
issues: See weakness. Primarly more comparison with baselines on the training time.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a highly-efficient data structure for encoding multiple 3D shapes in a low dimensional latent space. The key idea is to use a hybrid neural network / octree structure to store signed distance function (SDF) values. The latent vector defines both the Octree structure and, via a small MLP, the SDF values. The results show that the method exhibits excellent compression ratios and reconstruction accuracy. 
strengths_and_weaknesses: Strengths
=========
- ”Neural fields” models are known to be quite limited in terms of the scale of scenes they can represent. Leveraging Octrees therefore has great potential to get neural fields models to scale to large scenes, but a significant challenge lies in how to get the Octree structure to generalize across shapes. This paper presents a solution to this problem by encoding the Octree shape with the latent code.
- The architecture of the model is well designed and elegant. I especially like how the latent code recursively defines the Octree structure via splitting, much like you would construct a traditional octree. 
- The paper is very clearly written and nicely illustrated. A lot of effort has been put into communicating and explaining the idea.

Weaknesses
==========
- There is not much discussion in the paper of the tradeoffs or weaknesses of the approach. How long does recursive decoding of the Octree take, for example?
- The genrealization experiments are quite limited. How many sample points of the target object are used during the optimization and how is this affected by sampling fewer points? Can the latent vector be predicted from an image?

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, this is a good paper with a strong contribution in terms of a new hybrid neural+Octree data structure that can have applications in many areas in robotics and computer vision.
issues: Please address the questions in weaknesses above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a hierarchical, recursive latent space to learn an object shape representation.
Starting from a root latent vector, a neural network recursively maps this latent vector into 8 more latent vectors refining the initial one. This way, the authors achieve higher surface reconstruction capabilities while keeping computation and memory requirements low.
strengths_and_weaknesses: Strengths:
The paper is well written and I enjoyed reading it.
The idea, to my knowledge, of having a hierarchical latent space for representing an object shape is novel.

Weaknesses:
However, I have two concerns:
1) The biggest concern about this paper is the relation to robotics. I have no idea why this was submitted to CoRL.
2) The root latent vector has to contain all information about the object. Does this not counteract the idea of having an hierarchical representation of an object?
robotics_focus: Irrelevant to robotics
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Unfortunately, there is no relation to robotics in this paper. I have chosen weak accept, as I generally think the paper deserves publication, but I highly doubt that CoRL is the right venue for this work.
issues: Please clarify why it is sufficient to have a single root latent vector to represent fine details and why this is not possible with a monolithic model.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes an octree-based implicit representation of 3D shapes. Explicit representations of such data (point clouds, meshes) present a high memory footprint that increases with resolution, while the proposed approach represents a shape with a single latent vector and a decoding neural network able to generalize to multiple shapes, resulting in a very efficient compression ratio while preserving a high accuracy.
strengths_and_weaknesses: **Strengths:**
* Quantitative results show a clear improvement over prior work, considering the trade off between reconstruction accuracy, storage requirements and inference time. The high compression ratio, coupled with a fast decoding time make the contribution a very appealing solution for embedded robotics downstream tasks.
* The proposed method does not simply overfit a neural network on each 3D shape but can be trained with multiple complex shapes simultaneously. When the training dataset is sufficiently diverse, the approach exhibits good generalization properties to unseen shapes, showing that the network does not only memorize training shapes but learns to combine geometric primitives at different level of details. This claim is supported by several convincing experiments.
* The recursive decoding scheme based on an octree data structure is novel and relevant: it enables to limit the number of network parameters because decoding functions are shared between different resolutions. At inference time, the user can choose the desired level of detail, depending on throughput and accuracy requirements.

**Weaknesses:**
* Details are missing concerning how ground truth labels (occupancy, signed distance function and normal vectors of each cell) are defined. The paper states that input shapes are represented as an oriented point cloud (L112), but datasets used in experiments seems to provide meshes (L204). For the sake of reproducibility, I think that the data format of the input dataset should be clarified. While the preprocessing from point cloud to surface might be done with classical approaches, more details would be useful for an uninformed reader.
* No experiments are conducted on downstream robotics tasks (stated in the limitations section). The differentiability of the resulting surface w.r.t the input latent vector is an interesting property which might enable to learn the shapes from weaker (2D) supervision signals.  This is suggested in the related work section (L106), but not supported by empirical experiments.

**Questions/Suggestions:**
* As a conclusion of the proposed experiments, I understand that the decoding network embeds a mapping from latent keys to the geometrical primitives present in the training set, whereas the input vector encodes these latent keys for different resolutions in a compact way. As a result, a new shape can be integrated in the framework without modifying the network weights ("generalization" experiment), as long as all geometric primitives present in the shape have been observed in other shapes during training. Is that correct ?
* One drawback of implicit representations is the offline training time. How long training takes to fit a single shape ? A large dataset ? How fast is it to optimize only the input vector of a new shape in a trained network, compared with retraining everything ?
* I would expect that simple augmentations during training (random 3D rotations and crops of the shapes) should help to improve the generalization ability of the decoding network.
* Do you think that the proposed method could be applied on 2D signals using a quadtree data structure ?
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend "Strong Accept" for this paper. I believe that the contribution is quite strong, with a clear writing and thorough experiments. I believe that the paper is suitable for publication in its current form, even if some minor details should be added for reproducibility.
issues: As explained above, I believe that details are lacking concerning the input data: oriented 3D point cloud, meshes ? In particular, more explanation about how SDF labels are computed from a sparse point cloud would make the reproducibility easier.

I am quite confused about the "curriculum learning" description. L178 "Instead of initiating from the desired final LoD, we change the first LoD to be lower". Does it mean that only the LoD at lower resolutions are trained in the first place, and then finer details are introduced ? L182 "Once we reach a desired probability threshold theta, ..." This notation is used for network weights earlier in the paper and it is not clear to me which probability is mentioned. I think the writing of this section can be improved.

I am curious about answers and comments about my remarks in the Questions/Suggestions section, however I am not asking for additional experiments concerning them during the rebuttal.
review_title: null

Meta-review: metareview: The paper proposes an efficient implicit recursive octree auto-decoder (iROAD) to encode large scale 3D shape datasets in real time. Overall, the reviews are positive. Reviewers agree that the approach is new and well-motivated, and the experiments are convincing. 

Authors are encouraged to respond to reviewers' comments and questions. Authors should also clarify how this relates to robot learning.

Update:

Reviewers appreciate the detailed responses and additional experiments, and unanimously agree that the paper should be accepted; the work presents a solid contribution with impressive empirical results that will be beneficial to the robotics community as a strong solution for 3D vision. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)