Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work tackles on reconstructing the implicit surface from images. The main contribution of the work is to propose a hierarchical volume encoding for neural implicit surface reconstruction, which can be easily be plug-and-played on top of other existing works. Although the hierarchical volume encoding scheme has been widely used in computer vision community, experiments on widely used datasets, such as DTU, BlendedMVS show that the work clearly enhances other methods.
strength_and_weaknesses: **Strength**

- Performance

The work shows impressive performance both quantitatively and qualitatively. I was impressed by figure1 & 5 reconstructing high frequency details from images.

- Simple and efficient scheme

Using hierarchical features for implicit reconstruction makes sense and is straightforward. I also enjoyed a simple idea of obtaining sparse high resolution volume by using multi-stage optimization and dilating the low resolution reconstructed implicit function. 


**Weakness** 

- Using hierachical feature volume encoding has been used in the field of computer vision for a long time with various tasks, including surface reconstuction, such as [1]. 

This idea is not brand new, however, I did enjoy the performance increase with respect to the simplicity of the method.


- The work can be strengthened by adding more experiments for NeuralWarp + HIVE. 

The authors have reported all scores for both Neus and NeuralWarp with and without HIVE for table 1 & 2. I think adding restuls for NeuralWarp + HIVE in Figure 6 and ablation study can further strengthen the paper, because this paper shows enhancement of the results based on previous works. Because this work is an add on to other methods, I believe these results are necessary to show the performance increase. 

- The work can be strengthened by adding more quantitative metric than Chamfer distance. 

Table 1, 2 shows the Chamfer distance compared to other methods. While the Chamfer is a good metric to compare the similarity of to points clouds, there has been many questions [2] whether the Chamfer distance could capture the high frequency details or overall ditributional similarity between two point clouds. I would suggest adding normal consistency as in [3] or some other metrics that show that the method can capture the high frequency surfaces.

[1] Chibane et al. Implicit functions in feature space for 3d shape reconstruction and completion. CVPR, 2020.

[2] Wu et al. Density-aware Chamfer Distance as a Comprehensive Metric for Point Cloud Completion. NeurIPS, 2021

[3] Mescheder et al. Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR 2019

clarity,_quality,_novelty_and_reproducibility: - Clarity & Quality

The manuscript is overall easy to understand with nice figures. Although some parts of manuscript, such as the third contribution in page 2 should be modified to more concrete and specific sentence. Also, one could add captions on Table 1 & 2 explicitly stating the table is about Chamfer distance. 

- Novelty

As stated in the weakness section, although using hierarchical feature volume encoding is not new, I enjoy the simplicity.

- Reproducibility 

The authors promised to release the code.  

summary_of_the_review: Although the proposed hierarchical volume encoding itself is not novel in the field of surface reconstruction, I believe that the pros of having better performance while being simple is very practical and will be beneficial to community if the additional figures and experiments are done as stated in the weakness section. I would like to hear what other reviewers before making the final decision.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an improved design of spatial encoding for neural surface reconstruction methods, e.g. NeuS and VolSDF. Instead of using frequency positional encoding as in NeRF, the authors propose to explicitly encode the spatial information with multi-scale voxels storing feature vectors. In particular, the encoding consists of dense voxel grids up to the resolution of 256^3, whereas subsequent higher resolutions (512^3 and 1024^3) are derived from sparsified results from the previous levels. Experiments are focused on the DTU benchmark with additional visualizations on BlendedMVS and EPBL, showing improvements over frequency encoding.
strength_and_weaknesses: Strengths:
+ This paper presents a simple and effective improvement over frequency encoding used in neural surface reconstruction methods like NeuS and VolSDF. Using explicit (hybrid) representations have recently shown success to accelerate convergence and improve view synthesis results [A,B,C]; this paper applies the same concept to surface reconstruction.
+ The proposed HIVE adds significant improvement over baseline models (NeuS and VolSDF) in reconstruction quality.
+ The method description is easy to follow and understand.

Weaknesses:
- The authors have missed very important references that improves NeRF with explicit (hybrid) encodings, listed as [A,B,C]. In particular, Plenoxels [A] have already proposed explicit encoding with a voxel pruning strategy. In some sense, this paper proposed almost exactly the same concept, but only applied to surface reconstruction. In addition, the authors have ignored a large body of prior works on using octrees to represent 3D scenes with hierarchical voxel grids. A survey can be found in [D].
- I think it would be good to have discussions on the drawbacks of the proposed HIVE. One limitation I can see is that HIVE is currently limited to the resolution of 1024^3. Although the higher resolutions can be initialized from sparsified voxels (with an additional dilation step), the amortized memory growth will be N^2 (proportional to the surface area) and eventually it will still be memory bounded. So it doesn't seem like HIVE is much more scalable.
- Another drawback of HIVE is that granular details are not recoverable if certain voxels were pruned at resolution 256^3. So there is a heavy assumption that the coarse shapes should be fully captured at 256^3. This is not a limitation for more naïve frequency encoding from NeRF.
- The improvement on BlendedMVS scenes seems significant over NeuS and somewhat better than NeuralWarp, but the resolution of the recovered surfaces do not seem to be as high resolution as expected (1024^3?). It would be great if the authors could clarify.
- The NeuS+hash results seem questionable. From the results of Muller et al., Instant NGP has even more representation capability than the original NeRF, and artifacts shown in Fig 6(c) are not to be expected. It would be good if the authors could clarify; otherwise I would not think this is a fair and faithful comparison.
- Minor: Fig 6 is hard to understand without sufficient descriptions in the captions. 
- Minor: I don't think equations 7 and 8 are necessary; they are just binary dilation and index lookup operations.

[A] Yu et al. "Plenoxels: Radiance Fields without Neural Networks." CVPR 2022
[B] Sun et al. "Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction." CVPR 2022
[C] Chen et al. "TensoRF: Tensorial Radiance Fields." ECCV 2022
[D] Knoll. "A survey of octree volume rendering methods." 2006
clarity,_quality,_novelty_and_reproducibility: The paper is written and described clearly, and it is easy to follow. The proposed concept is simple so it should be easily reproducible. However, I have doubts on the originality and novelty of this work. It is very similar to Plenoxels but only applied to the application of surface reconstruction. There is also not much additional insights on how the proposed HIVE differ from prior works like Plenoxels.
summary_of_the_review: I think this is a nice paper that presents surface reconstruction results that have shown observable improvements, but I have major concerns on the novelty of this work. While I understand novelty is not the sole evaluation criterion, I think there are insufficient discussions on (a) how this work differ from the others, (b) deeper insights on the technical parts of the methods, (c) limitations of the proposed method. I am thus leaning towards a reject at this point.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper describes a series of experiments in which a hierarchical feature volume is used to encode a latent representation of a scene. The volume defines a continuous feature field through linear interpolation of the features associated with discrete voxel locations. The resultant feature at any point in space can then be passed as input to a neural network or networks to render images of the encoded scene. A variety of rendering methods can make use of this same scene representation, such as NeRF, NeuS, etc. The paper also describes a method of progressively optimizing the feature volumes in a coarse-to-fine order, and a sparsification strategy for the higher-resolution volumes to avoid using too much memory.
strength_and_weaknesses: The paper identified a valid issue with many neural scene representations, i.e. the inductive bias towards lower-frequency radiance, density, and/or signed distance fields which often results in overly-smooth renderings. The paper also proposes a sensible strategy to mitigate this issue, and show that their technique significantly reduces reconstruction errors when applied to a variety of scene rendering models. The ablation studies provide interesting insights into the value added by the hierarchical volumes and the influence of various hyperparameters.

The key weakness of the paper is that it lacks technical novelty. However, it is not clear from reading the manuscript this is the case. There are important gaps in the related work cited, and the related work that is cited is given a fairly cursory treatment. Instant Neural Graphics Primitives introduces a very similar approach in which a scene is volumetrically encoded, and it can be used interchangeably with the proposed hierarchical volumes. However, Instant-NGP is only mentioned at the very end of the manuscript. An experiment is performed to compare the performance of Instant-NGP vs. HIVE and the results indicate that HIVE leads to superior performance. However, the experiment is not described in detail and as such it is hard to interpret the results. The manuscript mentions an advantage of HIVE over Instant-NGP, namely that it does not experience hash collisions which can impair performance, but does not mention the corresponding advantage of Instant-NGP over HIVE which is that the memory cost can be kept to O(1) in the scene volume whereas HIVE is O(N) in the scene volume (with some scalar multiplier resulting from sparsification). Was the amount of memory used to represent the scenes balanced between the two approaches for the experiment presented in Table 1? The manuscript also provides no runtime analysis of the proposed approach, so we do not know how it compares to Instant-NGP which is known to enable fast inference of scene representations.

The manuscript also completely fails to mention a number of highly related works on hierarchical sparse representation of scenes for neural reconstruction, such as Neural Geometric Level of Detail (Takikawa et al. 2021), PlenOctrees (Yu et al. 2021), and ACORN (Martel et al. 2021). NGLoD in particular introduces an extremely similar scene representation. Unfortunately the two approaches are evaluated on different datasets and thus we do not know which performs better or why.

clarity,_quality,_novelty_and_reproducibility: The submitted manuscript is overall of good quality, but lacks clarity on a few specific points. One is the treatment of Instant NGP as described above, which leaves out a lot of detail such that it would be impossible to reproduce the experiment. A few statements that are critical of prior approaches to neural reconstruction are not clear or concrete on what the issues being pinpointed are. For example, in section 3.3 the manuscript states that MLP-based renderings have “problems in surface reconstruction, because the simple MLP architecture makes learning and optimization inefficient,” but it is not clear what is meant by a “simple” MLP, what is the distinction between “learning” and “optimization,” or what kind of efficiency is lacking. Similarly, in the related work the manuscript states that when using NeRF to represent a scene, “the surface extracted from the implicit network usually has some defect,” but it is not clear which defects are meant or which surface extraction method is being reference, as NeRF does not represent surfaces and thus does not provide an unambiguous method of surface extraction. 
summary_of_the_review: The manuscript provides new experiments showing that the addition of a sparse hierarchical volume encoding to a neural scene renderer can improve performance of a variety of existing baseline methods. However, the technical novelty over other previously published sparse hierarchical scene representations is very small, and the manuscript not only fails to make a case for the use of the proposed technique over the highly similar existing approaches.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a multi-scale learnable feature volume encoding for neural implicit surface reconstruction. 
The main idea of the paper is to train a neural implicit surface function from coarse to fine in 3 stages. In each stage, a learnable volume feature encoding is concatenated to the input of the implicit function. The authors also proposed a volume sparsify procedure to discard the empty voxel for memory efficiency. 
The proposed strategy can be widely adopted in different coordinate-based neural implicit surface papers.

The underlying technique of this paper is very simple and easy to understand, yet it significantly improves the surface reconstruction quality of the previous state-of-the-art methods in various public available datasets.
strength_and_weaknesses: Strength

- The proposed strategy can be widely adopted in different coordinate-based neural implicit surface papers.

- The underlying technique of this paper is very simple and easy to understand, yet it significantly improves the surface reconstruction quality of the previous state-of-the-art methods in various public available datasets.



Weaknesses

Please explain more on the claims in the paper : 
- In page 4, Sec 3.3, "This works well for novel view rendering but has problems in surface reconstruction, because the simple MLP architecture makes learning and optimization inefficient".
Why a simple MLP makes the learning inefficient? Why is it inefficient in geometry learning, but good at RGB learning? 

- In page 5, on top: "This feature volume can naturally encode the knowledge about the 3D space of the object".
Why is a randomly initialized volume encoding can "naturally" encode the knowledge about an object? How is this volume encoding looks like when fully trained? More visualization would be helpful here in revealing the use of such encoding.


Additional experiments suggestions:

I am curious to see whether the proposed method can improve the RGB reconstruction or not? It will be good to showcase some novel view synthesis evaluations. And it would be good to explain more on why this encoding can or cannot help RGB reconstruction.

Typos: 
In page 4, first line, "with which another implicit", 
clarity,_quality,_novelty_and_reproducibility: Clarity: some of the claims in the paper is not properly addressed of proofed by experiments.

Reproducibility: The idea of the paper is clearly state and should be easy for readers to reproduce.
summary_of_the_review: 
I found the idea of this paper very simple yet very effective.
The proposed method can also be easily applied to other works. The paper achieves state-of-the-art performance on surface reconstruction. 
But some claims in the paper is not properly addressed and nor proofed via experiments. Please see the weakness. 

I think the paper at least provides good empirical results on utilizing volume encoding in improving SDF surface reconstruction. On the other hand, it would be good if the authors could also provide more insights on how such encoding assists the geometry reconstruction, and whether such encoding can be applied to broader use (such as novel synthesis).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was reviewed by four experts in the field. The reviewers raised many concerns regarding the paper: 1) The technical novelty over other previously published sparse hierarchical scene representations (e.g. Plenoxels, Instant NGP) is marginal. 2) Several experiments (e.g. NeuS+Hash) should be improved to be considered fair comparison. Considering the reviewers' concerns, we regret that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
justification_for_why_not_higher_score: Most reviewers have doubts about the originality and novelty of this work. It is very similar to Plenoxels but only applied to the application of surface reconstruction.
Decision: Reject