Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles the task of surface reconstruction from images. They use the volume rendering based approach popularized by NeRF, and thus are also interested in maintaining high novel view synthesis performance. In particular they follow recent work (IDF, VolSDF) that embed an SDF into the volume rendering pipeline in order to get more accurate surfaces, and propose an approach for achieving high frequency detail. To do this, they incorporate the SDF in a new fashion, use implicit displacement fields [32] and use a coarse to fine strategy. 
strengths_and_weaknesses: Strengths
- Good explanation of previous ways that SDFs have been incorporated into volume rendering
- Good explanation for the proposed way to incorporate SDFs and the intuition behind it.
- Experimented on well established datasets/benchmarks and shows significant improvements on surface reconstruction while maintaining similar or better view synthesis quality
- Good ablation for the high frequency detail methods, e.g. basic encoding vs displacement field method, and the benefit of the coarse to fine approach.

Weaknesses
- No theoretical explanation of the benefits of the new SDF incorporation method compared to previous methods, except for simplicity of the derivation. You have done an excellent job motivating your approach mathematically, but haven't clearly explained what benefits that has over the other methods. You should explain in more detail why directly modelling the transparency function would be better.
- No ablations showing how the new way to incorporate SDFs differs in performance to the way VolSDF/NeuS does it.
questions: In section 3.2 the work describing the decomposition into a base and displacement field is not cited at all (should be [32]). Given that it is a major part of the approach and the use of that work is not cited anywhere except in related work, this is quite important to do.
limitations: Both limitations and potential negative societal impacts have been discussed in the conclusion.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors address the objective of improving the accuracy of 3D scene reconstruction based on neural rendering approaches. For this purpose, they first provide a theoretical discussion on how the signed distance function (SDF) can be embedded into volume rendering as well as the relationship between the SDF, the transparency function, the volume density and the weighting function. In addition, criteria for functions that are suitable to map signed distances to transparency and a respectively suitable class of functions is discussed, but the choices follow previous work (in particular, the integration of the SDF follows VolSDF, and the selection of a logistic sigmoid function with learnable scale parameter for Psi follows NeuS). This is used in a multi-scale fitting framework where the SDF is modeled as a combination of a base distance function and a (implicit) displacement function (similar to references [reference 15] and [reference 32]) along the normal of the base distance function, and positional encoding is used for each of these functions separately, where the frequency of the positional encoding is increased according to the coarse-to-fine strategy by Park et al. [reference 22]. Both functions are additionally represented by separate MLPs, and the used loss includes a radiance term and an additional Eikonol regularization of the SDF, where the norm of the gradient of the SDF is used to weight the scale parameter in a spatially varying manner.

Experiments on the DTU Dataset, NeRF Synthetic Dataset and BlendedMVS dataset with comparisons to NeuS and VolSDF show the potential of the proposed approach.
strengths_and_weaknesses: ## Originality/novelty and technical soundness:
pro:
+ The approach seems reasonable.
+ The major novelty seems combining these concepts in the NeRF framework and using a spatially varying scale parameter which, together with the combination of all the concepts, leads to accuracy improvements over the other approaches (NeuS, VolSDF) in the comparison. 

con:
- However, there is a larger overlap with existing work regarding the use of separate networks to represent a base SDF and displacement fields [references 15 and 32], the integration of the SDF into volume rendering [reference 30], using an unbiased density function using logistic sigmoids and learnable control parameter [reference 28]).



## Clarity and exposition:
pro:
+ The paper is well-structured and easy to follow. Arguments are clear.
+ Figures/Tables and the respective captions are informative.
+ The relevance of the topic is sufficiently motivated.
+ The supplemental provides further implementation details in more detail.

con:
- While text quality is mostly ok, there are typos, e.g. 'view-dependent' appears several times as 'view dependent', 'low/high frequency' should be written as 'low/high-frequency' if it is followed by a noun it refers to (e.g., 'components', 'content', 'surfaces', etc.), 'state of the art algorithm' should read 'state-of-the-art algorithm', 'coarse to fine optimization' should read 'coarse-to-fine optimization'.

- Some selections of reference lists seem arbitrary, e.g., the reference lists on NeRF improvements (in the introduction) and multi-view stereo reconstruction (in the related work section). It is not clear why only these references have been used. For NeRF improvements, it might be helpful to cluster references regarding the aspect the techniques are focused on, e.g. faster training/inference, less constrained conditions, video inputs, generalization capabilities, multi-scale representations, etc.. Regarding voxel-based 3D reconstruction approaches there are many more variants, e.g.:
- - Izadi et al., KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera, and its extensions such as:
- - Niessner et al., Real-time 3D reconstruction at scale using voxel hashing
- - Whelan et al., Kintinuous: Spatially extended kinectfusion
- - Dai et al., Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration
- - Klingensmith et al., Chisel: Real Time Large Scale 3D Reconstruction Onboard a Mobile Device using Spatially Hashed Signed Distance Fields
- - Stotko et al., Efficient 3D Reconstruction and Streaming for Group-Scale Multi-Client Live Telepresence

In addition, there are more variants for surface meshing such as:
- - Fuhrmann and Goesele, Floating Scale Surface Reconstruction
- - Calakli and Taubin, SSD: Smooth signed distance surface reconstruction
- - Berger et al., A survey of surface reconstruction from point clouds



## Evaluation:
pro:
+ The authors provide qualitative and quantitative results with comparisons to reasonable baselines (VolSDF and NeuS), where the proposed approach is demonstrated to lead to improvements in the reconstruction result.
+ An ablation study shows the effects of high-frequency positional encoding, coarse-to-fine optimization and the use of the implicit displacement function.
+ The supplemental provides further ablations on the coarse-to-fine parameter alpha_d and the number of frequency bands L as well as ablations for adaptive transparency.

con:
- The MipNeRF approach that focuses on scene representation at continuously-valued scale also improves the representation of fine-grained details. It is not clear why this approach has only been mentioned in the list in the introduction, but not in the discussion of related work or among the competing techniques used for evaluation.
- Training and inference times could be mentioned.
- An ablation study of the effect of the regularizers would be interesting as well.
- Additionally, the stability of the reconstructions of fine-grained structural details could be shown with respect to the number of views. Maybe the proposed approach gets the same quality of the reconstruction of NeRF with significant less input views.
- There is no detailed discussion of problematic cases or failure cases in the main paper. Limitations regarding inaccurate reconstructions of linear structures (i.e. ropes) mentioned in the supplemental should rather be mentioned in the main paper. In addition, e.g., for the cases where the proposed approach does not outperform its competitors in Table 1, respective visualizations could be provided to demonstrate the kind of errors and where these are larger than for the other methods. This would provide insights on further improvement potential.



## Reproducibility:
pro:
+ The information in the paper together with the attached code seems sufficient to re-implement the approach.

con:
- It is not entirely clarified, whether the code will also be made available afterwards.



## References:
con:
- See comment under 'Exposition'.
- For several of the references, only the arxiv reference is provided and not the final conference/journal reference. This should be revised accordingly.



## Post-rebuttal recommendation:
The paper addressed my major concerns in terms of ...
... the detailed discussion of the works of VolSDF [30], NeuS [28] and Yifan et al.'s approach [32] which helps to better see the embedding of the proposed approach in the context of related work,
... the improvements regarding the exposition (typos, also mentioning MipNeRF in the related work, mentioning the used Chamfer distance formula, etc.),
... the promised code release,
... the additional details regarding training/inference times,
... the shift of limitations into the main paper,
... the added visualization of local errors.
As a result, I increase my rating from borderline accept to weak accept.
questions: ## Exposition:
- As mentioned above under 'strengths/weaknesses', some selections of reference lists seem arbitrary. For NeRF improvements, it might be helpful to cluster references regarding the aspect the techniques are focused on, e.g. faster training/inference, less constrained conditions, video inputs, generalization capabilities, multi-scale representations, etc.. The lists of papers for 3D reconstruction and surface reconstruction also seem arbitrary in the current formulation. Referring to the respective works or respective surveys could solve this.

## Evaluation:
- I would recommend to also provide the equation for the Chamfer distance, as there are different definitions used in literature.
- Regarding Figures 1 and 3, showing the local errors on the surface with respect to the ground truth might additionally better highlight where deviations are larger.
- As mentioned above, the MipNeRF approach focuses on scene representation at continuously-valued scale which improves the representation of fine-grained details. Why has this approach has only been mentioned in the list in the introduction, but not in the discussion of related work or among the competing techniques used for evaluation.
- As mentioned above, training and inference times could be mentioned.
- As mentioned above, an ablation study of the effect of the regularizers would be interesting as well.
- As mentioned above, the stability of the reconstructions of fine-grained structural details could be shown with respect to the number of views. Maybe the proposed approach gets the same quality of the reconstruction of NeRF with significant less input views.
- As mentioned above, there is no detailed discussion of problematic cases or failure cases in the main paper. Limitations regarding inaccurate reconstructions of linear structures (i.e. ropes) mentioned in the supplemental should rather be mentioned in the main paper. In addition, e.g., for the cases where the proposed approach does not outperform its competitors in Table 1, respective visualizations could be provided to demonstrate the kind of errors and where these are larger than for the other methods. This would provide insights on further improvement potential.
- A discussion or comparison to the recent approaches Instant-NGP (Muller et al., Instant neural graphics primitives with a multiresolution hash encoding) and SVLF (Tremblay et al., RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis) would be interesting, however, it seems as if these works were not yet published at the NeuRIPS submission deadline.

## Reproducibility:
- Will code be release upon paper acceptance?
limitations: The authors have adequately addressed the potential negative societal impact of their work. Limitations in terms of the need for optimizing an additional implicit function or overfitting have only been shortly discussed in the last section on conclusions. As mentioned above, the discussion of limitations needs to be improved.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary:  
 This paper builds on the recent efforts in multi-view neural implicit surface reconstruction and improves in the reconstruction quality by introducing a different way of modelling SDFs, applying the implicit displacement fields to this task, and also proposing locally-adaptive scaling factors. They show decent improvements on multiple object-level dataset for both surface reconstruction and view synthesis.

strengths_and_weaknesses: *Strengths*
 - I like the idea of locally-adaptive scaling factor s, which makes a lot of sense and I think can be a general add-on for other methods.
 - The paper is well written overall, easy to follow.
 
 *Weaknesses*
- In 3.1, what is the motivation of modelling SDFs with transparency? I don't see the clear advantage of doing so compared to modelling with the density function as done in VolSDF, or weighting function in NeuS. Indeed, modelling with the transparency is indeed conceptually simper than NeuS, but fundamentally it is very similar to the combination of NeuS and VolSDF.
- I don't think the method has fully utilize the benefit of implicit displacement field. For example, intuitively speaking, using a base model to model the coarse shape and another network to model the details should speed up the optimization process a lot, since the coarse shape can be very quickly obtained, and learning displacements is an easier optimization task than learning fully SDFs. Any experiments show the runtime benefit?  
- According to both quantitative and qualitative results, even after adding 3 different contributions and course-to-fine strategies, the proposed approach is only marginally better. I am not fully convinced by the usefulness of your different contributions. Moreover, because of the use of an extra MLPs, you even need more computational resources.
 
 

questions: - I wonder if you simply change the way of modelling SDFs of NeuS to yours, how much benefit can you obtain (without the displacement field) for NeuS?
 
 
 - No enough acknowledgement to the IDF paper in 3.2. For example, $\mathbf{x}_b = \mathbf{x} - f_d(\mathbf{x}_b)\mathbf{n}_b$ (the key idea of implicit displacement field) mentioned already in IDF. 
 
 - Why not do the same as in the IDF paper where they use two SIRENs for the base model and displacement model but using different frequencies?
 
 
 - PSNR calculation. Do you calculate the PSNR on the training views or the held-out testing views?
 
 - Ablation study. Table 3 confuses me since it lacks of discussion of different componenets. Why Base+H is much worse than only Base? Why changing from only base to IDF does not gurantee better results (DTU experiment)? And it seems to me that most beneficial components are coming from the coarse-to-fine strategy and the locally-adaptive scales.
 

limitations: Limitations and societal impact are briefly discussed in the conclusion.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper present s new method for surface reconstruction using the recently popular neural volume rendering approach. The authors suggest improving existing methods, which often lack high-frequency details. To do so, the authors present various changes. First, they propose transforming a learned sign distance function into the transparency function, rather than the physical density or the weighting function as done in relevant baselines. Additionally, following previous works, the authors decompose the sign distance function into base and displacement functions, accompanied by a gradually increasing frequency learning strategy. Lastly, they propose improving certain regions by weighing based on the properties of the sign distance function. The paper presents reconstruction results compared to existing baselines at three different datasets.
strengths_and_weaknesses: Strengths:

1. The need for high-frequency details improvement is well motivated and understood.
2. I find some of the suggested improvements innovative and contributive to future works. More specifically, I like the idea of transforming the transparency and examining its derivate and relation to the weighting function, and the idea of the adaptive weighting strategy. I mostly appreciate the concept of weighting based on the eikonal equation satisfaction.
3. The related work section is concrete, well written, and covers the paper area for newcomers.

Weaknesses:

1. The contribution of each component is not straightforward. Although the authors conducted an ablation study, they have not presented qualitative results or discussed the potential implications of each component for the overall improvement. 
2. Following but separately from the above, I’m concerned with the need for the base and displacement decomposition. Because this decomposition is not unique, and since high frequencies encoding adds noise, more possible wrongly reconstructions exist that overfit the rendering. I believe that a visualization of the decomposition would emphasize its benefits. Another alternative is to show an ablation using the decomposition (“IDF”) with frequencies as the baseline NeuS.
3. There is a clear difference between the presented VolSDF baseline results shown in this paper and other resources. I direct the authors to the original VolSDF paper for results of the DTU and BlendedMVS datasets (figures 5 and 10), and to the recent work RefNeRF (https://dorverbin.github.io/refnerf/) for the synthetic NeRF results (see video). I would appreciate it if the authors could explain this difference.
4. There are some missing methodology implementation details. For example, it is unclear what are the discretization formulas for the transparency and the volume rendering integral approximations, given a set of samples along the ray. Also, I believe the authors should state how multiple surface intersections are handled (line 173). Additionally, it is not certain how the displacement constraint is imposed, nor its relation to equation 13.
5. The superiority of the presented methods is mostly given quantitively but not qualitatively. It is also apparent in the PSNR results, surpassing all other methods, but the paper does not show rendering comparisons. Similar to the ablation study, I believe that additional qualitative results (even in the supplementary) would help the paper statements.
questions: Except for the mentioned concerns raised in the previous section, I would appreciate it if the authors would answer the following questions:

1. Could the authors explain their choice of the logistic sigmoid function for Psi?
2. (Relates to the above) Have the authors considered using the Gaussian or Laplace distributions CDF as the transformation function Psi? Why would one choice will be better than the other?
3. I suspect equation 9 has a typo, and it should be f_d(x_b)? If not, please explain the transition from equation 8.
4. The intention behind equation 15 is unclear. Could the authors please explain?
limitations: The authors addressed the method limitations. However, I encourage the authors to move the limitations section to the main paper in the revised version.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper improves the detail representation of geometric details for implicit surface reconstruction using multiview images. 
It contains 3 contributions: 

1. an analysis of the SDF and values in the volumetric rendering, which leads to formally deriving the suitable transformation function to use SDF for volumetric rendering.
2. an improved implicit displacement field for better detail reconstruction, which contains several changes to the original implicit displacement field to robustly work with 2D supervision
3. a strategy to improve the optimization by adapting the mapping function in (1) spatially.
strengths_and_weaknesses: # Strengths:
This paper is very well written and shows impressive improvements compared to the state-of-the-art for reconstructing geometric details.
The three steps are well motivated and the ablation shows that they all contribute to the improvement. The entirety of the paper provides a good basis for NeRF-based multi-view 3D reconstruction.

# Weakness:
I think while this paper was very clear in stating the 3 contributions (section 3.1-section 3.3), it's not very clear what exactly is novel (proposed by the paper) and what is from prior work. This is particularly true in section 3.2. L175-201 seem to be purely prior work [32], which should be clarified and condensed. More effort should be put into explaining and motivating the differences.

There is a small section in the supplemental (B.2) explaining hierarchical sampling based on $cs$. But I don't see any result demonstrating the effectiveness of this sampling strategy. 
questions: 1. Regarding the displacement constrain $\Psi'_s$, in the supplemental, the authors mentioned using some progressive strategy to gradually increase the constraint. If I understand correctly, this means gradually requiring the displacement to be spatially sparse and more focused on the surface area. The motivation is not very well explained. Shouldn't I want the opposite - by requiring a tighter spatial bound around the base surface at the beginning of the training, the base SDF can quickly converge, so that when optimizing the displacement $f_d$, $f_b$ remains stable and hopefully the composed function $f$ converges faster. Why is this not the case?
2. Can I understand the adaptive transparency function described in (15) as a form of hard example mining? What do you think $\|\nabla f\|$ would be a good indicator? BTW, should it be $\\|\nabla f\\|$ instead of $\|\nabla f\|$?
3. Table 3: Is X+C2F adding C2F to X+H?  FULL is adding adaptive transparency to IDF+H+C2F? If this is not the case, then the IDF+adaptive s is missing

limitations: Yes.
review_title: null

Meta-review: metareview: This paper presents a new way to build centered weights for volume rendering, utilize displacement maps, adaptive scale, as well as other techniques to provide better high frequency details in neural SDF representations.   
The reviewers also acknowledged the rebuttal and the revision and the authors addressing their main concerns.  
confidence: Certain
Decision: Accept