Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method that combines volume rendering with surface rendering techniques for multi-view reconstruction. The method represents the geometry of objects using a neural implicit representation that outputs the SDF at query locations. In order to enable volume rendering, the output SDF is mapped to a density function centered around the surface. Furthermore, they also encode the appearance of the object into the neural implicit representation that is needed for multi-view reconstruction. The main contribution is a novel weight function that is centered (unbiased) around the surface while being occlusion-aware. Therefore, the paper resolves major limitations of volume rendering (to few constraints on surface) and surface rendering (very sparse gradients). The paper extensively discusses different formulations of the weight function (properties, weaknesses, strengths). The method is evaluated on the DTU as well as the BlendedMVS dataset and compared to existing state-of-the-art methods like NeRF (volume rendering) and IDR (surface rendering). Moreover, they evaluate some aspects of the method in an ablation study.

main_review: Here, I order the strengths and weaknesses according to their importance and contribution to the final score as well as additional comments that should be resolved in the final version of the paper.

#Strengths:

1. The paper proposes a simple yet elegant formulation for multi-view surface reconstruction using neural implicit representations. The results are impressive and show that it is very beneficial to combine volume rendering and surface rendering to solve the well known problems of noisy surfaces (NeRF) and requiring a mask for supervision (IDR). This is particularly shown in Figure 6.

2. The paper extensively discusses the strengths and limitations of existing weight functions used in volumetric rendering. Especially, the illustrations in Figure 2 and Figure 3 are very useful to understand the problem and the proposed solution. 

3. The introduction as well as the related work very well motivate the paper with respect to existing surface and volume rendering methods. This helps the reader to understand the importance of the problem and the proposed solution.

4. By providing an intuition behind different weight functions, the different solutions and their strengths/weaknesses are very well motivated. 

#Weaknesses:

1. Clarity:
The paragraph, where the solution is explained (L185 - L193) is very unclear. It is very hard to understand how the solution is derived from Equation 5 and Equation 6. This should be definitely made clearer for the final version of the paper as it is crucial for the reader to understand the derivation of the main contribution. I would like to see how you derive and combine equation 7 from equation 5 and 6. This could then be also added to the supplementary material for the final paper with an intuition in the main paper.

2. Results/Ablations 

- Firstly, as you also encode the appearance and especially NeRF is primarily designed for novel view synthesis, it would be interesting to see how this method performs on novel view synthesis. For doing this, I suggest to visualize novel views from the scenes and also run a quantitative evaluation using PSNR and SSIM.

- Secondly, as you already compare to UNISURF, it would also be interesting to see the experiments on the Indoor Scene Dataset as shown in UNISURF (Figure 8, p. 8).

- Thirdly, one missing experiment is the ablation of all loss terms. I would be particularly interested in an ablation of the Eikonal term to see whether this is also the reason for the better performance compared to UNISURF. 


3. Related Work:
There are some relevant works missing on multi-view reconstruction. Particularly, the two works below that are one of the first works using machine learning for multi-view reconstruction.

- Kar et al., Learning a multi-view stereo machine, NeurIPS 2017
- Choy et al., 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction, ECCV 2016

In the section about Neural Implicit Representations, the following work is missing:

- Sitzmann et al., DeepVoxels: Learning Persistent 3D Feature Embeddings, CVPR 2019

Moreover, I do only partially agree with the discussion of the differences between this work and the concurrent work UNISURF. Although it is correct that UNSIRF represents the geometry as occupancy values, you can also naturally extract the surface at the decision boundary (0.5), which is just a different level set. I doubt that this difference is the reason for the performance improvement over UNISURF. A more in depth discussion would be helpful for the community to understand what improves performance and what does not. 

4. Lacking justification of certain design choices:

- To me, the choice of the logistic density distribution is not motivated enough. I assume the main benefit is the easy integration to the sigmoid function that is used in the final solution (Equation 7). I would like to see a brief comment on that. 


5. Clarity/Flow of Reading
The clarity of the writing and flow of reading is limited by very long sentences (e.g. Abstract L11 - L14), obscure formulations (e.g. L87 'which makes the gradient only be backpropagated to a local region near the intersection', L118 'neural networks of Multi-layer Perceptron (MLP), L), and wrong/inconsistent prepositions (e.g. requirements of weight function (L147-L148) vs. requirements on weight function (L149)). This significantly breaks the flow while reading the paper and makes it harder to understand everything.


#Additional Comments:

- The writing of the paper should be reviewed and revised in general (uncountable vs. countable nouns, third person s, prepositions, missing or too many a/the (e.g. L56, L86), etc.). This would make the reading much easier and redirects the mental capacity of the reader to the content. 

- A few formulations could be improved for better clarity.
	- L31 - L32: would get optimization stuck --> other formulation 
	- L100: sample region --> sampling region 
	- L108: zero-set --> zero level set
	- L111: train the network of SDF --> other formulation
	- L120: method to training the SDF network --> method to train the SDF network
	- L134: To learn the parameters of MLPS of the SDF and the color field --> To learn the parameters of the SDF and color networks (or something similar)
	- L157 - 158: near the view point --> closer to the view point
	- Theorem 1: depth --> depth values
	- Theorem 1: going from the outside of the surface --> going from the outside of the object (there is no inside outside of a surface)

- It would be very good to visualize the meshes (especially in Figure 7) with some sort of error color coding (similar to Tatarchenko et al. 2019, Figure 12). This would guide the reader to better see where the different configuration fail/work.

- Reference 5 and 6 are the same. They should be merged.

- The failure case in the appendix is not really informative (Fig. 6). Some more failure cases caused by lack of texture should be shown together with the input views (to see the lack of texture). Guiding the reader more (directly in the figure) would help as well. 




#Questions

- What is the reason for using different sampling points for the color and alpha values? (Appendix L57-59)
- Why do the results for some scenes get worse when using the mask supervision? What is the intuition behind that? (Table 1)

limitations_and_societal_impact: They are discussed sufficiently.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents a novel multi-view reconstruction model that couples an implicit SDF representation (as in IDR [31]) with an unbiased volumetric rendering function (as in NeRF [20]). The volumetric rendering function is deliberately "de-biased" by redefining the opacity values $\alpha_i$ such that the weight is maximized exactly on the surface defined by the zero level set of the SDF.

The resulting model enjoys the benefits of both surface-based representation for high-fidelity surface reconstruction, and dense volumetric rendering which provides effective gradients that are not restricted locally to the surface and facilitate surface details. The reconstructed surfaces are much better compared to IDR and NeRF. This result can potentially also be very useful for further downstream applications that require smooth effective gradient signals.
main_review: ## Strengths
### S1 - method
The idea of combining volumetric rendering with SDF is actually very neat, which is better than the analytical gradient used in DVR and IDR that is only defined on surface and hence very local. This is very well implemented by carefully designing the alpha values to observe the surface properties of the SDF, driven by mathematical proofs. The resulting model is clean and easily interpretable.

### S2 - results
The reconstructed surfaces are clearly better than existing methods, eg IDR, NeRF, UNISURF, with smooth surfaces (unlike noisy shapes extracted from NeRF) and fine details, handling thin structures well (much better than IDR). The model can also perform reconstruction without object masks without much loss in quality, whereas DVR and IDR require mask supervision.

Thorough ablation and analysis results, including visualizations of the training progression, are presented and provide a lot of insights to the model.

### S3 - writing
The paper is very well written and highly polished. I really enjoyed reading it. The motivations are clearly explained and illustrated with examples and great figures. Technical descriptions are followed by motivation/intuition and careful proofs, and very well structured such that the readers could easily follow through.


## Weaknesses
### W1 - image synthesis
There are no comparison on the image synthesis quality. It seems the rendered image appear less visually appealing compared to NeRF, which however is a trade-off for obtaining explicit surface.

### W2 - (minor comment) more applications
Although the paper is already very complete in itself, it mostly follows the experimental setup of previous method (which is good). There are also many other interesting applications now that the model can obtain smooth, non-local gradients with volumetric rendering while maintaining a surface representation. For example, does that helps with the speed of convergence? Have the authors tried optimizing camera poses from coarse estimates as IDR does, and would it tolerate more noise? Does it help with the cases with fewer views?

## Typos
- Line 176 & 183: $\bar{\phi}$ should be $\phi$
- Line 152: remove "{"
- Line 162: "a unbiased" -> "an unbiased"


---
## Post-Rebuttal
I appreciate the authors' efforts in the rebuttal. This is a solid submission with a novel method and great results. I will keep my original rating.
limitations_and_societal_impact: A brief example of slightly inaccurate reconstructions is provided in the supplementary material. Similar imperfections are also visible in other examples, such as in purely specular or saturated regions. But I wonder whether there are more extreme major cases.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents a method for implicit 3D surface reconstruction from posed 2D images, where the 3D surface is represented with an SDF. The authors advocate a new way of using neural volume rendering methods for surface reconstruction, where the density field is induced by the optimized SDF instead of direct MLP outputs (as in e.g. NeRF). In addition, the authors described two key properties that should be satisfied for volume rendering with SDF representations and proposed a novel solution. Experimental results on the DTU multiview dataset shows its advantage over state-of-the-art baseline methods.
main_review: Strengths:
+ The paper is very well-written, with a clear problem statement and a thorough overview of related literature.
+ To take advantage of the nice optimization properties in neural volume rendering methods e.g. NeRF, the authors pointed out two important properties which should be satisfied for extracting surfaces with SDF as the representation: unbiasedness and occlusion-awareness. The authors suggested two simple solutions satisfying only each of the property, and then propose a nice solution to blend the two together. This makes a complicated concept easy to understand. The authors also provided a nice theoretical proof on how the proposed solution satisfy both properties.
+ The proposed method NeuS is compared with extensive recent baseline methods and significantly outperforms both qualitatively and quantitatively. Ablation studies and failure cases are also provided, which helps understand the contribution of each component.

Weaknesses:
- It was mentioned that a proof on how Eq. 6 is not unbiased would be provided, but only an empirical example was provided in the supplementary materials. It is still unclear to me how the construction of Eq. 6 is not unbiased.
- Although the results are superior, it would be helpful to provide a comparison on the training/inference speed. Since NeuS is still a volume rendering method with hierarchical sampling as adopted in NeRF, I would expect it to be almost as slow as NeRF to render. How slow would NeuS be compared to IDR?
- It would be good to also evaluate how well the recovered 3D representation actually satisfy an SDF, as SDF recovery is also one main goal of the paper. This would be important since efficient rendering (e.g. with sphere tracing) and relighting could be applied on true SDF shapes. The current evaluation seem to focus on 3D shape reconstruction in terms of surface accuracy (measured by Chamfer distances). In Fig. 4 of the supplementary material, the exterior region does not seem to correctly satisfy an SDF. 

I think it would also be interesting to see whether Neus would be able to reconstruct surfaces on forward-facing scenes, such as in the LLFF dataset originally considered in NeRF. Does the reparametrization of coordinates using NDC affect the solution?

Other minor problems:
- It is unclear what $s$ in L123 -- is it a hyperparameter that should be tuned?
- $\delta_i$ is undefined in Eq. 6. (I'm guessing it corresponds to $t_{i+1}-t_i$?)
- L183: the equation seems off, as $\bar{\phi} = \phi_s \circ f$ (Eq. 2).
- L192: the differential is missing.
limitations_and_societal_impact: Adequately addressed
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a neural surface reconstruction method which they call called NeuS. The novelty of this method is derived from their use of signed distance functions to represent geometry, and a proposed volume rendering method for accumulating information along rays. With these changes they are better able to estimate depth in more complicated surface geometry, demonstrate clearly superior shape predictions, and achieve state of the art quantitative results on the DTU dataset. 
main_review: This paper is quite strong. It identified an issue with NERF style volumetric rendering method, and solved  it by representing geometry with an SDF. The proposed volume rendering method which allows for this change is well motivated and explained. Both the qualitative and quantitative results are convincing and compelling. 

One point on concern is the comparison to other methods. I am not an expert in this field but it seems there has been quite an explosion of nerf-like papers since its release, and from a quick lit review of them it seems like NERF would no longer be considered state of the art in novel view synthesis, as referenced in the baselines section. NERF was released only last year and so there is leeway here for concurrent, or close to concurrent research, but I would perhaps revise this claim. If you wanted to raise my score higher it might be nice to compare directly to papers like "Neural Sparse Voxel Fields" which I see you reference, or other more recent papers which seem to tackle the same shortcomings in the original paper. 

other comments: 
- lines 67 to 70 are true of some methods but not all. This overly general statement is misleading. 
- There needs to be some editing for grammar, especially in the intro section. 

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The paper presented a novel idea of combining volumetric rendering with 3D surface representation (SDF) and achieved SOTA results. All the reviewers liked the paper and gave very constructive comments. Please include the results on novel view synthesis, evaluation of SDF quality, details of training  & inference time in the final version. Also, please carefully proof read the paper.
Decision: Accept (Spotlight)