Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The paper proposed a method to recover both geometry and appearance of objects. They 
achieve this by combing both SDF representations and volume rendering. Given a set of 
multi-view images, they use a network to predict the SDF and radiance filed of an arbitrary point, which 
is mapped to the volume density using a bell-shaped analytic function. Then they perform 
volume rendering to predict the color for the pixel. They minimize the color difference as 
well as an Eikonal loss to train the networks. They also propose a sampling scheme to  adaptively sample points along the ray based on the derived theoretical bound. The results 
show that the proposed method can generate both accurate 3D mesh and faithful view-dependent 
appearance.

main_review: 
### Strengths 
1. Compared to previous volume-based methods, the proposed method can generate better mesh for the 
captures scenes and produces comparable appearance.  

2. Compared to previous SDF-based methods, the proposed method can accurate reconstruct the geometry 
with the need for foreground masks, which makes the acquisition setup much simpler.    

3. The idea of mapping SDF to volume density is interesting and novel.

4. The paper develops a novel method for adaptively sampling points along the ray, and it gives 
a theoretical proof for the sampling scheme. Such a sampling scheme get rids of using coarse-fine 
networks as in NeRF, and also leads to better accuracy. 

### Weakness
1. In the sampling scheme, it's not clear what the upsampling means exactly. It seems to me 
that the method is gradually adding new samples to the current sampling points. My concern is 
that whether such a scheme will cause memory and efficiency problem when there are many sampling 
points considering that the network needs to be evaluated at each point. How much time does 
the sampling take? 

2. As far as I understanding, after the sampling iterations finish, the sampled points  
$\mathcal{T}$ and $\beta_{+}$ satisfying the desired bound. However, in Line 11, the algorithm 
generates a new set of samples. Do this new set of samples also meet the bound? If not, 
how does this affect the performance? Also when there are more than 64 intervals, how these 
$64$ samples are generated? Does it mean some of the intervals will have zero samples?

3. How does the method work for scenes with a lot of self-occlusions and thin structures? 
Most of the objects in the scene have relatively simple convex geometry.  It is desirable to show 
results on the NeRF dataset such as the Lego, the Ship and the Plant scene. 

4. The paper only shows object-level reconstructions. Since the method does not need image masks,  
I would expect the method to also work on complex scenes.  How does the method work on large-scale scenes as those shows in the NeRF paper? 

5. In the results in Figure A4 (supp), it seems that the recovered appearance tends to lose 
details and is a bit blurry, as shown in the Bull and the Dog scene. What is the reason
for it?

6. The proposed method can generate accurate meshes from the SDF. One question is that whether it's possible to direct use the colors of points on the isosurface to represent the appearance of the objects? Or volume rendering is still needed to predict the colors?  

7. Another limitation of the highlighted is that the texture/normal ambiguity. The method tends 
to treat the texture changes as normal changes in the reconstructions, such as 
the floor in the second scene of Figure 5.  

Overall, I appreciate the idea of the paper, and the results shown in the paper are convincing.  
I expect the authors to answer the questions above and add further clarifications to the paper. 


limitations_and_societal_impact: Please see the main review.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: NeRF and follow-up papers have led to impressive improvements in view synthesis when trained from multiple views of a single scene. This is achieved through volumetric rendering of an opacity field and a decoupled reflectance network. A limitation of NeRF is that the opacity field might not be faithful to the true geometry, as long as the rendered RGB images reproduce the correct appearance.

The proposed work builds on ideas developed in the NeRF literature and adapts them to 3D surface reconstruction. Contributions include deriving opacity from a truncated signed distance function, which explicity models the inside-outside topology of the surface. The validity of the signed distance function is enforced though an Eikonal loss term. The authors present a novel sampling technique for volumetric rendering, based on an approximation of the current opacity with bounded error. 

main_review: I consider this paper a strong submission that clearly identifies the limited applicability of NeRF-like goemetry representations to surface reconstruction. The authors present an intuitive geometry representation based on a truncated signed distance function, and an opacity field that is derived from the distance using a Laplace distribution. The width of the distribution is a learnable global parameter, which appears to allow for a gradual optimization from a relatively transparent initial geometry, to a well-defined, fully-opaque surface upon convergence.

The paper includes a thorough theoretical discussion of an adaptive sampling technique along the viewing ray, as well as a proof of error bounds on the opacity estimate that is used for sampling. The sampling technique is verified empirically in the supplemental material.

The presented experiments are comprehensive, comparing the accuracy of the reconstructed geometry to a state-of-the-art traditional technique (COLMAP), a recent learned technique (IDR), as well as a surface extracted from NeRF's opacity field. Both qualitatively and quantitatively, the presented 3D reconstructions compare very favourably to COLMAP and NeRF. It is exciting that the technique does not require silhouette masks as input, which promises to significantly increase the technique's practical viability.

The paper also includes an experiment that demonstrates improved disentanglement of geometry and reflectance compared to NeRF. It is surprising to this reviewer that the radiance network yields reliable predictions when sampled for positions near the surface of the new geometry, even though at training time the samples are placed near the original surface (guided by adaptive sampling). Could the authors comment why this apparent lack of supervision is not an issue?

### Limitations

I would expect a surface-based representation to show limitations for reconstruction problems that involve scenes without clearly defined surfaces, for example smoke or flames. The presented results do not include results on highly detailed geometry, such as hair or plants, which would require a level of detail that might exceed the resolution of the SDF, but could be represented with "soft visibility" for parts of the scene. Could the authors elaborate on the design space between representations with unconstrained opacity, and surface-based representation?

In direct comparison to baseline NeRF (see video), novel views appear to have somewhat smoother geometry and duller specular highlights. The paper would benefit from an analysis of implicit smoothness priors in the current implementation (Eikonal loss? L1 regularization in opacity transform?)

### Further comments

 - The relative impact of the Eikonal loss term should be verified with an ablation study.

 - The sampling ablation from the supplemental would be useful addition to the main paper.



limitations_and_societal_impact: The potential for negative societal impact of this work is adequately discussed. See Main Review for a discussion of other limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a neural approach based on NeRF, for the purpose of learning a 3D representation of a shape from captured images. Instead of allowing volume density to be freely allocated along rays, as in NeRF, this paper proposes concentrating volume density around a single surface. To do this, the method represents the shape as a neural SDF, and defines a volumetric density around the zero level set of the SDF using the CDF of the Laplace distribution. This choice enables the paper to derive a bound on the estimated opacity for any segment along a ray, and this is used to draw samples such that the error is limited to some epsilon. The paper experimentally shows that the resulting recovered surfaces are closer to the ground-truth geometry than surfaces extracted from level sets of NeRF's volumetric geometry, as well as meshes produced by COLMAP. Additionally, the paper experimentally shows that the proposed method is able to render novel views that are close in quality to those rendered by NeRF.
main_review: I like the paper and the general research direction, but I have some concerns regarding the experimental evaluation.

Strengths:

Overall, think that the idea of hybrid surface-volume representation and rendering is well-motivated, and this paper develops a well thought-out approach to recover NeRF-like models whose volume densities are concentrated around a surface. I intellectually like the idea of choosing a volumetric function of the SDF that lets us bound the opacity estimation.

The proposed method clearly outperforms IDR and level sets extracted from NeRF, in terms of surface reconstruction quality.

I think that the disentanglement results are very cool and thought-provoking.

Weaknesses:

I think that the main weaknesses of the paper are in the thoroughness of the experimental evaluation. I am listing a few of my questions/concerns below:

It is unclear how many actual MLP evaluations along a ray are used for each of the baselines and ablations in A1. I am concerned that the experiments are not controlling for the number of MLP evaluations along each ray when comparing different methods. For example, is it correct that VolSDF with 5 iterations is using more than 832 MLP evaluations per ray (5*128 in order to compute the error bounds for each sample for each iteration + 128 for estimating \hat{O} + final 64 "fresh" samples, and this is without accounting for the "upsampling" done in each iteration)? If this is true, then I think that the comparisons are potentially quite unfair, because NeRF results would probably be much better with this number of samples per ray (instead of 64+128 as in the default NeRF implementation). Could the authors please clarify this?

I think it would be useful to make the ablation figure in A1 into a more complete comparison across multiple scenes, and additionally clarify how many actual MLP queries along a ray are used by each of these methods, so readers can form a reasonable basis for comparison.

I've noticed these same noisy salt-and-pepper artifacts (like those seen in Figure 2 and mentioned on line 158) when experimenting with NeRF if I use stratified random sampling of points along the ray for the "coarse" MLP at test time (as opposed to training time, where NeRF draws random stratified samples along the ray for the "coarse" MLP, at test time, NeRF uses regularly-spaced samples for the "coarse" MLP). Is it possible that these artifacts are actually due to the NeRF implementation forgetting to turn off random sampling at test time?

The scenes shown in the supplementary video are composed of materials that have pretty simple view-dependent appearance (not too shiny and not illuminated by obviously high-frequency lighting). Furthermore, they are only rendered from a limited range of views, so the view-dependent appearance showcased in these results is not very complex. I would be very interested to see how this method performs on scenes like the Blender material ball samples (in NeRF's Blender dataset) that are much shinier and contain reflections of high-frequency lighting. Also, I'd suggest including rendered camera path results from the BlendedMVS dataset and camera paths with much larger motion (full 360s around an object maybe?) in the supplementary video, because it would help readers get a better idea how VolSDF performs.

I'm confused by the statement on line 290, which says that NeRF computes L(z,v) with a linear function. My understanding is that NeRF uses a single layer with a ReLU linearity (width = 128) that takes in z and v, followed by a linear layer that outputs the view-dependent emitted radiance RGB value. Could the authors please comment on the NeRF baseline implementation?

Another concern I have regarding comparisons to the NeRF baseline: line 59 of the supplementary PDF says that the VolSDF model has 4 layers of width 256 that represent the view-dependent emitted radiance. I think it would make sense to use the same higher-capacity architecture for the NeRF baseline for the experiments in Table 1 and the disentanglement comparisons.

A few other questions:

Does the representation still need to be initialized as a unit sphere SDF (supplementary line 57) in this hybrid surface/volume rendering setup? I would guess that this initialization is less important than it would be in a purely surface-based representation like IDR.

My understanding is that the alpha and beta parameters of the Laplace CDF are global for the whole scene. Would it make sense to instead have them be local, so that different regions of the scene could be more volumetric vs. more surface-like? 

The choice of setting alpha to the reciprocal of beta seems unintuitive, can the authors please elaborate why both can't be free parameters during optimization? What happens to the results if both are free?

Edits and suggestions:

Since the main goal of this paper is to improve the surface representation recovered from NeRF-like models (as opposed to improving view synthesis quality), I think it would be good to include more motivation why we should care about better surface reconstruction from these methods. Maybe mentioning robotics applications or other scenarios where we would like to use differentiable rendering to recover geometry for some application that actually needs high-quality geometry (one could argue that in view synthesis, geometry doesn't matter if the views are rendered well)?

I think that the usage of the term "light field" in this paper is slightly different from how it is typically used in graphics, and this may be confusing to readers. This paper uses "light field" to mean the view-dependent emitted radiance function in NeRF (how much radiance is emitted in any direction at each point), which is a quantity that is accumulated to render the radiance travelling along a ray. However, the typical usage of "light field" within graphics (see [Levoy and Hanrahan, SIGGRAPH 1996]) refers to the radiance along rays, which is actually the quantity that is produced from volume rendering. These usages of the term "light field" are slightly different, because the actual light field function would represent the accumulated opacity along all rays instead of just what is being emitted by particles at each location. I suggest changing the usage of "light field" in this paper to avoid confusion.

Line 22-23: remove "so called"

Line 69: add dash between non and Lambertian

Lines 105, 139, 147, etc.: capitalize "Equation" to match capitalization of "Figure" and "Section"

Line 114: it --> the volume rendering integral

First line of Fig. 3 caption: missing space after "1"

Line 313: malice --> malicious

I'd recommend proofreading the references. For example, CVPR is cited differently in [12] and [14], [17] was published at SIGGRAPH, NeRF [21] should have the R and F capitalized, NeRD [2] should have the R and D capitalized, NeRV [34] should have the R and V capitalized, the Adam optimizer was published at ICLR and not just arXiv, etc.
limitations_and_societal_impact: The paper includes an interesting discussion of potential future directions and a figure with example failure cases. I think that it would be nice to include a more thorough discussion of limitations in the updated draft.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The paper introduces a novel method for multi-view scene reconstruction and image synthesis. The authors combine volumetric and surface based neural rendering techniques to obtain highly accurate surface models without need for object mask annotation. The paper presents theoretical framework motivating the SDF-based density field parametrization and the adaptive sampling approach. The validation shows that the reconstructed shapes are surpassing the state of the art.
main_review: [Originality] 
The idea of parametrizing volumetric density through signed distance function is novel, very elegant and effective. I expect it to have significant impact on the community.

[Quality]
The algorithms are well motivated and based on theoretical analysis.

[Clarity]
The paper is clearly written and easy to follow. I particularly like the elegant definition of the SDF and surface manifold in Eq. 1. The proofs of theoretical relations from the paper are provided in the supplement though they could be expanded to cover more details. E.g. I am not entirely sure how the second inequality in the proof of Lemma 2 emerges from the derivation of Lemma 1.
The paper covers implementation details needed for reproduction and, furthermore, code was also provided.

[Significance]
The capability of surface-based inverse neural rendering without mask requirement lifts one of the major practical limitations from this category of methods and it should allow for easier and higher quality surface reconstruction methods in the future.

[Questions, Comments and Concerns]

1) Why is the density around the surface modeled using Laplacian distribution? Is there a physical reason for this particular distribution or is it just mathematical convenience?

2) The normal as input parameter of L makes sense but I would still expect explicit ablation testing the same model with and without this new feature. Does it affect reconstruction quality or is it only useful for the appearance swap demo in Sec 4.2?

3) The quantitative evaluation does not fully capture the image reconstruction quality. The shape reconstruction without masks alone is a great achievement but it would still be useful to learn limits of the image reconstruction quality. The PSNR scores seem competitive with Nerf(++), however, closer inspection of Fig. 5 shows that image features are rather smeared. The images are also of relatively low resolution. It would be useful to provide a metric more sensitive to perceptual qualities of the images such as PSNR. Further, how would the proposed technique compare to some of the recent surface-based techniques focusing on image reconstruction quality such as "Stable View Synthesis" [Riegler and Koltun 2021]? (I believe it should also be cited)


4) Minor comments:
- L122: transperancy -> transparency
- L209: while keep -> while keeping


[Conclusion]

Overall, this is a very high quality and high impact potential paper. I believe that the proposed hybrid formulation is a very promising way forward for surface reconstruction methods. My only notable concern with the paper is the extent of image reconstruction quality evaluation. However, it does not have a direct impact on the main contribution and it can easily be improved in the final version. I strongly recommend accepting.

limitations_and_societal_impact: [Societal impact]
The authors (and me) do not expect negative societal impacts. It could potentially lead to more effective 3D data capture, storage and transfer which is useful in many fields including cultural heritage preservation.



[Limitations]
The paper lacks listing of limitations (Or I could not find it). Perhaps with the exception of future work discussion in the conclusion. In the video I noticed some instability of the ray-traced surface in the skull example (may or may not be related to the representation) and potentially lack of spatial fidelity of the reconstructed image details. The method also shares the encoding/decoding computational complexity with its predecessors.
review_title: null

Meta-review: This submission introduces a timely and important contribution that integrates SDF representations with neural volume rendering.  All reviewers are positive, and three of the four reviewers strongly recommend acceptance.  The AC agrees.  The authors should try to address the reviewers' concerns in the camera-ready version. This includes adding the ablation studies from the rebuttal period, incorporating the discussion as L3KT suggested, among others. 
Decision: Accept (Oral)