Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Implicit representation has become a popular technique for 3D scene reconstruction. Existing methods do not utilize the multi-view geometry constraints in the learning. In this paper, by leveraging sparse geometry from structure-from-motion (SFM) and photometric constraints in multi-view stereo. 
 
The paper compares the proposed method with both classical (colmap [28]) and recent deep learning baselines (IDR [40], VolSDF [39], NeuS [33], and NeuralWarp [7]).

strengths_and_weaknesses: Strengths

1) The paper builds on top of other methods such as IDF [40] in handling multi-view geometry constraints while learning the model. 

Weaknesses

1) Some of the top-performing SDF methods from poinclouds are not cited. Using the pointclouds from SfM methods, we can generate implicit 3D representations using methods such as points2surf:

Erler et al. Points2Surf: Learning Implicit Surfaces from Point Cloud Patches, ECCV 2020.

2) The chamfer distance shown in Table 1 has minor issues. For example, the proposed method is not the best performing method on scan 83. NeuS achieves better results. 

3) The paper heavily relies on existing structure-from-motion pipelines  [51, 47, 22, 19] to compute the camera parameters.  In some sense, the MLP networks are essentially utilized to further refine the multi-view geometry constraints. Furthermore, if the classical methods fail on a scene it would be hard to use this method. The improvement in the results need to be explained more carefully since the geometry constraints are already implicitly incorporated while using the  structure-from-motion pipelines. 
questions: The proposed method uses an 8-layer MLP for generating features from 3d points. We can also use a permutation-invariant method such as pointNet to generate features from 3D points. All the baselines are somewhat constrained to IDF-style algorithm. It would be good to also consider other baselines like Point2Surf or DVR in addition to the baselines shown:

limitations: The paper clearly mentions the limitations, especially the challenge in improving computationally efficiency.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a new method for geometry-consistent neural implicit surfaces learning. The proposed method includes a theoretical analysis of the gap between volume rendering and point-based SDF modeling and a solution by leveraging sparse points from SfM and utilizing patchmatch to provide geometry consistent supervision. This method is evaluated on DTU and BlendedMVS and achieves improvements compared with existing methods. 
strengths_and_weaknesses: In this paper, the main contribution is incorporating depth supervision (from sparse points) and patch match supervision into NeuS’s optimization framework, which leads to better reconstruction results. It seems that the improvement mainly comes from integrating patch match, shown in Tab. 2, which has been demonstrated by NeuralWarp. So the discussion about these two methods is expected. 

The paper is well organized, and the evaluations are sufficient to support the proposed method. 
questions: 1. What’s the key difference between NeuraWarp and Geo-NeuS? Both methods adopted patch-match based optimization for better geometry reconstruction. The authors should provide a detailed discussion between these two methods and show their strengths, especially in the introduction and related works. 

2. Sparse points from SfM are utilized to supervise the SDF network, where the sparse points with a radius filter are supposed on the surface (line 162). However, in practice, the sparse points from SfM are usually noisy, and wrongly incorporating these points with large errors may decrease the reconstruction quality.  

3. The authors claim that discrete sampling can cause bias (line 138) and linear interpolation is adopted to get surface points (line 195). What’s the error of the extracted surface points by these two methods, i.e., the proposed method of Geo-NeuS and the hierarchical sampling method in NeuS? This experiment may support the author’s assumption and theoretical analysis. 

4. The authors are encouraged to conduct a qualitative comparison between NeuralWarp and Geo-NeuS.  

5. The training time is 16h with 300k iterations on NVIDIA 2080TI (line 259) while the training time of NeuS reported in its original paper on NVIDIA 2080TI is also about 16h. However, Geo-NeuS requires extra computation for depth and patch match supervision. I’m not sure whether this number is correct. Besides, what about the accuracy with fewer iterations, as discussed in line 313, such as 150k or 200k iterations?   

 
limitations: Limitations or failure cases are not discussed in this paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents neural implicit surfaces learning by enforcing explicit SDF constraints. It analyzed the gap between volume rendering integration and implicit SDF learning. Based on that, it proposes an explicit surface point SDF supervision where the surface points are estimated with SFM. in addition, it locates the zero-level set of SDF networks and a multiview photometric consistency loss is proposed to explicitly supervise the training of SDF networks. The results demonstrate high quality 3d scene geometry reconstruction, even on thin structures and large flat regions. 
strengths_and_weaknesses: - The paper is well organized and written. 
- Comprehensive experiments demonstrate the geometry improvements brought by the proposed 2 losses.
- It provides theoretical analysis for the geometry bias, although I would like to see an evaluation of the bias after introducing the 2 losses (see details below)
- Such warping patch losses have been explored in recent papers, such as NeuralWarp and MVCGAN, CVPR 2022
questions: - For the multiview photometric consistency loss, it is not clear to me how the reference view is picked w.r.t. the source view. Is it pure random or within a perturbation range? Eq. 19 determines the ray intersection with the implicit surface. However, it seems to me it does not guarantee the intersection point will be visible for all the reference views. If the point occluded in the reference view, will the multiview constraint still be valid?  

- It is also not clear to me why this approach does not require foreground mask while still being able to reconstruct the 3d scene without background. 

- Please clarify how the ray sample weights w are converted from SDF values. 

- Both the SDF loss and the multiview consistency loss seem reasonable to help regularize the geometry learning. However, it is not clear to me how they are directly linked to the motivation given in sec 3.1. Will it be possible to evaluate the bias w/wo the proposed losses? 

limitations: - The multiview photometric consistency loss does not consider the view dependent effects. it remains questionable for reflective or specular objects. 

- It will be interesting to show the effect of number of reference cameras on the rendering quality. On one hand, with less reference views, the SFM might produce noisy result and affect the SDF guidance loss. It is not clear so far how the quality of SFM points affect the 3d scene reconstruction. On the other hand, multiview consistency loss might help regularize the neural field distribution and compared to baseline models, it might still converge with sparser training cameras.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors of this manuscript propose Geo-NeuS, a novel neural implicit 3D reconstruction method.
They combine a SDF-based neural implicit surface representation, that is usually only optimized with the reconstruction loss, with explicit multi-view constraints.
More specifically, they incorporate supervision in the form of i) the sparse point cloud obtained from SfM, ii) photometric consistency from classic multi-view stereo.
They first theoretically analyze the difference of volume rendering and surface rendering-based approaches wrt. 3D reconstruction, and then show experimentally that adding the proposed constraints leads to better 3D reconstruction and more accurate surfaces. 
strengths_and_weaknesses: # Strengths

1.) The proposed method clearly improves quantitatively (Tab 1) as well as qualitatively (Fig 4) over the state-of-the-art.

2.) The proposed additional losses are well-grounded, not too complicated in general, and clearly have benefits for the task of 3D reconstruction (Tab. 2)

3.) The ablation study (Tab. 3) of applying the proposed losses either via expected depth maps or via surface points found via root finding is very interesting. 

4.) While the findings of the theoretical analysis of volume vs surface rendering approaches for 3D reconstruction (Sec. 3.1) are expected, it is still valuable to have it.

5.) The manuscript is written clearly, has a good structure, and experimental evaluation is enough to show the benefits of the proposed system.

# Weaknesses

1.) BlendedMVS is not evaluated quantitatively, and I couldn't find an argument for this.

2.) It is not clear to me why the explicit SDF supervision (Sec. 3.2) is done with occlusion handling (L. 165) and view-aware (L. 173). It is only stated that "the introduced SDF loss is consistent with the process of color rendering" (L. 177 - 178). Instead, in every iteration, a subset of the sparse point cloud could be sampled and the loss can be applied on the 3D location without any occlusion reasoning etc. which seems simpler and more straight-forward. I believe a good reason / ablation study for this complicated setup is missing

3.) The described process of finding the surface intersection (L. 197ff) is very similar to proposed root-finding methods from ray marching-based approaches for neural implicit surfaces like [23] and a short note on this+citation on this would be helpful for the reader.

4.) The fact that the photometric consistency loss is only applied on grey-scale images (L. 211ff) is interesting, and an ablation study on this would be helpful.

5.) NCC is used as the phometric consistency metric. Have the authors investigated other measures as well? This could be an interesting ablation study (but not a must in this manuscript).

6.) It is not clear how the patch size of 11x11 (L. 227) was determined.

7.) The fact that Colmap's best trim parameter is 7 (L. 253) should be cited, e.g. [23]. 

8.) The visual ablation study (Fig 5) could be bigger with zoom-in windows to better see the differences, similar to Fig 1 of Supp Mat.

9.) Table 3 / "Geometry bias of volumetric integration": very interesting, but details are missing. Are here the expected depth maps used obtained via volume rendering? I think at least the supp mat should contain relevant formulas how the quantities are obtained.

10.) Appendix C: Why is NeRF's rendering quality worse than NeuS / Geo NeuS?

11.) Would be interesting to further discuss or which situations the losses help in particular, e.g. mostly for specular areas?

12.) Fig 4 caption typo: NueS -> NeuS

questions: 1.) Why is BlendedMVS not evaluated quantitatively?

2.) Why is the explicit SDF supervision (Sec. 3.2) implemented with occlusion handing and view-aware, instead of a simple loss via sampling in 3D space?

3.) Why are grey-scale images used for the photoconsistency loss? Could an ablation study help to support this design choice?

4.) Why did the authors use a patch size of 11x11?

5.) Where / For which type of surfaces do the proposed losses help in particular (e.g. specular surfaces, homogeneous areas, etc)?
limitations: The authors discussion on limitations and negative societal impact (L. 331 - 334) is quite limited. The authors could tackle more complex datasets / more complex scenes if the proposed system does not fail on DTU and the selected BlendedMVS scenes. What happens if photoconsistency is not given, e.g., because of strong specular highlights? How does the model perform in the presence of transparent surfaces? How could the model handle sparse set of inputs instead of the considered dense coverage of scenes? These could be starting points for an interesting limitation discussion.
review_title: null

Meta-review: metareview: This paper introduces new and useful losses, presents a good experimental setup, supply analysis on the bias, and is clearly written. I encourage the authors to discuss similarities and differences to NeuraWarp and pointcloud->SDF methods in their revision. 
confidence: Certain
Decision: Accept