Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a promising framework for near-field photometric stereo utilizing volume and surface rendering. The proposed method is tested on a synthesis dataset and a real dataset. Their results have reached state-of-the-art. 

strengths_and_weaknesses: ---Originality---
1) The overall idea shows the originality, especially in how to combine surface and volume rendering.
2) Imposing the shadow's constraint on the occupancy field in the 3D space is enlighting. 
---Quality---
1) Eq.8: the author used $C_v$ to denote the volume-rendered image and $C_s$ to indicate surface rendered image. However, no formula illustrates how $C_v$ is calculated, which  confuses reading. 
2) Figure 2: two "MLP" blocks are shown in two separate branches, which makes me think there are two different MLPs estimating occupancy along the surface-to-light segment and the camera ray. However, according to Eq.6, the MLP for shadow handling is identical to the MLP for the occupancy field. Please try to redraw this part.
---Clarity---
1) Figure 2's caption, lines 2-3: How the points along the rays are sampled is unclear to me. The author mentions to sample $N_V$ points around the surface and $N_L$ points on the surface-to-light segment. I assume $N_L$ points are sampled uniformly, but without knowing the scene's scale, it is unclear how to ensure the sampled points along the camera ray are "around the surface."
2) Although the author tries to explain the necessity of the shadow clues for surface reconstruction in Section 4.4 and Section B in the supplementary material, it is still unclear to me why the shadow becomes a strong constraint on the model that can substitute the constraint from multi-view photo-consistency. There are two kinds of cast shadows in the synthesis dataset, including the foreground shadow on the object, and the background shadow. According to the results in section F, table S2, and section 4.2 table 2, the visibility of the background shadow is very important for depth estimation. The author should analyze the contribution of two different kinds of shadows separately.
3) According to the presenting results, the proposed method has reached an ideal visual and quantitative result in their synthetic dataset. Even the background is well reconstructed with little artifacts. However, the author didn't directly explain why their method is not sensitive to the background (it is almost perfect) and can reconstruct an accurate background simultaneously. (e.g., UNISURF [A] applies NeRF++ [B] for the complex background and assumes black for the simple background. However, their reconstructed background still contains visible artifacts).

[A] Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2021

[B] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv.org, 2020.

---Significance---

1) There is no intuitive explanation for the role of modeling shadow for shape estimation. What is the problem aiming to solve by considering the shadow, the ambiguity, additional clue of global shape information, useful clue for depth estimation, or even light refinement? 
2) The evaluation is mainly conducted on the synthetic dataset built by the authors, which is less convincing. Although the paper shows the result of the LUCES dataset in the supplementary material, the depth estimation is not satisfactory (4.39 vs. 3.17 for the state-of-the-art). This evaluation could not provide strong support to the proposed method.
3) The compared methods, NeRF and UNISURF, are implemented in a naive way. It is unclear whether the authors have also sampled points along the light ray in those methods. If not, it can be expected that the reconstruction results are not good because NeRF and UNISURF are mainly applied in multi-view stereo, which is constrained by the multi-view photo consistency that is lacking in single-view photometric stereo. Therefore, using them for comparison is less convincing. 
4) As said before, the background shadow's visibility greatly influences the performance. Therefore, this work has lots of pre-setting to the environment (the background is better to be the light color, and the intensity should be bright enough), which will limit the application scope of this method. Moreover, The background's complexity may influence the background shadow's visibility and the shading of the scenes, which further impact the performance of the methods. While the background in the synthesis dataset and LUCES is still an ideal case, there is a lack of such discussions on a more realistic scenario.

questions: Except for the problems listed above, I wonder how well the proposed method can perform under a more complicated background.

limitations: The authors have addressed parts of the limitations and potential negative social  impact of their work. However, the impacts of the background's complexity and the shadow's visibility are not stated.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This pape proposed a method that aims at solving the near-field photometric stereo via neural reflectance field representation. The core insight of this paper is to use both the shading and cast-shadows of the view as the cues to recover both the visible and invisible parts of a scene. 
The paper adopts the NeRF-like neural field representation to represent the occupancy field, reflectance field of a scene. They then use lighting, BRDF modeling and shadow rendering to reconstruct the observed images. The networks are jointly optimized via volume rendering loss, smoothness loss and color loss. 
strengths_and_weaknesses: Strengths
- The proposed method can estimate the geometry of the invisible region of the scene from a single-view point. The paper is interesting in estimating invisible regions from the shadows. And unlike previous shadow-based methods, this paper doesn't require explicitly shadow detection as input.
- The method presents state-of-the-art geometry reconstruction results compare to prior near-field photometric stereo methods. 
Each component of the method, such as shadow and shading, is validated in the ablation studies.
- The paper is well writen and easy to follow. 

Weaknesses
- Each part of the proposed method is not novel. The method consists of three major parts: neural reflectance field representation follows [38] in occupancy field estimation and normal computation; the color and BRDF modeling follow [15,24] to use the sphere gaussian bases, SG weights and diffuse albedo; the shadow computation is slightly different to [48], but it is similar to [24] where they both check the visibility of the surface point. Hence, in the proposed method and rendering procedure of this paper, the novelty is very limited.

- It is suggested to evaluate this method in more challenging real and synthetic datasets. The method was only tested on two datasets: a simple synthetic dataset and a real dataset. For the synthetic dataset: the background is clean and uniform; how will this paper perform in backgrounds with more textures?

- I am curious to see how the number of lights affects the accuracy of the geometry in the invisible part of the scene. The only cues for invisible regions are the casted shadows on the ground. If lights fail to illuminate the invisible regions from different angles, how will the accuracy be affected?

questions: Please see the quesitons in the Weaknesses part.
limitations: - Limitations are well discussed in the paper.
- Suggestions: It will be great to isolate the invisible region of the scene and show quantitatively how the proposed shadow rendering improve the accuracy in these regions.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper is about using neural radiance field to reconstruction the geometry of an object lit by varying directional light sources, leveraging cues of cast shadow. The optimization uses an efficient way to trace rays for shadow computation.
strengths_and_weaknesses: The setup is interesting as in using shadow cues to reconstruct 3D geometry when multi-view captures are not available. The scope of the paper is quite limited, in terms of the problem setup and the practicality. The setup is demanding -- real directional light plus known lighting direction. This highly controlled lighting setup seems to only be feasible in a laboratory or a light stage setting. This may be the reason the paper only shows toy synthetic examples. Another 'contrived' setup is in the simple scene / background. Cast shadow depends on the object geometry, the lighting, and also the surface the shadow is cast on. The geometry of the surface and the geometry of the object can be entangled to cause ambiguity. This is more severe in real-world examples. The task can be unnecessarily (due to such setup being unlikely in real-world scenarios) challenging and ambiguous without any priors on the object geometry. The reconstructed geometry is not on par with multi-view settings, and degrades drastically as the number of lights decreases. My biggest concern about this paper is it makes the reconstruction task unnecessarily challenging and impractical.
questions: - to make the setup slightly more practical, how well does the method perform when shadow is softer, i.e., slightly larger light source, or small area lights?

- line 149: has the proposed shadow generation method been validated over direct regression method using an MLP?

- fig 4: should also show the nearest neighbor training samples
limitations: The calibrated lighting is more demanding than calibrated camera pose, which can be easily acquired, even though not entirely accurate, on most modern smartphones. The lighting needs to be real directional light source, which is difficult to acquire in real-world scenarios.

It would be also valuable to discuss potential applications for the proposed setup and algorithm.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed a single-view scene reconstruction method under different point light conditions based on the neural reflectance field. It uses the inverse-square law and BRDF to model the shading and root finding to model the shadow. By explicitly utilizing both shading and shadow cues, the model can reconstruct the scene from single view, even for some invisible regions.
The authors evaluated the model with different objects and showed that the method can well reconstruct the geometry with images from single view but different point lights.

strengths_and_weaknesses: ### Strengths

A novel NeRF-based method that faithfully reconstructs a scene from a fixed viewpoint with multiple single point light conditions by explicitly modeling the shading and shadow. By carefully modeling the single point light shading by inverse-square law and BRDF, and shadow by root-finding. This method gives an excellent result.
The paper clearly illustrated the method in detail. The authors compared their method with other methods quantitatively and qualitatively. They also evaluated their model with multiple occluding objects, showing it can reconstruct the unseen area with shading and shadow cues.
 
### Weakness

The authors studied the contribution of different components (shading and shadow) of their method. But the impact of different loss terms was not included. Specially, at L171, how the value of $\alpha$ for normal smoothness loss is found is not illustrated.
   

questions: How’s the normal smoothness loss affect your method? Is that similar to UNISURF? From your result, especially on LEGO, the loss of detail of studs and track seems very serious. Is that caused by normal smoothness loss or by more complicated light reflection at those areas?

limitations: Yes, the authors addressed different aspect of limitations of the work in the paper and supplementary materials. They also pointed out this method can recover some non-directly seen area, that may address some privacy issues.
review_title: null

Meta-review: metareview: This paper had reviews ranging from borderline reject to strong accept.  The most negative reviewer had concerns about the assumptions in the framework (point light sources), and the loss of accuracy as the number of light sources decreases, but the remaining reviewers were compelled by the ability to hand scenes with lights not at infinity and the integration of the shadow constraints to give constraints on the structures of scene parts not directly viewed.

Overall I agree with the three positive reviewers that this paper considers an interesting variation of the photometric stereo problem with coherent experimental evaluation that shows the contributions of each of the different pieces of their overall system

Therefore I accept this paper.

confidence: Certain
Decision: Accept