Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents the neural uncalibrated photometric stereo method under the single directional lighting setup. Following recent neural-field-based algorithm, the method simultaneously predicts surface attributes and lighting parameters in a unified self-supervised learning method. Shadows are explicitly considered for filtering out its contribution in the optimization process. In the evaluation, the proposed method achieved state-of-the-art performance on the public DiLiGenT benchmark.
strengths_and_weaknesses: Strength:
- The proposed method is very clearly written and the algorithms are easy to understand.
- The benchmark study shows that the proposed method is superior to the competitor.
- It is very interesting that the loss is propagated from the cast shadow since the shadow mapping is generally indifferentiable.
- Ablation studies have been conducted for each of the technical components.
- The application of neural fields to uncalibrated photometric stereo is new among published studies.

Weakness:
- It seems an overclaim that there are no explicit assumptions about the light and reflectance since the light is obviously constrained to the single directional lighting excluding near and natural lighting.  As for the reflectance, how to handle spatially-varying BRDF is not clear since SpecularNet is only queried by nh and vh without the location of the pixel. It seems that the proposed method only scales a single BRDF by the specular coefficient rho_k, which obviously limits its representability. The more proper manner seems to define multiple bases (neural networks) and compute multiple coefficients as was done in [1].

- The behavior of ShadowNet is not clear to me. Since the gradient of binarization layer is static, the loss doesn’t seem to be propagated. Therefore, it seems to me that the shadow is not explicitly considered for recovering the geometry unlike the shape-from-shadow and it is only used to detect outliers to make the reconstruction loss robust to cast shadows as is similar with [1]. If so, it seems that simply detecting shadows statistically and reflecting them in the loss function would be sufficient. This work may need evidence that results from ShadowNet are superior to statistical shadow removal.

- The physical parameters recovered by the proposed method have not been properly evaluated, so it is not possible to examine whether the formulation is appropriate. From the results presented in Fig17 of supplementary, it appears that the proposed method couldn’t successfully recover physical attributes such as shadows and reflectance, which makes the proposed method less useful.

- All results are based on public evaluation data sets taken in carefully controlled environments. In addition to those results, I would have liked the authors to have shown the actual results taken by authors to demonstrate its availability in real use-case.

[1] Li and Li, Neural Reflectance for Shape Recovery with Shadow Handling, CVPR2022
questions: - The concurrent work [1] also introduced the neural fields for the photometric stereo. This work and [1] seem very close to each other for the modeling of shadow and BRDF even though this work solves the uncalibrated task unlike [1]. Other than this difference, is there any critical difference between these works that makes this work more significant?

- The proposed method certainly works in the environment assumed in the benchmark, but there is some question as to whether this research direction is the right for the photometric stereo field. Since many applications in photometric stereo are time-critical (i.e., surface defect inspection), it is questionable whether the small improvement in accuracy obtained with the neural field's approach is worth the sacrifice the efficiency in learning-based photometric stereo methods which only take less than a second to recover comparable results. In addition, I believe that data-driven methods can handle a more generic environment including the global illumination effects under a more complicated lighting environment than an inverse rendering that relies on a specific physically-based model. There is no point in simply improving benchmark scores if they are not immediate to real problems. Therefore, I would like to see a little more convincing motivation of this work.

limitations: The limitation is addressed in the paper.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposed a method that aims to solve the uncallibrated photometric stereo problem where the input light intensities and light directions are unknown. The key idea of this paper is to represent the surface normal, and surface reflectance, shadows, and lights as neural networks. And then jointly train those neural networks via the image reconstruction loss in an unsupervised manner. 

The proposed method explicitly considers the clues from specular effects and cast shadows for solving this problem. 
Specifically, the paper proposed: a PositionNet for outputting the surface normal, specular weights, and diffuse weights of the object; a SpecularNet that takes half angle and surface normal as input, outputting the specular component; a ShadowNet that explicitly predict the shadow map; and LightNet that takes RGB images as input, outputting the light intensities and directions.

The performance of the method was evaluated on DiLiGenT and some other real datasets and achieved lower normal estimation errors than other state-of-the-art algorithms.
strengths_and_weaknesses: Strengths 
- The paper is well written and easy to follow.
- The evaluation of the proposed method is extensive and complete, and supports the claimed effectiveness of each component of the method.

 

Weaknesses
- The design of the PositionNet and SpecularNet is very similar to the design of [26] ("Neural Reflectance for Shape Recovery with Shadow Handling"). But I didn't find a proper reference to [26] to state the difference: both [26] and this work have a network outputting the surface normal, diffuse albedo, and specularity; both [26] and this work have a SpecularNet which take normal, half-angle, and view direction and outputting the specular effects of the surface; both [26] and this work explicitly compute the shadow map which is then used in image rendering; both [26] and this work use a warm-up loss for early-stage supervision of the shadow map. I encourage the authors to discuss the differences to help clarify the position of this work.

- How does the SpecularNet work for objects with specular-SV-BRDF? Especially for a object with two different materials: soft and shiny specular effects? (such as a object make up with two different metals: one with high roughness, one with low roughness).
The input of SpecularNet is only the surface normal, half angle, and view direction. So for different positions over the surface, they will have the same SpecularNet, which means that it assumes all the surface points will have the same specular effects (same specular BRDF curve, or same specular roughness). But this isn't the case for most of the objects in the world: for some objects with spatially-varying BRDF where both soft specular effects and very shinny and sharp specular effects exist on the surface. 

- Unsupervised or weak-supervised? The method is claimed to be an unsupervised method, but as mentioned in line 194, the method uses the light estimation results from YS97 as a weak-supervision signal to train the LightNet. Besides, from Supplementary Table 9, it seems that without using YS97 as weak-supervision in the early-stage warm-up will cause a significant drop in accuracy.

- For the design of LightNet, since we need to train the LightNet in a weak-supervised manner from YS97, why not just use light directions from YS97 as the initialization and directly optimize on these lights. (i.e. why not explicitly optimized the light directions instead of optimizing the parameters of LightNet?)

- Since we need to train the LightNet in a weak-supervised manner, why don't we choose a better method as the "teacher network"? For example, why not use light direction and intensities from SDPS[12] as the guidance to guide the training of LightNet?

- The ambiguity in ShadowNet. I believe that there is ambiguity in the design of the proposed ShadowNet. From equations (3) and (4), it seems that the authors aim to use ShadowNet to represent the cast shadows. In the object's appearance, both pixels that are in cast-shadows and attach-shadows have 0 intensity. 
But their causes are completely different; the cast shadows are caused by objects self-occluding the light source; the attach-shadows are caused by the surface back-facing the light source. And surfaces with extremely dark materials may sometimes also have zero pixel intensity under certain light directions. And the proposed method will have ambiguity under the above situations. The downside of this ambiguity can actually be seen in Fig 4, 5th column, where the generated shadow map shows the inconsistent boundaries in many objects (e.g Standing K. and Gourd2, Apple).  
questions: How will the number of input images affect the testing time of this method? e.g., how long does it take to test with 16 images at input?
limitations: - The rendering equation doesn't take inter-reflection into account. Hence, in theory, the proposed method will fail if the inter-reflection effects dominate the surface. 
- The photometric stereo images are usually taken in a dark room with limited lighting exposures, which may also introduce high-level image noise. I encourage the authors to discuss more about the robustness of this method when the image assumptions are violated in these aspects.

Suggestion:
- I encourage authors to list the testing time of other most recent inverse rendering-based methods [44,25,26] for comparison.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a novel training framework and model design for unsupervised photometric stereo, which is the problem of estimating surface normals of an object given a set of images with uncalibrated lighting positions. The model leverages Neural Intrinsic Fields, which are a set of MLPs which represent shadows, positions, specularity and light, as well as a CNN encoder aimed at aiding the representational power of the system. The training procedure uses a set of domain-specific loss functions and a warm-up strategy aimed at improving stability and generalization. Ablation studies are provided, which show the effectiveness of the model and the training procedure, and comparisons with previous work show quantitative improvements. 
strengths_and_weaknesses: *Summary* 

This paper provides valuable ideas for training methods for unsupervised photometric stereo, which is an important problems in computer vision and graphics. The proposed method is sound, ablation studies are provided that validate the effectiveness of the design choices of their method, and the model combines advances in machine learning with domain-specific ideas in a smart way. However, the paper is not well written, presentation is arguably poor and comparisons with previous work do not show a huge improvement. Given that the paper is hard to read and comparisons with previous method do not show significant improvements, it is difficult to recommend this paper for acceptance at its current state. 

Strenghts:

- This paper introduces novel ideas for uncalibrated photometric stereo, combining advances in machine learning with ideas from more classical computer vision and graphics. 
- The proposed method is technically sound, and ablation studies show the impact of each component in the overall generalization capabilities of the model. Evaluation is somewhat exhaustive. 
- Comparisons with previous work are complete and exhaustive to the best of my knowledge. 

Weaknesses:

- The paper needs revision in the way it is presented and written. It is hard to follow and some sentences are very difficult to understand. The paper would benefit strongly from a revision on this level. Further, it is presented in a way that assumes a lot of previous knowledge from the reader. Please see below for suggestions on references that may help the render understand better this paper and its context.
- Previous work is not adequately studied and it is not clear where this paper sits on the literature. By reading the title and some parts of the paper, it would seem that this is a NeRF paper with applications for uncalibrated photometric stereo. This is not the case, and I believe the paper would benefit from a more clear distinction with Radiance Fields methods. 
- Comparisons with previous work could be presented better, in my opinion. For example, standard deviations should be shown next to averages (eg Table 1) and sometimes previous work shows better perfomance that the proposed method, but the authors decide not to highlight that (eg Table 3, CW20 is better in AVG dir but this is not in bold). It is not entirely clear how much better the proposed method is with respect to previous work, and, when it is not, why previous work is better. 
questions: - On line 34, it is claimed that supervised UPS methods fail at generalizing to data points outside the training distribution. The authors thus present unsupervised learning as a better alternative for enhancing generalization. This argument is repeated throughout the paper, but, why is this the case? Why is unsupervised learning inherently better than learning from labels in this particular scenario? Please explain better or demonstrate quantitatively.
- The paper could benefit from referring the reader to important papers that would help in contextualization and learning about this particular domain. For Neural Fields, I would suggest referring to (Neural Fields in Visual Computing and Beyond, Xie et al.), which should help the reader understand that this UPS paper is not that closely related to NeRF papers as the title and the related work suggest. For material models and intrinsic decomposition, I would suggest (A Survey on Intrinsic Images: Delving Deep Into Lambert and Beyond, Garces et al.). 
- Please discuss the limitations of the material model in terms of which effects are not modelled with the current image formation assumptions, as many effects cannot be represented by NeIF but this is not clear to the reader. 
- Please explain terminology that is not explained nor it is obvious to the reader. For example, GBR, Lambertian objects, etc.
- Please include standard deviations in the tables, to better understand whether or not the improvements are significant. 
- Please double-check the references (eg line 185).

limitations: The paper does not include any negative impact analysis of their work. It is difficult to imagine any direct negative impact of surface normal estimation, but I would argue that this method is computationally very expensive at test time and it may be detrimental in terms of energy consumption. In terms of limitations, the paper includes an adequate limitation analysis. To complete it, I would suggest the authors to discuss the limitations of the material model that their method assumes for training, as it does not account for complex light phenomena, including sub-surface scattering, transmittance or iridescence.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: An unsupervised method is proposed for uncalibrated photometric stereo. It works by parameterising lighting, reflectance, normal map and shadows by neural nets, a.k.a. neural intrinsic fields a.k.a. NeIF. These networks are supervised by rendering loss, with additional regularisations on boundary normal directions, lighting directions, BRDF monotonicity and shadows. Experiments show overall superior results than unsupervised or uncalibrated methods.


strengths_and_weaknesses: Strengths:
- The method is uncalibrated and unsupervised, meaning it does not require any additional information other than input images themselves.
- Although implicit neural nets are not new to PS, the intrinsic decomposition paradigm makes sense and the method is technically sound.
- Experiment results are convincing. 

Weakness:
- I think the paper could benefit from being more upfront about its assumptions and limitations. For example, lighting is assumed to be distant, and BRDF being a bivariate one (with half angle and difference angle parameterisation) with a single specular lobe. 
- While the BRDF parameterisation does allow variations between specular and diffuse, one could argue the single specular lobe assumption is too restrictive for diverse, spatially varying materials, e.g. an object made with two specular materials with different BRDFs.
- I need more convincing arguments about why and how the bas-relief ambiguity is resolved here. The boundary normal azimuth penalty in eq(6) does not fully disambiguate light/normal directions. However the experiments seem to suggest both normal and light directions are in fact successfully recovered. 
- One strength of this paper is that it deals with varying light intensities. However there is no mention to how much light intensity varies in experiments (e.g. diligent dataset). I wonder how the results would change if you apply a random scale to each input images.
- Minor typos in reference style:
    - line 25: [4], [34] -> [4,34]
    - line 76 [bi2022neural]
    - line 185 [8758025]
    - missing citation numbers in figure 3
questions: What is ours(16) and ours(19) in table 1,2? Are these results with sparse and all input views because I cannot find any explanation in texts?




limitations: I think there is room for improvement. See my comments above.
review_title: null

Meta-review: metareview: This paper had reviews ranging from a Reject to a Weak accept.

The key shared concerns among reviewers were concern about how much is really new relative to the CVPR paper [Li et. al, 2022].  The most negative reviewer (who is quite expert in this field) engaged strongly in the discussion with the authors, highlighting sustained concerns about novelty, substantially slower speed at rendering time, and the loss of the ability to render materials with interesting reflectance properties.

I found this review most accurate and detailed.  The remaining reviews, while borderline or weakly positive, retained concerns about the limited lighting and reflectance properties.

Therefore I am deciding to reject this paper
confidence: Less certain
Decision: Reject