Title: SfPUEL: Shape from Polarization under Unknown Environment Light
Venue: NeurIPS 2024
Accepted: yes
Keywords: shape-from-polarization

photometric 3D reconstruction

physics-based vision
Abstract: Shape from polarization (SfP) benefits from advancements like polarization cameras for single-shot normal estimation, but its performance heavily relies on light conditions. This paper proposes SfPUEL, an end-to-end SfP method to jointly estimate surface normal and material under unknown environment light. To handle this challenging light condition, we design a transformer-based framework for enhancing the perception of global context features. We further propose to integrate photometric stereo (PS) priors from pretrained models to enrich extracted features for high-quality normal predictions. As metallic and dielectric materials exhibit different BRDFs, SfPUEL additionally predicts dielectric and metallic material segmentation to further boost performance. Experimental results on synthetic and our collected real-world dataset demonstrate that SfPUEL significantly outperforms existing SfP and single-shot normal estimation methods. The code and dataset is available at https://github.com/YouweiLyu/SfPUEL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SfPUEL, which estimates surface normals and material (metallic or dielectric) under unknown environmental light using a single polarization image. The proposed network integrates polarization information with photometric stereo priors using a photometric stereo feature extractor and a polarization feature extractor with DoLP cross-attention. By jointly estimating material segmentation and surface normals from the global context features, it helps predict surface normals and materials.\n\nSTRENGTHS:\n* Synthetic and real polarization datasets for various materials and environmental maps.\n* A novel network architecture consisting of:\n1. Polarization and photometric stereo feature extraction\n2. DoLP cross-attention block\n3. Global context extractor\n* Reasonable ablation study (Table 3) with sufficient SOTA results compared to various normal estimation methods (Figures 5, 6, 7, 8 and Tables 1, 2).\n\nWEAKNESSES:\n* There are no comparison results with other material segmentation methods.\n* The method does not clearly show high-frequency, spatially-varying material examples, such as dielectric surfaces with dense strip patterns using metallic surfaces. Each object is composed of a single material only.\n* Closely related to the limitation, which is beyond the scope of the paper, there is no consideration of appearance.\n\nQUESTIONS:\nThe paper is well-written, and the network architecture is described well in both the main and supplemental papers. The published code also helps to address any further questions.\n\nLIMITATIONS:\nThe author addressed the following limitations:\n\n1. Requires an object mask\n2. Supports dielectric and metallic objects\n\nI might add that this paper is also limited to normal and material estimation only. Given the normal and material components, future research could focus on estimating material appearance parameters.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses Shape from Polarization under Unknown Environment Light (SfPUEL) from a single polarimetric image. Existing SfP methods have the ambiguity of surface normal caused by unknown illumination and materials and make some assumptions on reflection type or illumination. This paper introduces a novel SfP framework based on a transformer that considers the global context. This paper also proposes to combine SfP with pre-trained photometric stereo (PS) priors by using cross-attention based on DoLPs. For guidance to resolve the ambiguity by materials, the network outputs segmentation of dielectric and metallic materials along with surface normals. The experimental results show that the proposed method quantitatively and qualitatively outperforms the SOTA normal estimation methods using a single image or polarimetric image.\n\nSTRENGTHS:\n+ A novel transformer-based framework for SfP effectively constrains the surface normal by considering the global context and exploiting pre-trained PS priors with DoLP cross-attention.\n+ This paper proposes a joint estimation of material segmentation and surface normals to resolve the ambiguity caused by materials.\n+ The reconstructed normals are significantly better than the SOTA normal estimation methods.\n\nWEAKNESSES:\n- Since photometric stereo inherently requires multiple images captured under different lighting conditions, the pre-trained PS model is not supposed to take polarimetric images under the same lighting conditions as the input. This would lead to unexpected behavior and make it unclear how generalizable the features extracted from the pre-trained PS model are.\n- A polarized light source like a sunny sky affects the polarization of reflected light [20], especially for a specular dominant surface, such as a black dielectric surface and smooth metallic surface. Since the authors seem to create synthetic data from unpolarized environment maps, the proposed learning-based method cannot handle such cases.\n\nQUESTIONS:\n- Since the proposed method only requires a single polarimetric image (and its mask), is it possible to capture real-world data more casually if GT is not acquired? If so, testing a wider range of real-world data can validate the generalizability of the proposed method qualitatively.\n\nLIMITATIONS:\n- As mentioned in Weakness, the proposed method cannot handle a specular dominant surface under polarized illumination.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tries to solve normal estimation tasks using a single image captured from a polarization camera under unknown environmental light. To handle unknown environmental light conditions, this paper adopted a deep learning-based strategy to take advantage of inductive bias from the training dataset. Compared with previous methods, this paper additionally utilizes a pre-trained photometric stereo network and material segmentation tasks. The network architecture generally follows the Transformer and is trained with the synthetic dataset created by the authors. The proposed method is evaluated on the six real-world objects and shows reasonably good results compared with the GT normal map. It also shows results comparable to the multi-view SfP method (PANDORA).\n\nSTRENGTHS:\nThe paper shows that utilizing the pre-trained photometric stereo network can help the single-shot estimation of normal from the polarization images, which can be shown as a novelty of the paper. The paper also proposes some architectures that can be applied to the polarization images, such as the DoLP cross-attention block. The authors also captured some synthetic and real datasets, which can help future research if they are opened. Also, regardless of technical novelty, the quality of the results seems fairly good on both synthetic and real datasets.\n\nWEAKNESSES:\nThe proposed network seems to rely strongly on the photometric stereo network SDM-UniPS. It not only borrows its architecture but also adopts network weight for extracting the photometric stereo features. Such dependency on the specific network can be treated as a weakness. Also, refer to the first question, which includes the question about SDM-UniPS.\n\nIt is unclear why the BSDF type is separated into only two: dielectric and metallic (conductor). The authors provided examples and short statements but didn’t explain why they are different. This makes it weak to add the material segmentation tasks only for two types.\n\nQUESTIONS:\n1. It is unclear how polarization images can be treated as the photometric stereo input. The only theoretical reason I found it on the L42-44, both SfP and PS estimate the result of the same pixel position, with measurement under different conditions. Yet, PS usually assumes the difference in lighting condition, and SfP assumes the difference in polarization direction. Are there any more specific analysis for this? Besides, will it work if we replace SDM-UniPS with other PS networks and follow a similar training strategy?\n2. Related to the second weakness, what makes the difference (e.g., the complex-valued IOR), and how is the difference observed (in both RGB and polarization domain) between dielectric and conductor material? To make the paper more self-contained and concrete, an explanation about this seems necessary.\n3. Also, the following questions are related to the practicability of the method.\n- This method also estimates the material segmentation, but the results for real data’s material segmentation are missing. Are there any reasons for excluding this?\n- Since trained synthetic data consists of dielectric and conductor material, can this method deal with diffuse objects? Also, objects cannot be classified into simple dielectric or conductor in the real world—for instance, multi-layered materials or coated materials. What happens if we use the proposed method for these objects? Such limitations should be clearly stated, and a single statement in the Limitations section does not seem enough.\n- It seems the method assumes the environment light is unpolarized. Since there is also a polarized light source, is it considered in the training dataset?\n4. Is there any plan to make the training dataset and full test dataset publicly available? Such effort to collect datasets will be helpful for future research.\n\nLIMITATIONS:\nNo potential negative social impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}