SfPUEL: Shape from Polarization under Unknown Environment Light
Authors: Youwei Lyu
Heng Guo
Kailong Zhang
Si Li
Boxin Shi
Venue: NeurIPS 2024 poster (2024)
Keywords: shape-from-polarization
photometric 3D reconstruction
physics-based vision
Primary area: machine_vision
TL;DR: Single-view shape from polarization by integrating pretrained model knowledge under unknown environment illumination.

Abstract: Shape from polarization (SfP) benefits from advancements like polarization cameras for single-shot normal estimation, but its performance heavily relies on light conditions. This paper proposes SfPUEL, an end-to-end SfP method to jointly estimate surface normal and material under unknown environment light. To handle this challenging light condition, we design a transformer-based framework for enhancing the perception of global context features. We further propose to integrate photometric stereo (PS) priors from pretrained models to enrich extracted features for high-quality normal predictions. As metallic and dielectric materials exhibit different BRDFs, SfPUEL additionally predicts dielectric and metallic material segmentation to further boost performance. Experimental results on synthetic and our collected real-world dataset demonstrate that SfPUEL significantly outperforms existing SfP and single-shot normal estimation methods. The code and dataset is available at https://github.com/YouweiLyu/SfPUEL.

SfPUEL: Shape from Polarization under Unknown Environment Light
Youwei Lyu
Heng Guo
Kailong Zhang!
Si Li
Boxin Shi2.
Sccooo of Artificial Intelligence,
Beijing University of Posts and Telecommunications
State Key Laboratory for Multimedia Information Processing,
School of Computer Science, Peking University
National Engineering Research Center of Visual Technology,
School of Computer Science, Peking University
(youweilv, guoheng, zhangkailong HisiHHbupt.eet edu. en, shiboxin@pku. .du. cn
Abstract
Shape from polarization (SfP) benefits from advancements like polarization cam- eras for single-shot normal estimation, but its performance heavily relies on light conditions. This paper proposes SfPUEL, an end-to-end SfP method to jointly estimate surface normal and material under unknown environment light. To handle this challenging light condition, we design transformer- hancing the perception global context features. We further propose integrate photometric stereo (PS) priors from pretrained models to enrich extracted features for high-qualite normal predictions. As metallic and diclectric materials exhibit different BRDFs, SIPUEL additionally predicts dielectric and metallic material segmentation to further boost performance. Experimental results on synthetic and our collected real-world dataset demonstrate that SfPUEL significantly outperforms existing SfP and single-shot normal estimation methods. The code and dataset is available httptt::///// // /gi thub. com woom//oouveilyy//SPUEI
framework for en-
Introduction
Single-shot surface normal estimation is fundamental task in 3D reconstruction, which is widely utilized diverse fields such as robotics, graphics, and virtual reality. With the aid a snapshot polarization camera, surface normal can be obtained from single-shot polarization image recording image observations under different polarization states, also known as shape from polarization (StP). Benefiting from the passive imaging mechanism. SfP is expected work under more general conditions compared to photometric stereo which requires controlled lights. Also, the high-resolution polarization image allows SfP to recover more detailed surface shape.
Despite the prominent advantages, x and π a /-ambiiuutsss are inherent the azimuth angle estimation of surface normal from SiP (38). The r-ambiguity can be partially resolved with handcrafted priors like shape convexity (38), but handling the x /2-ambiguity related to determining specular and diffuse reflection types is more challenging. Specular and diffuse reflections can blend at any degree under divergent environment lights, as shown in Fig. [[a), which leads to noisy angle of linear polarization (AoLP) maps. Light information and object geometry entangled in polarization cues make SIP highly
Equual contribution. "orresppondig author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Environment light varistion
Polarization image imputs
SIPUEL
DeepSiP
PANDORA
Dieleetrie Metallic
Image
Aoo.P
Image
AoLP
Image
AoL.P
(a)
(b)
#Vicw-1
#Vicw-1
#Viow-35
Figure 1: (a) Rendered images and AoLP of metallic/dielectric spheres suggest SIP challenges under environment lights: AoLP maps vary with different illumination and material types. (b) SIPUEL predicts surface normal from singe-shot polarized images under unknown environment light. The visualized results show SfPUEL achieves better shape prediction over the DeepSfP 44, which is even comparable with the multiview SfP method PANDORA (15).
method
ill-posed under unknown environment light. In addition, metallic and dielectric surfaces exhibit different polarization BRDFs under the same illumination, which causes AoLP maps to vary different materials, further compounding the normal estimation problem. To avoid the light issue, previous non-learning-basee based methods make strong assumptions regarding reflection types (38) 3 36),
which hardly work under general environment light. Current state-of-the-arr methods found that specific light conditions such sunny sky (25), frontal flash illumination (16), and unknown distant light (35) can provide additional constraints to make disambiguation tractable. However, these controlled light setups undermine the passive imaging merit of SfP. In this paper, we aim to propose method for Shape from Polarization under Unknown Environment Light, abbreviated as SYPUEL.
For unknown environment lights, each point on the object's surface is lit by direct illumination and reflected light from other points, and the global light and geometry information also blends in the per-pixel polarization properties. Therefore, propose to extract the global context features by transformer- based framework, which incorporates image-level and pixel-level feature interactions.
Another key observation is the consistency between SfP and PS tasks, which inspires us to integrate PS priors enhance the representation ability of global context features. Specifically, SfP and PS both rely on measurement variations at the same pixel position across multiple images to estimate the surface normal. Therefore, off-the-shelf PS network backbones and pre-trained weights can be potentially adopted in the SfP context. On the other hand, the inputs of PS and SfP contain different physical information. PS relies on radiance variations under different light directions, while SfP utilizes radiance filtered by different polarizer angles. A polarization-relat related module is also required to extract the domain-specific features. Based on these observations, we propose a transformer by effectivvly integrating the PS prior from pretrained model and encoding polarimetric cues from polarization images. Specifically, we utilize the feature encoder from a recent PS network SDM-UniPS (26) as the PS prior extraction module (PSPEM) with fixed weights. To embed the polarization information, we make a copy of PSPEM as the polarization feature extraction module (PolFEM) with learnable weights. To integrate polarization information with PS priors, we design a novel PS and SfP feature fusion block, which takes the degree of linear polarization (DoLP) to generate the mask probability for conducting two-source masked cross attention.
For the material issue in SIP, previous approaches mostly predict surface normals of dielectric objects only (3) 4. (7). Considering the influence of materials polarization properties, propose to jointly estimate semantic material segmentation (metallic and diclectric) and surface normals from the global context features. Simultaneous supervision of the two tasks could guide the network to narrow down the latent space of global context features and boost surface normal predictions.
To validate the effectiveness of our method under different material types, we further collect real-world dataset as a complement to existing SIP datasets [ 4). Specifically, our dataset contains 4 dielectric and 2 metallic objects, which are captured under diverse environment illuminations. For quantitative evaluation, the ground truth (GT) surface normals are also calibrated. To summarize, the main contributions of this paper are follows:

---

We introduce SfPUEL, a hansformer-basee framework to solve shape from polarization under general unknown environment light via global context feature extraction;
We jointly utilize polarimetric and the representative pretrained priors from PS to solve SfP for the first time;
We solve SfP together with metallic and dielectric material segmentation, which further boosts surface normal prediction.
2
Related works
Shape from Polarization the single perspective setup, manually designed constraints, such as the surface smooth assumption (46 47) and shape convexity (38) 22 3) could narrow down the solution space SfP. Besides, incorporating shading information by formulating the imaging process of polarimetric observations as linear equations (47) or PDEs (34) also helps the disambiguation.
Introducing perspective projection in the polarization image formation model can help to determine the normal direction of plane (10). Studies show the polarization information is easily affected by light conditions (19), and the polarization property pattern under specific illumination can provide extra constraints for normal and SVBRDF optimization (8. I7 25 23 B5). To complement the affected polarization cues under different illumination and enhance the model performance, we propose transformer- based network with the aid of photometric stereo priors from pretrained models.
Shape from Polarization with Additional Measurements Polarimetric measurements could be incorporated with additional information for normal estimation with mitigated ambiguity. Coarse depth maps obtained by the ToF camera assist in predicting accurate surface normal (27). With observations from additional perspectives, the azimuth ambiguity could be effectively resolved by tangent space consistency (9) and disparities (2I). On the other hand, polarization cues also benefit multi-view stereo performance by generating accurate correspondence between two views (2) especially texture-less 113 39) and specular-dominant regions (37). Recently, a coordinate-basee neural representation approach is proposed for multi-view polarimetric inverse rendering (55).
Single-image-based Normal Estimation and Generation Surface normal estimation from single RGB image is exceedingly challenging. Handcrafted priors (29) are designed to relieve the ill-posedness of the problem. With the advent of deep learning, considering global illumination (31 and introducing inductive biases such as normal probability distributions (5) and neighboring normal relations (6) in network design helps to prevent training bias and further improve model performance. Inspired by the generative abilities of diffusion models, researchers propose generation techniques to jointly estimate depth and normal maps (18) from a single image, or generate novel views by taking single image as a visual prompt for 3D reconstruction (32) 48). Compared to generation approaches, our method could produce reliable normal predictions with better fidelity, thanks incorporating polarimetric measurements.
3 Prerequisites of Polarization
polarization image can be obtained by attaching a polarizer with angle & to a camera. The image measurement received by the camera sensor is given by
1 Imx
Imin
1max
Amin
1o
cos 2(1 +)).
(1)
in which Imax/min is the maximum/minimum intensity received by the sensor after a polarizer, and denotes the oscillation orientation of the polarized components. Stokes vectors represent the polarization status of taken images, denoted as S = [Sp. S1, S2, Sy] So represents the intensity of perceived light, and S1 S2 denote polarized components in the orientations 0" and 45", respectively. The Stokes vectors can be computed via polarization images:
S [To= + /45e Isp + I135=) /2, Ine Isos, Ia5e I1350]
(2)
Similar to previous SfP methods, we only use the first three components of Stokes vector since linear polarizers are used for polarization acquisition.
3

---

DeLF
Paletaler Eroeest
PolFEM
Ceometry
Global context features
Plesklooel
Cross
KeLF
DeLF Crost-Attetion
PulSP festures
image-lavel
MLP
erhanced featuree
PSPEM
F
Fanh
Input
Pol&PS Feature Extractor
Global Context Extractor
Output
Figure 2: Network structure of the proposed SIPUEL.
Following the PBRDF model (8) two important polarization properties can be utilized for normal estimation, i.e., AoLP and DoLP. AoLP denotes the oscillation orientation linear polarization, correlated with the azimuth angle of surface normal. DoLP is the proportion of polarization compo- nents that reflects the zenith angle of the surface normal and the material refractive index. These two polarization properties be computed using Stokes vectors:
S2 S1
V S
AoL. arctang
DoLP
2
So
Method
4.1 Network Architecture
SfPUEL aims to estimate surface normal and material segmentation under environment illumination by taking input polarization images, image intensities (Sg). AoLP and DoLP maps, and the object mask. Fig. 2 presents the framework structure of SIPUEL, which can be divided into two parts. The first part Pol&PS Feature Extractor consists of the polarization feature extraction module (PolFEM) module and the photometric stereo prior extraction module (PSPEM), which encode the information from the two fields, respectively. We propose novel DoLP cross-attention block to fuse the intermediate features from the two modules. The preceded Global Context Extractor adopts the image-level and pixel-level attention mechanisms to generate the global context features. The prediction heads finally predict material segmentation and normal vectors with two MLPs, respectively.
Pol&PS Feature Extractor SDM-UniPS 22 has demonstrated appealing performance and flexi- bility on normal estimation under natural ligh Thus, our PSPEM adopts its encoder backbone that is composed of ConvNeXt-T (33) to produce representative PS prior features at four stages with the dimensions of (96, 192, 384, 768). The four stages features are then fused with an aggregation module, i.e., UPerNet (5I). The weights of PSPEM are frozen in the training course. To obtain features from the polarimetric measurements, we make a copy PSPEM as the PolFEM backbone so that the hierarchical polarization features are aligned with the PS prior features for fusion. PolFEM encodes fcatures from individual input polarization images a share-wr weighted manner, which lacks information interactions between images. Thus, further add polarization encoding block to obtain features from DoLPP and AolP maps. The AoI.P map is encoded by (cos 21/,, sin 2t/) to be contiguous in the representation space (30). This encoding block contains six cascaded convolutional blocks and a zero convolutional layer. Each 2D convolution in the convolutional block has a x3 kernel. The zero convolutional layer consists of kernel size of and stride of I with all weights initialized as zero at the start of training. The additional encoded polarization features are fused with
We employ SDM-UniPP UniPS (6) to provide the PS priors since 1DM-UnniS 26 and our method are based on the environment light setting.

---

7
Norm
MatMul
Ziaaod
DoLP
Mask probabiliyy
Attention mask
Normalized
attention
Efficintt Addilive
MLP
Attention
Eleement wise mutiplication
K
MLP
Attention
addition
Pol images
matrix
Figure 3: The structure of the DoL. cross attention block for combining polarization and PS features, encourages the network to concentrate on polarization features with high fidelity.
features generated from the PolFEM shallow layer by addition. Pol&PS Feature Extractor produces d1 di-dimensional feature map Frois E Rkxhhwx, where k denotes the number of input polarization images and h, are height and width of the original input images, respectively.
Feature Fusion with DoLP Cross-attention Block In Pol&PS Feature Extractor, we use four DoLP cross-altention blocks to fuse the intermediate features from PolFEM and PSPEM DoLP is defined the ratio of the polarization intensity to the total light intensity. In the context of SIP, low DoLP values suggest the desired polarization signals are relatively low compared to unpolarized intensities, and these weak polarization signals be easily affected by noise in the imaging process. The network is expected to focus on polarization cues in high-fidelity regions but not fit on unreliable information. Therefore,
propose the masked cross-attention block for the spatial interaction of polarization and PS features. The structure of the DoLP cross-attention block is illustrated Fig. B DoLP is activated by exponential function attention mask. For the hierarchical feature fusion in Pol&PS Feature Extractor, we take the feature from PolFEM the key K and value V, and the feature from PSPEM the query Q. We then use the DoLP cross attention to integrate the fused features from two sources, which is given by:
get the mask probability and then generate the
Tised
Q + Norm(Mask a k,)())) V.
where o denotes the spatial interaction between the query and the key and denotes the matrix multi- plication. To ensure network efficiency, We employ the efficient additive attention mechanism (44) with linear complexity regarding the number of pixels. The Mask(A, p) operation randomly masks the elements in the attention matrix A with probability of p. (p) generates the mask probability by given the DoL.P value which is defined as:
C
(p)
(5)
ap +1'
in which the hyperparameter modulates the decay of the mask probability DoLP increases. We empirically set a to our implementation. multiplied by the normalized masked attention matrix and then added with Q to derive the fused feature.
Global Context Extractor Global Context Extractor takes features FrOps from Pol&PS Feature Extractor and produces the global context features by successively conducting image-level and pixel- level attention. The features FraPS are extracted from individual images in a share-weighted manner, without any image-level interaction, while polarization properties such as AoLP and DoL.P are calculated by the four polarization images. To enrich the extracted features in the image dimension, we pass F through the image attention block, which contains five multi-head self-attention blocks and feed-forward networks. We adopt the light-axis attention module of the SDM-UniPS (26) and its weights the image-axis attention block. After five interactions in the image axis, we acquire de- dimensional enhanced features Feuh xx To reduce the image dimension and generate the pixel-corresponte features, we introduce per-pixel geometry queries for cross attention with Fanh- The geometry query is learnable embedding like positional coding. For information interaction over the image, we use the pixel-level attention mechanism pixel-sampling Transformer (26) to extract
Fig. displays only DoLP cross-attention block just for better visualization.
5

---

Panoramic Camera
Polarization Camera
PINEAPPLEE
OWL
ROOSTER
Object
Figure 4: Capture setup and overview of our real-world dataset. The environment light map is placed at the up-right corner of each object image.
the global context features. The aggregated global context features are denoted as F&b E Rhwxds After getting Felb we adopt two two-layer MLPs with ReLU activation functions to estimate the material type logits m and the surface normal orientation ń in a per-pixel manner, respectively. We find that jointly estimating material segmentation and surface normals helps to improve the normal accuracy.
Loss Functions To supervise normal estimation, we adopt the cosine similarity loss Caormal to penalize the cosine distance between the ground truth and predicted normal vectors. The widely used cross-entropy loss Lmatteral employed for supervising two-class material type predictions. We use coefficients An and Am to balance the two loss functions in the training process. The total loss for supervising SfPUEL framework denoted as follows:
Ax nnomalll + Au Emntterial
4.2 Dataset
Synthetic Data Generation Ba et al. (4) create the SiP dataset consisting of polarization images and the ground truth normal maps, but the limited amount of data (less than 300) captured in only three different environment illumination can easily lead to transformer-bia based model overfitting. Since there is no large-scale SfP dataset under environment light available, we create a synthetic dataset by Mitsuba2 (41) to enable model training and validation. We collect 2,074 high- high-quality SVBRDFs, 799 HDR environment maps (5)),, and 244 object meshes [3 50 14] for data synthesis. We use 1,983 SVBRDFs, 651 environment maps, and 200 object meshes as the source data to generate training dataset composed of 20,000 sets of images with the resolution of 512x512, and the remaining materials are adopted for rendering 1,000 validation data with the same resolution as training ones.
During the data generation, several augmentation approaches are employed to enrich the data distribution. For augmentation of object shape, we smooth the macro-structure of the mesh model with 3 different levels using Blender and then increase the model's 's fine fineigraeedd details by applying various displacements on the mesh. For augmentation of SVBRDF, we randomly change the value of diffuse albedo in HSV space and shift and scale the UV map of the object to generate richer texture on the object. We also scale the mean of the roughness map to mimic various material appearances.
Real Data Acquisition Deschaintre et al. (16). Ngo et al. (40), and Lyu et al. (35) only provide polarization data taken under the controlled light conditions, and the test data in DeepSiP only contains dielectric objects. Therefore, we create real-world StP dataset consisting of dielectric and metallic objects. We capture polarization images under 6 different light conditions, including indoor and outdoor scenes. We use Lucid Triton RGB polarization camera and RICOH THETA SC2 to record the panoramic image as the calibrated environment light. The camera rig, objects, and the environment light are shown Fig. 4 For quantitative evaluation of SiP under environment light,
htanss//hinnaucccccco//ppotttttt hotp:piiiitioiimmmoooaaaaatttoio potpppaitttiooncameeee camera/

---

Input
S/PUEL
GT
SIPUEL
GT
Table 1: Quantitative evaluation of StP and single-image-basee methods on the synthetic data.
Angular error Mean
Accuracy (%) f
Material
Method
Median
RMSE
11.25
22.5
30.0
Accuracy
SIPW Ci
34.62
29.28
41.59
17.7
42.2
53.8
20.60
16.73
25.76
36.5
67.6
78.8
Figure 5: Qualitative evaluation of our material segmentation and normal estimation the syn- thetic data.
UNE 5
37.96
33.46
44.52
15.0
36.7
47.4
DSINE 6
20.33
17.26
24.61
34.7
69.6
80.1
STPEEL
12.39
9.449
16.57
62.7
88.2
92.8
97.8
Input
SIPW (30
DPP (4)
One-2. 2:345.3345 12
UNE (5)
DSINE 6
SYPURL
GT
25.93
13.31
NAA
30.83
26.35
13,05
52.17
33.96
N/A
48.50
26.87
12.47
Figure 6: Qualitative evaluation of our method compared to the state-of-thh-aar art SfP and linll-image based approaches on the synthetic data. The number below each result represents mean angular error.
obtain the "ground truth" normal map for each object image following the acquisition pipeline proposed in (45).
4.3
Implementation Details
Our model is implemented with PyTorch 42). We initialize PolFEM with the weights of SDM- UniPS (26) feature encoder for facilitating model training. We adopt the Adam optimizer with parameters 31 0.9, $1 0.99, and weight decay of 0.05. We set the batch size of 8 and trained the framework for 50 epochs on our large-scale synthetic dataset. The initial learning rate set to 1 x 10-4 and is halved every 10 epochs. During the training stage, We randomly crop the input images with the resolution of 512x512 to patches of 128x 128 for augmentation. The hyperparameters Am and A set to 0.1 and respectively. All experiments four NVIDIA RTX 3090 cards, where the training process takes about 40 hours.
conducted Ubuntu 20.04 LTS with
Experiments
To verify the effectiveness of SIPUEL, we compare with state-of-the-art SfP methods, where SiPW (30) is designed for scene-level normal recovery, and DeepSiP (4) performs normal esti- mation under natural illumination. We also include single-shot RGB-based surface normal estimation methods UNE (5), DSINE (6), and the generation method One-2-3-45 (22). We first compare our method with the baselines our synthetic and real-worke datasets. Then, conduct ablation studies by removing each module one after another in SfPUEL to evaluate the effectiveness our proposed method.
For quantitative evaluations, we adopt six metrics: i.e., mean and median angle error, root-mean- square error (30) (lower denotes better results), and angular accuracy percentage (higher denotes better results) between our estimated and the GT surface normals. We also evaluate material predictions using the classification accuracy percentage ("Material Accuracy", higher denotes better).
7

---

Table Quantitative evaluation on our real- data and the PANDORA (5) dataset.
Our world Datnset
PANDORA Dataset
15)
Method
Angular (*)
(%)
Angular errer (*)
Accuracy (96)
Mcan
Median
RMSE
11.25
22.5"
30.0"
Moan
Modian
RMSE
11.25
22.5
30.0"
SIPW 80
36.29
31,43
42.97
10.8
34.7
52.2
41.08
34,05
49.92
11.8
34.1
46.5
SeP
21.98
20.05
25.46
20.2
58.9
77.5
22.77
19.18
27.51
26.7
59.3
73.8
UNE 6
36.46
32.26
42.26
7.96
28.7
45.6
46.70
42.81
53.65
6.31
32.4
DSINE
21.61
19.77
25.06
28.4
65.4
77.6
17.75
15.00
21.81
38,9
74.0
85.3
SAPUEL
Input
STPW 330
DeepSfP (E)
One-2-3-45 82
UNE (5)
DSINE 6
SIPUEL
GT
43.56
26.75
N/A
31.67
14.28
7,108
41.39
24.69
20.89
N/A
32,58
16.71
9.658
Figure 7: Qualitative results of our method on real data compared to the state-of-the-art approaches. The number below each normal map represents mean angular error.
5.1 Evaluation on Synthetic Data
As shown in Fig. 5 our framework produces high-quality normal maps by taking single-view polarization images even if metallic and dielectric objects coexist in the scene. Our method also predicts reliable material labels on the pixel level. The material prediction accuracy up to 97.8% as summarized in [
We evaluate the performance of method compared two learning-based StP and four single- image-based approaches. The quantitative results are summarized on 1,000 synthetic samples including dielectric and metallic objects, listed in Table [T] The qualitative comparisons displayed Fig. (6) Our method outperforms all other approaches in both qualitative and quantitative evaluation. SIPW (30) is proposed to estimate surface normal of scene-level objects, so it produces good results on the Boat scene in Fig. 6 but cannot generalize well on Armadillo. DeepSiP (4) produces blurry normal maps, while our method could recover more satisfactory results with finer details. The
methods UNE (5) and One-2-3-45 (32) solely rely on the RGB information and generate coarse shape, losing high-frequency details. The recent normal estimation approach DSINE (6) is capable of producing generally satisfactory results but lags behind our method terms of detail recovery.
5.2 Evaluation on Real Data
To evaluate the generalization ability and performance of existing methods and SIPUEL, we utilize our real-world dataset and the 105 sets of images under 35 views of 3 objects released by PAN- DORA (I5) as the benchmark for quantitative comparisons. Since no ground-truth normal available in PANDORA (15), we take the normal maps estimated by PANDORA (15) as the reference. Quanti-

---

Tnpur
SpM-LiiPS/ipe.
w/o PolFEM
vanilla cross atin wio material est
w/o pol enc
SYPUEL
GT
O
D
D
D
D
23,38
13.81
14.01
18,69
14:29
14.70
14.63
14:44
12.99
Figure 8: Qualitative results of ablation study on KETTLE (metallic) and ROOSTER (dielectric).
tative evaluation results are provided in Table 2 on the two datasets. We also visualize qualitative results of the surface normal estimation
dataset, as shown in Fig, i
As shown in Fig. 7 SfPW (30 fails to estimate the rough shape of the objects on real data. DeepSfP and UNE (5) successfully produce the outline of the objects but rarely recover the geometric details, such the texture of OWL and PINEAPPLE surfaces. DSINE (6) generates the comparable normal prediction but struggles to get the accurate shape of reflective objects like BIRD. Our method, which takes polarization images a single view, outperforms the state-of-the-ar SIP approaches and faithfully recovers normal maps on objects with different shapes and material types.
5.3 Ablation Study
We conduct ablative experiments on our real-world dataset to analyze the importance of each design SIPUEL. The quantitative evaluation listed Table 3 and the visual results are shown Fig. g
First, we verify the consistency between SfP and PS tasks, ie., the PS model could provide useful priors for SfP tasks, which inspires
PS Priors
design the architecture of SfPUEL, We test SDM-UniPS (26) by taking as input I) only one polarization image; 2) four different polarization images. The quantitative results are listed in "SDM-UniPS w/ pol." and "SDM-UniPS w/ 4 pol." of 3 When SDM-UniPS is fed only with one polarization image, the normal result is still plausible, as shown SDDM-UniPSIppa could provide informative features for normal estimation solely from RGB information. As the number of input polarization images increases, the mean angular error of normal prediction decreases from 19.46" to 15.73 indicating that the PS network SDM-UniPS (26) is generalizable to produce reasonable normal predictions from different polarization images, and its pretrained weights can potentially boost the performance of StP.
of Fig. 8 This result suggests that the pretrained model
PolFEM Then we validate the necessity introducing PolFEM to encode the polarization feature rather than directly finetuning PSPEM in Pol&PS Feature Extractor. As shown in "w/o PolFEM", the performance of finetuning PSPEM is worse than SIPUEL, since the training process affects the PS priors from the pretrained model.
DoLP Cross-attention Block We validate the effectiveness of the DoLP-guided mask mechanism in the cross-attentiin block by removing the Mask operation. i.e., directly using the vanilla cross attention for polarization and PS feature fusion. Quantitative results in the "vanilla eross attn" row show that this strategy helps to enhance the generalization ability on the real data.
Material Segmentation Our framework simultaneously estimates material segmentation and the normal orientation at each pixel. We retrain the framework disabling the material estimation MLP, and the quantitative results are listed in the "w/o material est" row. The performance degenerates especially on the metallic KETTLE, demonstrating the benefit of incorporating material segmentation.
Injection of Additional Polarization Information In PolFEM, we add the polarization encoder to complement polarization information from computed DoLP and encoded AoLP maps. After removing this module and retaining the model, we find angular error metrics marginally increase and two accuracy indices drop on the real-world dataset. The polarization encoder has a lightweight CNN with about 0. 14M parameters, so we keep this structure to boost the performance of our framework.
Additional visualization results on our real-world dataset and the PANDORA 15] dataset are provided in the supplementary material.

---

Image
Normal
DoLP
AoLP
Table Ablation study on our real-world dataset.
Settings
Median
22.5
SDM-UliPS pol.
29.4
64.0
SDM-UIIPS wi-4 pol.
15.25
13.53
18.97
42.9
77.4
88.5
weo PolFEM
12.11
10.55
14.71
$7.0
9012
96.0
vanilia cross amn
11.86
10.14
14.79
59.3
89.8
95.7
wlo material est
12.02
10.48
14.58
56.4
89.0
96.1
11.35
Figure 9: Qualitative results of our method on the objects with rough surfaces.
SIPUEL
6 Limitations
Compared to UNE (5) and DSINE (6), our method additionally requires a mask as input to annotate object regions like previous SIP approaches DeepSIP (4) and polarNet (1).. We use Segment Anything Model (28) to generate the object mask in our experiments. In the data generation pipeline, render the synthetic dataset using unpolarized environmental maps since large-scale polarized environment light data have not yet been collected. Resorting to polarimetric measurements, StPUEL can produce appealing normal maps dielectric and metallic objects with reflective surfaces, but the performance can degenerate as polarization information gets invalid. Surfaces with complicated micro-structtrrss, e.g., rough stone and fabric, can depolarize the reflected light, where method can fail to work. As shown in Fig. [9] The DoL.P of the two objects are near zero and the AoL.P maps are noisy and less informative. The diffuse characteristics of rough sanded surfaces and fabric materials greatly mitigate the normal dependency on the polarization properties and degrade SIP performance, in line with previous studies 35). The over-exposed regions in images also affect polarization cues, which lead to artifacts in results such as estimated normals on the back of BIRD Fig.
Conclusion
This paper presents SIPUEL, an SfP framework to estimate surface normal and material segmentation from single-shot polarized images captured under unknown environment light. SfP encounters challenges of spatially varying mixed reflections under environment illumination. To handle this problem, we design transformer-base considering the consistency between SiP and PS, we propose integrating priors from PS pretrained models into the SIP task for the first time. In addition, considering the BRDF divergence of metallic and dielectric objects, we propose to jointly estimate per-pixe material types with normal vectors, which further improves normal predictions and extends our method to diverse material types. The synthetic and our collected real-world dataset including metallic and dielectric objects demonstrate that SiPUEL significantly surpasses existing methods.
framework to explore the global context features. Insightfully
Acknowledgment
This work was supported by Hebei Natural Science Foundation Project No. F2024502017. 242001012 National Science and Technology Major Project (Grant No. 20212D0109803), National Natural Science Foundation of China (Grant No. 62136001, $2088102, U23B2052, 6247072911), and Program for Youth Innovative Research Team of BUPT No. 2023YOTDO2.
References
[1] ambientCG free public domain pbr materials, https://ambient: com/ This work is licensed under Creative Commons CCO 1.0 Universal License.
[2] Gary A Atkinson and Edwin R Hancock. Multi-view surface reconstruction using polarization. In Proceedings of International Conference on Computer Vision (ICCV), 2005.
[3] Gary Atkinson and Edwin Hancock, Recovery of surface orientation from diffuse polariza-
tion. Transactions Image Processing (TIP), 2006.
10

---

[4] Yunhao Ba, Alex Gilbert, Franklin Wang, Jinfa Yang. Rui Chen, Yiqin Wang, Lei Yan. Boxin Shi, and Achuta Kadambi. Deep shape from polarization. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
[5]
Gwangbin Bac, Ignas Budvytis, and Roberto Cipolla. Estimating and exploiting the aleatoric uncertainty in surface normal estimation. In Proceedings of International Conference on Computer Vision (ICCV), 2021.
[6] Gwangbin Bae and Andrew J. Davison. Rethinking inductive biases for surface normal esti- mation. In Proceedings of Conference on Computer Vision and Pattern Recognition ( (CVPR), 2024.
[7] Seung-Hwan Baek, Tizian Zelmer, Hyunjin Ku, Inseung Hwang, Xin Tong, Wenzel Jakob, and Min H. Kim. Image-based acquisition and modeling of polarimetric reflectance. Transactions
on Graphics (TOG), 2020.
[8] Seung-Hwan Baek, Daniel S Jeon, Xin Tong, and Min H Kim. Simultaneous acquisition of polarimetric SVBRDF and normals. Transactions on Graphics (TOG), 2018.
[9] Xu Cao, Hiroaki Santo, Fumio Okura, and Yasuyuki Matsushita. Multi-view azimuth stereo via tangent space consistency. In Proceedings of Conference on Computer Vision and Pattern Recognition ( (CVPR), 2023.
[10] Guangcheng Chen, Li He, Yisheng Guan, and Hong Zhang. Perspective phase angle model for polarimetric 3d reconstruction. In Proceedings of European Conference on Computer Vision
(ECCV), 2022.
[11] Edward Collett. Field guide to polarization. Spie Bellingham, WA, 2005.
[12] Blender Online Community. Blender 3D modelling and rendering package. Blender
Foundation, 2024.
[13] Zhaopeng Cui, Jinwei Gu, Boxin Shi, Ping Tan, and Jan Kautz. Polarimetric multi-view stereo.
In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[14] Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In Proceedings of SIGGRAPHI, 1996.
[15] Akshat Dave, Yongyi Zhao, and Ashok Veeraraghavan. PANDORA: Polarization-aided neural decomposition of radiance. In Proceedings of European Conference on Computer Vision
(ECCV), 2022.
[16] Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. Single-image SVBRDF capture with rendering-aware deep network. Transactions on Gruphics (TOG), 2018.
[17] Valentin Deschaintre, Yiming Lin, and Abhijeet Ghosh. Deep polarization imaging for 3D shape and SVBRDF acquisition. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[18) Xiao Fu,, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and
Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In Proceedings of European Conference on Computer Vision (ECCV), 2024.
[19] Yoshiki Fukao, Ryo Kawahara, Shohei Nobuhara, and Ko Nishino. Polarimetric normal stereo.
In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[20] Xavier Glorot and Yoshua Bengio. Understanding the diffieulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, 2010.
[21] Tianyu Huang, Haoang Li, Kejing He, Congying Sui, Bin Li, and Yun-Hui Liu. Learning accurate 3d shape based on stereo polarimetric imaging. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2023,
[22] Cong Phuoc Huynh, Antonio Robies-Kelly. and Edwin Hancock. Shape and refractive index recovery from single-view polarisation images. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
[23] Inseung Hwang, Daniel S. Jeon, Adolfo Muñoz, Diego Gutierrez, Xin Tong, and Min H. Kim. Sparse ellipsometry: portable acquisition of polarimetric SVBRDF and shape with unstructured flash photography. Trunsactions on Graphics (TOG), 2022.
[24]
Tomoki Ichikawa, Yoshiki Fukao, Shohei Nobuhara, and Ko Nishino. Fresnel microfacet brdf: Unification of polari-radiomettii surface-body reflection. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[25] Tomoki Ichikawa, Matthew Purri, Ryo Kawahara, Shohei Nobuhara, Kristin Dana, and Ko Nishino. Shape from sky: Polarimetric normal recovery under the sky. In Proceedings of Conference Computer Vision and Patlern Recognition (CVPR), 2021.
[26] Satoshi Ikehata. Scalable, detailed and mask- free universal photometric stereo. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2023.
11

---

[27] Achuta Kadambi, Vage Taamazyan, Boxin Shi, and Ramesh Raskar. Depth sensing using geometrically constrained polarization normals. International Journal of Computer Vision (IJCV), 2017.
[28]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson,
Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In Proceedings of International Conference on Computer Vision (ICCV), 2023.
Lubor Ladicky, Bernhard Zeisl, and Mare Pollefeys. Discriminatively trained dense surface normal estimation. In Proceedings of European Conference on Computer Vision (ECCV), 2014. [30] Chenyang Lei, Chenyang Qi, Jiaxin Xie, Na Fan, Vladlen Koltun, and Qifeng Chen. Shape from polarization for complex scenes in the wild. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[29]
[31] Zhenggin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning to reconstruct shape and spatially-varying reflectance from single image. Transac- tions on Graphics (TOG), 2018.
[32] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One 2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. In Proceedings of Conference
Neural Information Processing Systems (NeurIPS), 2023. [33] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining
Xie. A convnet for the 2020s. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[34] Fotios Logothetis, Roberto Mecca, Fiorella Sgallari, and Roberto Cipolla. A differential approach to shape from polarisation: A level-set characterisation. International Journal of Computer Vision (IJCV), 2019.
[35] Youwei Lyu, Lingran Zhao, Si Li, and Boxin Shi. Shape from polarization with distant lighting estimation. Transactions on Pattern Analysis and Machine Intellgence (TPAMI), 2023.
[36] Ali H Mahmoud, Moumen T El-Melegy, and Aly A Farag. Direct method for shape recovery from polarization and shading. In Proceedings of International Conference Image Processing, 2012.
[37] Daisuke Miyazaki, Ryosuke Furuhashi, and Shinsaku Hiura. Shape estimation of concave
specular object from multiview polarization, Journal of Electronic Imaging, 2020.
[38] Daisuke Miyazaki, Masataka Kagesawa, and Katsushi Ikeuchi. Polarization- based transparent surface modeling from two views. In Proceedings of International Conference on Computer Vision (ICCV). 2003.
[39] Daisuke Miyazaki, Takuya Shigetomi, Masashi Baba, Ryo Furukawa, Shinsaku Hiura, and Naoki Asada. Surface normal estimation of black specular objects from multiview polarization images. Optical Engineering. 2016.
[40] Trung Thanh Ngo, Hajime Nagahara, and Rin ichiro Taniguchi. Surface normals and light directions from shading and polarization. Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021.
[41] Merlin Davie,, Delio Vicini, Tizian Zelmer, and Wenzel Jakob. Mitsuba A retargetable forward and inverse renderer. Transactions on Graphics (TOG), 2019.
[42] Adam Paszke. Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Proceedings of Conference on Neural Information Processing Systems (NeurIPS), 2019.
[43] Poly Haven - the public 3D asset library. https://polyyhaven. com/ This work is licensed under Creative Commons CCO 1.0 Universal License.
[44] AbdeIrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming- Hsuan Yang, and Fahad Shahbaz Khan. Swiftformer: Efficient additive attention for rranformme-bbaee real-time mobile vision applications. In Proceedings of International Conference on Computer
Vision (ICCV), 2023.
[45] Boxin Shi, Zhipeng Mo, Zhe Wu, Dinglong Duan, Sai-Kit Yeung, and Ping Tan. benchmark dataset and evaluation for non-lambertian and uncalibrated photometric stereo. Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019.
[46] William AP Smith, Ravi Ramamoorthi, and Silvia Tozza. Linear depth estimation from an uncalibrated, monocular polarisation image. In Proceedings of European Conference on Computer Vision (ECCV), 2016.
12

---

[47] William AP Smith, Ravi Ramamoorthi, and Silvia Tozza. Height-from Height-from-polaaisttoonn with unknown lighting albedo. Transactions on Pattern Analysis and Machine Intelligence
(TPAMI). 2018.
[48] Shitao Tang, Jiacheng Chen, Dilin Wang, Chengzhou Tang, Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, and Rakesh Ranjan. Mydiffusion+++: A dense high-resolution multi-view diffusion model for single to sparse-view 3d objjectreconstruction.. In Proceedings of European Conference on Computer Vision (ECCV), 2024.
[49] Shoji Tominaga and Tetsuya Yamamoto. Metal-dielectric object classification by polarization degree map. International Conference on Pattern Recognition, 2008
[50] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang. Chen Qian, Dahua Lin, and Ziwei Liu. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, Dataset: https Omn Object3D SubbjcttD-New/eeee Beae-Neewexpplle wewp/epplre/maaleeeeeeee
//openxlab.org org. wwtccc//atta Peattasets/Open//ee
[51] lete Xıao, Yinghheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unihed perceptual parsing for scene understanding. In Proceedings of European Conference on Computer Vision (ECCV),
2018.
[52] Zhengyou Zhang. A flexible new technique for camera calibration. Transactions Pattern Analysis and Machine Intelligence (TPAMI), 2000.
13

---

Supplementary Material
A Difference between Dielectric and Metallic Materials
We separate the material type into dielectric and metallic, due to polarization properties tightly correlated to these two material types. Polarimetric BRDFs (4) derived from Fresnel equations, and Fresnel analysis is also the main technique to distinguish dielectric from metallic materials. Early works (49 estimated dielectric/mettllic types by polarization information, which insight us to categorize material as the two types.
The difference between dielectric and metallic materials can be analyzed via the polarimetric BRDF (pBRDF) (8). The Fresnel terms F (0.: and F* (Oa; 17) in the pBRDF are depend on the refractive index (RI) v (8). is a real number dielectric materials, while it is complex number consisting of imaginary part denoted as the extinction coefficient (EC) for metallic materials (Τ
We provide visual comparisons between di- electric and metallic materials, as shown in Fig. TO Fig- 10 displays three synthetic spheres with (a) dielectric surface with white diffuse albedo; (b) a dielectric surface with black diffuse albedo; (c) metallic sur- face made of chromium. The three spheres have the same RI (in real number) and rough- ness and are rendered under the same illu- mination. The white dielectric sphere in Fig. [0]a] a) differs from the metallic sphere in image appearance and angle of linear po- larization (AoLP) distribution. The black dielectric sphere in Fig. bT)b) has simi- lar reflective appearance and an AoLP map the metallic sphere in Fig. [0[e). but the
(c)
Image
DoLP
90
AoLP
1900
degree of linear polarization (DoLP) of the dielectric sphere much higher than that of the metallic sphere. The differences in AoLP patterns and DoLP magnitudes can guide the diclectric/mettllcc material seg- mentation. This also why we categorize materials as dielectric and metallic and intro- duce material segmentation boost normal estimation.
Dielectric
Dielectric Odiff = 0
Metallic
Odiff = 1
EC=3.3
RI=3.2
RI=3.2
RI=3.2
Figure 10: Visual comparisons of appearance and po- larization propertics between dielectric and metallic spheres. Quir denotes diffuse albedo, RI and EC measured at the wavelength of 587.6 nm.
B Efficiency of Network Initialization with SDM-UniPS Weights
We initialize the Polarization Feature Extraction module (PolFEM) and the image-level attention module in Global Context Extractor with the pretrained weights from SDM-UniPS (26). To evaluate the impact of network initialization on framework performance, we initialized these two modules of SPPUEL with Xavier initialization (20) and trained the framework with the same strategy discussed Sec. 4.3 of the main paper. We find that the network struggled to converge even after being trained for over 80 epochs on the synthetic dataset. It suggests that initializing SfPUEL with the pretrained weights from SDM greatly facilitates the training process.
C Ground Truth Normal Acquisition
In our real dataset, polarization images and the ground truth normal maps of 6 objects are provided for quantitative evaluation. We acquire the GT normal maps following the guideline (45). We use EinScan-SP V2 SPECS Desktop 3D Scanner to scan the objects and generate the object meshes.
14

---

The six objects in our dataset and the scanned meshes are displayed in Fig. II We calibrate the polarization camera get camera intrinsic parameters (52), then conduct the image-mesh alignment to get the camera extrinsic parameters, and finally render the "ground truth" normal in Blender (12).
Figure 11: Six objects in our dataset and the corresponding scanned meshes.
D Material Estimation Results
To further validate SIPUEL on material estimation, we provide more results on synthetic and real- world data, as shown in Fig. and Fig. respectively.
Input
SIPUEL material
GT material
SIPUEL normal
GT normal
Figure 12: Material estimation of our method the synthetic data, where red denotes dielectric material and green denotes metallic material.
Image
Material
Figure 13: Material segmentation results of our method on the real-world data. Green and red denote metallic and dielectric materials, respectively.
15

---

E Model Size and Inference Time
We compare the model size (#Param) and test running time of the state-of-the-ar methods (i.e., SfPW (30), DeepSfP (4) UNE (5), DSINE (6), and One-2-3-45 (32) and our model. The test time of each method calculated by processing a single test sample with a resolution 512, and these experiments are conducted on the same device (Ubuntu 20.04 LTS with an NVIDIA RTX 3090 card), The results are listed in Table One-2-3-45 (32) has the most parameters and takes the longest time in the inference stage. The test time of our method is slightly longer than other single-shot-buse methods since the current model has not been optimized for computational efficiency. Adopting advanced lightweight attention mechanisms like efficient additive attention (44) in Global Context Extractor may help to reduce our model's computation complexity.
Table 4: Model size and computational costs comparisons.
Method
SfPW (30)
DeepSfP (4)
One-2-3-45 (32)
UNE (5)
DSINE 6 SIPUEL
Paaam 42.5M
10.8M
1.29G
72.4M
72.6M .4238
141M
Test time
.571s
1.06s
136s
.319s
1.61s
F Normal Estimation on Real Data
In the main paper, we display the normal predictions of SIPUEL on 4 objects compared to the state-of-the-ant methods. In this section, provide normal results the rest two objects in Fig. [4] In addition, we compare SIPUEL to PANDORA A (15), the multi-view 3D reconstruction method taking polarization images, well as SfP (30 4. 3D generation approach One-2-3-45 (32), and single-image-based approaches (5) (6 on the real data released by (15). The qualitative results are shown Fig. [5| Our method outperforms previous SiP and single-shot normal estimation approaches. Taking as imput single-view polarization images, SIPUEL also produces comparable results against the multi-view method (5). Moreover, Fig. I6 provides an additional qualitative
evaluation on four more objects to show the generalization ability of our method.
Input
SIPW (50
DeepSIP 4
One-2-3-45 S0
UNE (5)
DSINE 6
SYPUEL
GT
D
D
D
D
27.46
23.76
N/A
38.60
16.86
14.01
21 21.99
N/A
37,50
18:40
12.99
Figure 14: Qualitative results of our method on real data compared the state-of-the-art approaches. The number below each normal map represents mean angular error.
G Network Details
In this section, we introduce more details about the SIPUEL network. SIPUEL consists of two main parts: Pol&PS Feature Extractor and Global Context Extractor. Pol&PS Feature Extractor takes as input angle of linear polarization (AoLP) and degree of linear polarization (DoLP) maps, image intensities, polarization images, and the mask, which has two parallel branches: the polarization feature extraction module (PolFEM) and the photometric stereo prior extraction module (PSPEM), PolFEM and PSPEM produce features corresponding to individual input images in shared-weight manner. The backbone PolFEM has the same structure as that of PSPEM, and ConvNeXi-T (3) adopted as the image encoder in the two branches. Pyramid Pooling Module (PPM) of UPerNet (51) used for fusing hierarchical features from Image Encoder. In PolFEM, we propose to extract features directly from polarization properties using the polarization encoder. For efficient feature
16

---

Tnpur
SFPW (30
DeepSiP 4
One-2-3-45 b2
UNE (
DSINE 6
SIPUBL
PANDORA (TS)
54,02
18.13
N/A
49.45
12.37
10.65
42.55
19:25
NIA
38.34
15.81
15.69
42.06
53.89
29.17
20.45
Figure 15: Visual results of our method against previous approaches, including SIPW (30),
DeepSfP (4) One-2-3-45 (32), UNE (5), DSINE (6) and PANDORA (15). The number below each result denotes mean angular error.
Image
Normal
Figure 16: Qualitative evaluation of our method on additional four real-world objects.
fusion between PolFEM and PSPEM, we introduce the DoLP cross-attention block PSPEM. The polarization features encoded from PolFEM are taken as the query, and the PS features from PSPEM are taken as the key and the value the cross-attention block. After two-source feature fusion, the extracted features FPOPS from Pol&PS Feature Extractor are fed to 5 cascaded image-level self- attention blocks. The image-axis self-attention block has a vanilla transformer structure composed of multi-head self-attention blocks, layer normalization, and feed-forward networks, producing image-level enhanced feature Feah- Then, Fenh are sampled spatially, and we use cross-attention to query per-pixel features and conduct pixel-level self-attention to generate the global context features. Finally, the global features are fed into two MLPs to predict normal vectors and material logits.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main contributions and the scope of the proposed method have been elaborated in the abstract and introduction.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations of our method regarding the model input and the application scope at the end of Sec.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We have elaborated all details regarding the proposed network in Sec. 4.1 We additionally provide more details about the submodules in the network ensure our work can be reproduced.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have included the URL the data and code of this paper in the abstract.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have provided the implementation details including hyperparameters, optimizer type, and training details in Sec. 4.3
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have evaluated our method with different error matrices, which are sufficient to demonstrate the statistical significance of the experiments.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
18

---

Answer: [Yes]
Justification: We have provided the hardware information to conduct all the experiments in our paper, and the model complexity and execution time reported in the supplementary material.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research in the paper conform with the NeurIPS Code of Ethics in every respect.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of this work.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses risks for misuse.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have listed the license names of the used assets and cited related papers.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have provided the link to the dataset related this paper.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human
Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects
19


Reviews:
[{"review_id": "7DPU7ipruI", "reviewer": "Reviewer_66Sa", "summary": "This paper tries to solve normal estimation tasks using a single image captured from a polarization camera under unknown environmental light. To handle unknown environmental light conditions, this paper adopted a deep learning-based strategy to take advantage of inductive bias from the training dataset. Compared with previous methods, this paper additionally utilizes a pre-trained photometric stereo network and material segmentation tasks. The network architecture generally follows the Transformer and is trained with the synthetic dataset created by the authors. The proposed method is evaluated on the six real-world objects and shows reasonably good results compared with the GT normal map. It also shows results comparable to the multi-view SfP method (PANDORA).", "questions": "1. It is unclear how polarization images can be treated as the photometric stereo input. The only theoretical reason I found it on the L42-44, both SfP and PS estimate the result of the same pixel position, with measurement under different conditions. Yet, PS usually assumes the difference in lighting condition, and SfP assumes the difference in polarization direction. Are there any more specific analysis for this? Besides, will it work if we replace SDM-UniPS with other PS networks and follow a similar training strategy?\n2. Related to the second weakness, what makes the difference (e.g., the complex-valued IOR), and how is the difference observed (in both RGB and polarization domain) between dielectric and conductor material? To make the paper more self-contained and concrete, an explanation about this seems necessary.\n3. Also, the following questions are related to the practicability of the method.\n- This method also estimates the material segmentation, but the results for real data’s material segmentation are missing. Are there any reasons for excluding this?\n- Since trained synthetic data consists of dielectric and conductor material, can this method deal with diffuse objects? Also, objects cannot be classified into simple dielectric or conductor in the real world—for instance, multi-layered materials or coated materials. What happens if we use the proposed method for these objects? Such limitations should be clearly stated, and a single statement in the Limitations section does not seem enough.\n- It seems the method assumes the environment light is unpolarized. Since there is also a polarized light source, is it considered in the training dataset?\n4. Is there any plan to make the training dataset and full test dataset publicly available? Such effort to collect datasets will be helpful for future research.", "limitations": "No potential negative social impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper shows that utilizing the pre-trained photometric stereo network can help the single-shot estimation of normal from the polarization images, which can be shown as a novelty of the paper. The paper also proposes some architectures that can be applied to the polarization images, such as the DoLP cross-attention block. The authors also captured some synthetic and real datasets, which can help future research if they are opened. Also, regardless of technical novelty, the quality of the results seems fairly good on both synthetic and real datasets.", "weaknesses": "The proposed network seems to rely strongly on the photometric stereo network SDM-UniPS. It not only borrows its architecture but also adopts network weight for extracting the photometric stereo features. Such dependency on the specific network can be treated as a weakness. Also, refer to the first question, which includes the question about SDM-UniPS.\n\nIt is unclear why the BSDF type is separated into only two: dielectric and metallic (conductor). The authors provided examples and short statements but didn’t explain why they are different. This makes it weak to add the material segmentation tasks only for two types.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed and constructive suggestions. Below, we address the concerns raised by Reviewer 66Sa.\n\n> Q1.1 Analysis of photometric stereo (PS) and shape-from-polarization (SfP)\n\nPlease refer to *Further analysis of photometric stereo taking as input polarization images* in the global responses for the detailed discussion.\n\n> Q1.2 Can SDM-UniPS be replaced by other PS networks?\n\nYes, if the PS network could work under unknown environment light. More specifically, the network structure of SDM-UniPS can effectively extract global light features from images under the same viewpoint but varying irradiance, which is essential to handle the challenging shape estimation under unknown environment light. To be more generalized, any effective feature extractor can replace SDM-UniPS. In this paper, we adopt the off-the-shelf SDM-UniPS network backbone and the pre-trained weight due to its effective global feature for handling unknown environment light.\n\n> Q2 Why the BSDF type is separated into dielectric and metallic? What is the difference between dielectric and metallic materials?\n\nWe separate the material type into dielectric and metallic, due to polarization properties tightly correlated to these two material types. Polarimetric BRDFs [Baek et al. 2018; Ichikawa et al. 2023] are derived from Fresnel equations, and Fresnel analysis is also the main technique to distinguish dielectric from metallic materials. Early works [Tominaga and Yamamoto 2008] estimated dielectric/metallic types by polarization information, which insight us to categorize material as the two types.\n\nThe difference between dielectric and metallic materials can be analyzed via the polarimetric BRDF (pBRDF) [Baek et al. 2018]. The Fresnel terms $\\textbf{F}^{\\text{T}}{\\text{o}}(\\theta_o;\\eta)$ and $\\textbf{F}^{\\text{R}}(\\theta_d;\\eta)$ in the pBRDF are depend on the refractive index (RI) $\\eta$ [Baek et al. 2018]. $\\eta$ is a real number for dielectric materials, while it is a complex number consisting of an imaginary part denoted as the extinction coefficient (EC) for metallic materials [Collett 2005]. \n\nAs suggested, we provide visual comparisons between dielectric and metallic materials, as shown in Fig. **II** of the attached PDF file. Fig. **II** displays three synthetic spheres with (a) a dielectric surface with white diffuse albedo; (b) a dielectric surface with black diffuse albedo; (c) a metallic surface made of chromium. The three spheres have the same RI (in real number) and roughness and are rendered under the same illumination. The white dielectric sphere in Fig. **II(a)** differs from the metallic sphere in image appearance and angle of linear polarization (AoLP) distribution. The black dielectric sphere in Fig. **II(b)** has a similar reflective appearance and an AoLP map to the metallic sphere in Fig. **II(c)**, but the degree of linear polarization (DoLP) of the dielectric sphere is much higher than that of the metallic sphere. The differences in AoLP patterns and DoLP magnitudes can guide the dielectric/metallic material segmentation. This is also why we categorize materials as dielectric and metallic and introduce material segmentation to boost normal estimation. \n\nWe will add this discussion and the visual comparisons to the final version of our paper.\n\n> Q3.1 Including material segmentation results on real data\n\nAs suggested, we provide more material segmentation results on the real data, as shown in Fig. **III** of the attached PDF file. Our method produces plausible results on objects with dielectric surfaces, metallic surfaces, and surfaces with both material types, but we observe a failure case on the metallic kettle, where the material segmentation result is dielectric. We will add these material segmentation results in the final version. \n\n> Q3.2 Applications on more complex materials (e.g., diffuse, multi-layered, or coated materials) \n\nTo the best of our knowledge, the multi-layered and coated materials haven’t been addressed in SfP tasks, which is still an open problem due to the complex reflectance. We respectfully think it is out of the scope of this paper because we mainly focus on SfPUEL, i.e., shape from polarization under unknown environment light. \n\nFor diffuse objects, we test our method on a rough stone turtle and a fabric pillow as shown in Fig. **IV** of the attached PDF file. The DoLP of the two objects are near zero and the AoLP maps are noisy and less informative. The diffuse characteristics of rough sanded surfaces and fabric materials greatly mitigate the normal dependency on the polarization properties and degrade SfP performance, as stated by previous studies [Baek et al. 2020; Lyu et al. 2023]. As a result, our method fails to produce reliable normal maps on these two diffuse surfaces given invalid polarization cues. We will add this discussion in the final version of our paper.\n\n> Q3.3 Polarized light sources\n\nPlease refer to *Unpolarized environment light in training data* in the global responses.\n\n> Q4 Dataset release plan\n\nWe will release all the training & test datasets and the implementation code upon acceptance.\n\n**References** \n\n[Baek et al. 2018] Simultaneous Acquisition of Polarimetric SVBRDF and Normals. Baek et al. TOG, 2018.\n\n[Baek et al. 2020] Image-Based Acquisition and Modeling of Polarimetric Reflectance. Baek et al. 2020. TOG, 2020.\n\n[Collett 2005] Field Guide to Polarization. Collett 2005. SPIE.\n\n[Ichikawa et al. 2023] Fresnel Microfacet BRDF: Unification of Polari-Radiometric Surface-Body Reflection. Ichikawa et al. CVPR, 2023.\n\n[Ikehata 2022] Universal Photometric Stereo Network using Global Contexts. Satoshi Ikehata. CVPR, 2022.\n\n[Lyu et al. 2023] Shape from Polarization with Distant Lighting Estimation. Lyu et al. TPAMI, 2023.\n\n[Tominaga and Yamamoto 2008] Metal-dielectric object classification by polarization degree map. Tominaga and Yamamoto. ICPR, 2008.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bkdLnSXDc6", "reviewer": "Reviewer_7r7M", "summary": "This paper addresses Shape from Polarization under Unknown Environment Light (SfPUEL) from a single polarimetric image. Existing SfP methods have the ambiguity of surface normal caused by unknown illumination and materials and make some assumptions on reflection type or illumination. This paper introduces a novel SfP framework based on a transformer that considers the global context. This paper also proposes to combine SfP with pre-trained photometric stereo (PS) priors by using cross-attention based on DoLPs. For guidance to resolve the ambiguity by materials, the network outputs segmentation of dielectric and metallic materials along with surface normals. The experimental results show that the proposed method quantitatively and qualitatively outperforms the SOTA normal estimation methods using a single image or polarimetric image.", "questions": "- Since the proposed method only requires a single polarimetric image (and its mask), is it possible to capture real-world data more casually if GT is not acquired? If so, testing a wider range of real-world data can validate the generalizability of the proposed method qualitatively.", "limitations": "- As mentioned in Weakness, the proposed method cannot handle a specular dominant surface under polarized illumination.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "+ A novel transformer-based framework for SfP effectively constrains the surface normal by considering the global context and exploiting pre-trained PS priors with DoLP cross-attention.\n+ This paper proposes a joint estimation of material segmentation and surface normals to resolve the ambiguity caused by materials.\n+ The reconstructed normals are significantly better than the SOTA normal estimation methods.", "weaknesses": "- Since photometric stereo inherently requires multiple images captured under different lighting conditions, the pre-trained PS model is not supposed to take polarimetric images under the same lighting conditions as the input. This would lead to unexpected behavior and make it unclear how generalizable the features extracted from the pre-trained PS model are.\n- A polarized light source like a sunny sky affects the polarization of reflected light [20], especially for a specular dominant surface, such as a black dielectric surface and smooth metallic surface. Since the authors seem to create synthetic data from unpolarized environment maps, the proposed learning-based method cannot handle such cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful reviews and valuable questions. Below, we address the concerns raised by Reviewer 7r7M.\n\n> W1: Photometric stereo methods take as input polarization images\n\nAs suggested, we further verify the photometric stereo network taking as input polarization images by qualitative and quantitative experiments, as shown in Fig. **I** and Table **I** of the attached PDF file. Please refer to *Further analysis of photometric stereo taking as input polarization images* in the global responses.\n\n> W2: Using unpolarized light sources in dataset creation\n\nPlease refer to *Unpolarized environment light in training data* of the global responses for the detailed discussion.\n\n> Q: Qualitative evaluation on more real-world data\n\nThanks for the suggestion. We provide an additional qualitative evaluation on four more objects to show the generalizability of our method. The results are shown in Fig. **V** of the attached PDF file. Our method also generates plausible normal maps on the additional four objects. We will add this qualitative evaluation to the supplementary material in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YRfzoNkxzT", "reviewer": "Reviewer_bz5V", "summary": "This paper proposes SfPUEL, which estimates surface normals and material (metallic or dielectric) under unknown environmental light using a single polarization image. The proposed network integrates polarization information with photometric stereo priors using a photometric stereo feature extractor and a polarization feature extractor with DoLP cross-attention. By jointly estimating material segmentation and surface normals from the global context features, it helps predict surface normals and materials.", "questions": "The paper is well-written, and the network architecture is described well in both the main and supplemental papers. The published code also helps to address any further questions.", "limitations": "The author addressed the following limitations:\n\n1. Requires an object mask\n2. Supports dielectric and metallic objects\n\nI might add that this paper is also limited to normal and material estimation only. Given the normal and material components, future research could focus on estimating material appearance parameters.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* Synthetic and real polarization datasets for various materials and environmental maps.\n* A novel network architecture consisting of:\n1. Polarization and photometric stereo feature extraction\n2. DoLP cross-attention block\n3. Global context extractor\n* Reasonable ablation study (Table 3) with sufficient SOTA results compared to various normal estimation methods (Figures 5, 6, 7, 8 and Tables 1, 2).", "weaknesses": "* There are no comparison results with other material segmentation methods.\n* The method does not clearly show high-frequency, spatially-varying material examples, such as dielectric surfaces with dense strip patterns using metallic surfaces. Each object is composed of a single material only.\n* Closely related to the limitation, which is beyond the scope of the paper, there is no consideration of appearance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive reviews and questions. Below, we address the concerns raised by Reviewer bz5V.\n\n> W1:  Comparisons with other material segmentation methods.\n\nMaterial segmentation in our paper is only for boosting the performance of normal estimation, we do not intend to make the accuracy of material segmentation one of our contributions. As our focus is normal estimation under unknown environment light, adding the comparison of material segmentation could blur our focus. We will tone down the corresponding description of material segmentation in the paper. \n\n> W2:  Results on objects with more complex materials\n\nWe provide additional experiments on newly captured real data, containing spatially varying albedo and material types. The results have been provided in Fig. **III** and Fig. **V** of the attached PDF file, showing that our method can handle complex spatially varying materials. These experiments will be added in the final version.\n\n> Q1: Material appearance estimation & object mask input\n\nThis paper mainly focuses on SfPUEL, i.e., shape from polarization under the challenging **unknown environment light**. While jointly estimating normal, material, and appearance parameters is also an intriguing topic but out of the scope of this work. Also, our method requires an object mask as input, but it is not difficult to obtain the mask with the help of powerful SAM [Kirillov et al. 2023] nowadays. \n\n**Reference**\n\n[Kirillov et al. 2023] Segment anything. Kirillov et al. ICCV 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful and valuable comments. We are encouraged by reviewers’ positive comments: “This paper proposes a “novel” framework for shape-from-polarization” (Reviewers **7r7M** and **bz5V**); “the quality of the results seems fairly good” (Reviewer **66Sa**) and “significantly better than the SOTA normal estimation methods”  (Reviewer **7r7M**); “reasonable” ablation study with “sufficient SOTA results” has been conducted  (Reviewer **bz5V**).

Below, we address the common concerns raised by the reviewers. **The attached PDF file** contains experimental results suggested by reviewers:

- Qualitative and quantitative evaluation of SDM-UniPS taking as input different numbers of polarization images to further validate the effectiveness of a PS network using polarization images (Fig. **I** and Table **I**, as suggested by Reviewer **66Sa** and **7r7M**).
- Visual comparison of dielectric and metallic spheres on appearance and polarization properties to show the difference between two types of materials (Fig. **II**, as suggested by Reviewer **66Sa**).
- Material segmentation on the real-world data and more scenes (Fig. **III**, suggested by Reviewer **66Sa** and **bz5V**).
- Qualitative results of our method on diffuse objects for illustrating the method’s limitation (Fig. **IV**, suggested by Reviewer **66Sa**).
- More qualitative results of our method on real-world data (Fig. **V**, suggested by Reviewer **7r7M** and **bz5V**)

# Global Responses

## Further analysis of photometric stereo taking as input polarization images

We admit that photometric stereo (PS) and shape-from-polarization (SfP) methods are derived from different physical models. However, from the perspective of end-to-end network learning, both PS and SfP networks extract high-level features from multiple images with the same viewpoint but varying pixel radiance; the extracted features are decoded to predict the normal map. 

To further validate that the pre-trained PS network could produce reasonable features by taking as input polarization images, we conduct the experiments to compare the normal predictions of SDM-UniPS fed with $n$ different polarization images ($n\in\\{1,2,3,4\\}$), as shown in Fig. **I** and Table **I** of the attached PDF file. As the number of input polarization images increases from 1 to 4, the mean angular error of normal prediction on the real dataset decreases from 19.46$^\circ$ to 15.73$^\circ$, which indicates that the PS network is generalizable to produce plausible predictions and intermediate features for the SfP task, given the variations between polarization images. We will add this experiment in the supplementary material.

## Unpolarized environment light in training data

We agree that environment light is mostly polarized in real-world scenes. To render large-scale polarization images under polarized environmental light as a training dataset, polarized environment maps are essential. However, large-scale polarized environment maps have not yet been collected. Therefore, we create the large-scale dataset under unpolarized environmental light. Nevertheless, experimental results of normal estimation on real-world data show that our method is robust against polarized environmental light, given that real-world environmental light can indeed be polarized. We will add this discussion in the revised version.

## Material segmentation in our method

The main purpose of material segmentation in our method is to boost the performance of normal estimation. Our method focuses on surface normal estimation under unknown environment light rather than material segmentation. We are insighted by the observation that different material types can lead to different polarimetric measurements. So, we introduce material segmentation in our method to improve the normal predictions but do not make material segmentation accuracy one of the contributions. We will revise our paper to reduce the emphasis on material segmentation as a contribution of the paper to prevent any potential misunderstandings by the readers.

Meta-review: The authors made a comprehensive responses and finally the paper received three (all) accept recommendations. The area chair agreed with this recommendation.
Decision: Accept (poster)