SfPUEL: Shape from Polarization under Unknown Environment Light

Authors: youwei  lyu
heng  guo
kailong  zhang
si  li
boxin  shi
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 14:38:40

Abstract: Shape from polarization (SfP) benefits from advancements like polarization cameras for single-shot normal estimation, but its performance heavily relies on light conditions. This paper proposes SfPUEL, an end-to-end SfP method to jointly estimate surface normal and material under unknown environment light. To handle this challenging light condition, we design a transformer-based framework for enhancing the perception of global context features. We further propose to integrate photometric stereo (PS) priors from pretrained models to enrich extracted features for high-quality normal predictions. As metallic and dielectric materials exhibit different BRDFs, SfPUEL additionally predicts dielectric and metallic material segmentation to further boost performance. Experimental results on synthetic and our collected real-world dataset demonstrate that SfPUEL significantly outperforms existing SfP and single-shot normal estimation methods. The code and dataset is available at https://github.com/YouweiLyu/SfPUEL.

Main paper: 
 introduction: 
 Single-shot surface normal estimation is a fundamental task in 3D reconstruction, which is widely utilized in diverse fields such as robotics, graphics, and virtual reality. With the aid of a snapshot polarization camera, surface normal can be obtained from a single-shot polarization image recording image observations under different polarization states, also known as shape from polarization (SfP). Benefiting from the passive imaging mechanism, SfP is expected to work under more general conditions compared to photometric stereo which requires controlled lights. Also, the high-resolution polarization image allows SfP to recover a more detailed surface shape.Despite the prominent advantages, π and π/2-ambiguities are inherent in the azimuth angle estimation of surface normal from SfP (38) . The π-ambiguity can be partially resolved with handcrafted priors like shape convexity (38) , but handling the π/2-ambiguity related to determining specular and diffuse reflection types is more challenging. Specular and diffuse reflections can blend at any degree under divergent environment lights, as shown in Fig. 1 (a), which leads to noisy angle of linear polarization (AoLP) maps. Light information and object geometry entangled in polarization cues make SfP highly show SfPUEL achieves a better shape prediction over the state-of-the-art method DeepSfP (4), which is even comparable with the multiview SfP method PANDORA (15) .ill-posed under unknown environment light. In addition, metallic and dielectric surfaces exhibit different polarization BRDFs under the same illumination, which causes AoLP maps to vary on different materials, further compounding the normal estimation problem. To avoid the light issue, previous non-learning-based methods make strong assumptions regarding reflection types (38; 3; 36) , which hardly work under general environment light. Current state-of-the-art methods found that specific light conditions such as sunny sky (25) , frontal flash illumination (16) , and an unknown distant light (35) can provide additional constraints to make disambiguation tractable. However, these controlled light setups undermine the passive imaging merit of SfP. In this paper, we aim to propose a method for Shape from Polarization under Unknown Environment Light, abbreviated as SfPUEL.For unknown environment lights, each point on the object's surface is lit by direct illumination and reflected light from other points, and the global light and geometry information also blends in the per-pixel polarization properties. Therefore, we propose to extract the global context features by a transformer-based framework, which incorporates image-level and pixel-level feature interactions.Another key observation is the consistency between SfP and PS tasks, which inspires us to integrate PS priors to enhance the representation ability of global context features. Specifically, SfP and PS both rely on measurement variations at the same pixel position across multiple images to estimate the surface normal. Therefore, off-the-shelf PS network backbones and pre-trained weights can be potentially adopted in the SfP context. On the other hand, the inputs of PS and SfP contain different physical information. PS relies on radiance variations under different light directions, while SfP utilizes radiance filtered by different polarizer angles. A polarization-related module is also required to extract the domain-specific features. Based on these observations, we propose a transformer by effectively integrating the PS prior from a pretrained model and encoding polarimetric cues from polarization images. Specifically, we utilize the feature encoder from a recent PS network SDM-UniPS (26) as the PS prior extraction module (PSPEM) with fixed weights. To embed the polarization information, we make a copy of PSPEM as the polarization feature extraction module (PolFEM) with learnable weights. To integrate polarization information with PS priors, we design a novel PS and SfP feature fusion block, which takes the degree of linear polarization (DoLP) to generate the mask probability for conducting two-source masked cross attention.For the material issue in SfP, previous approaches mostly predict surface normals of dielectric objects only (3; 4; 17) . Considering the influence of materials on polarization properties, we propose to jointly estimate semantic material segmentation (metallic and dielectric) and surface normals from the global context features. Simultaneous supervision of the two tasks could guide the network to narrow down the latent space of global context features and boost surface normal predictions.To validate the effectiveness of our method under different material types, we further collect a real-world dataset as a complement to existing SfP datasets (15; 4) . Specifically, our dataset contains 4 dielectric and 2 metallic objects, which are captured under diverse environment illuminations. For quantitative evaluation, the ground truth (GT) surface normals are also calibrated. To summarize, the main contributions of this paper are as follows:• We introduce SfPUEL, a transformer-based framework to solve shape from polarization under general unknown environment light via global context feature extraction; • We jointly utilize polarimetric cue and the representative pretrained priors from PS to solve SfP for the first time; • We solve SfP together with metallic and dielectric material segmentation, which further boosts surface normal prediction. 
related works: 
 Shape from Polarization In the single perspective setup, manually designed constraints, such as the surface smooth assumption (46; 47) and shape convexity (38; 22; 3) could narrow down the solution space of SfP. Besides, incorporating shading information by formulating the imaging process of polarimetric observations as linear equations (47) or PDEs (34) also helps the disambiguation.Introducing perspective projection in the polarization image formation model can help to determine the normal direction of a plane (10) . Studies show the polarization information is easily affected by light conditions (19) , and the polarization property pattern under specific illumination can provide extra constraints for normal and SVBRDF optimization (8; 17; 25; 23; 35) . To complement the affected polarization cues under different illumination and enhance the model performance, we propose a transformer-based network with the aid of photometric stereo priors from pretrained models.Polarimetric measurements could be incorporated with additional information for normal estimation with mitigated ambiguity. Coarse depth maps obtained by the ToF camera assist in predicting accurate surface normal (27) . With observations from additional perspectives, the azimuth ambiguity could be effectively resolved by tangent space consistency (9) and disparities (21) . On the other hand, polarization cues also benefit multi-view stereo performance by generating accurate correspondence between two views (2) especially on texture-less (13; 39) and specular-dominant regions (37) . Recently, a coordinate-based neural representation approach is proposed for multi-view polarimetric inverse rendering (15) .Single-image-based Normal Estimation and Generation Surface normal estimation from a single RGB image is exceedingly challenging. Handcrafted priors (29) are designed to relieve the ill-posedness of the problem. With the advent of deep learning, considering global illumination (31) and introducing inductive biases such as normal probability distributions (5) and neighboring normal relations (6) in network design helps to prevent training bias and further improve model performance.Inspired by the generative abilities of diffusion models, researchers propose generation techniques to jointly estimate depth and normal maps (18) from a single image, or generate novel views by taking a single image as a visual prompt for 3D reconstruction (32; 48) . Compared to generation approaches, our method could produce reliable normal predictions with better fidelity, thanks to incorporating polarimetric measurements. 
prerequisites of polarization: 
 A polarization image can be obtained by attaching a polarizer with angle ϑ to a camera. The image measurement received by the camera sensor is given byEQUATIONin which I max/min is the maximum/minimum intensity received by the sensor after a polarizer, and ψ denotes the oscillation orientation of the polarized components. Stokes vectors represent the polarization status of taken images, denoted as S = [S 0 , S 1 , S 2 , S 3 ] ⊤ . S 0 represents the intensity of perceived light, and S 1 , S 2 denote polarized components in the orientations of 0 • and 45 • , respectively. The Stokes vectors can be computed via polarization images:EQUATIONSimilar to previous SfP methods, we only use the first three components of a Stokes vector since linear polarizers are used for polarization acquisition. I j R V 0 W B H F Z w T 6 g H U o m v W 1 D M 5 k h u S O W s Q t / x Y 0 L R d z 6 G + 7 8 G 9 P H Q l s P B A 7 n 3 M s 9 O W E i h U H P + 3 a W l l d W 1 9 Z z G / n N r e 2 d X X d v v 2 b i V H O o 8 l j G u h E y A 1 I o q K J A C Y 1 E A 4 t C C f V w c D X 2 6 / e g j Y j V H Q 4 T C C L W U 6 I r O E M r t d 3 D V s S w z 5 n M r k f t F s I D Z q D 6 o 7 Z b 8 I r e B H S R + D N S I D N U 2 u 5 X q x P z N A K F X D J j m r 6 X Y J A x j Y J L G O V b q Y G E 8 Q H r Q d N S x S I w Q T b J P 6 I n V u n Q b q z t U 0 g n 6 u + N j E X G D K P Q T o 7 T m n l v L P 7 n N V P s X g a Z U E m K o P j 0 U D e V F G M 6 L o N 2 h A a O c m g J 4 1 r Y r J T 3 m W Y c b W V 5 W 4 I / / + V F U j s r + u f F 0 m 2 p U K a z O n L k i B y T U + K T C 1 I m N 6 R C q o S T R / J M XF x K i l r 0 1 j E s u c T x Q S P W B s 4 C N Z L J C O h L 1 j X H 1 8 V f v e e S c X j 6 A 4 m C X N D M o x 4 w C k B L X n m o R M S G F E i s u v c c 4 A 9 Q D Y U f u 6 Z N a t u T Y E X i V 2 S G i r R 8 s w v Z x D T N G Q R U E G U 6 t t W A m 5 G J H A q W F 5 1 U s U S Q s d k y P q a R i R k y s 2 m + X N 8 o p U B D m K p X w R 4 q v 7 e y E i o 1 C T 0 9 W S R V s 1 7 h f i f 1 0 8 h u H Q z H i U p s I j O D g W p w B D j o g w 8 4 J J R E B N N C J V c Z 8 V 0 R C S h o C uCon vNe Xt-T back bon e DoLP Cross-Attention Following the pBRDF model ( 8), two important polarization properties can be utilized for normal estimation, i.e., AoLP and DoLP. AoLP denotes the oscillation orientation of linear polarization, correlated with the azimuth angle of surface normal. DoLP is the proportion of polarization components that reflects the zenith angle of the surface normal and the material refractive index. These two polarization properties can be computed using Stokes vectors: Pol&PS Feature Extractor SDM-UniPS (26) has demonstrated appealing performance and flexibility on normal estimation under natural light1 . Thus, our PSPEM adopts its encoder backbone that is composed of ConvNeXt-T (33) to produce representative PS prior features at four stages with the dimensions of (96, 192, 384, 768) . The four stages of features are then fused with an aggregation module, i.e., UPerNet (51) . The weights of PSPEM are frozen in the training course. To obtain features from the polarimetric measurements, we make a copy of PSPEM as the PolFEM backbone so that the hierarchical polarization features are aligned with the PS prior features for fusion. PolFEM encodes features from individual input polarization images in a share-weighted manner, which lacks information interactions between images. Thus, we further add a polarization encoding block to obtain features from DoLP and AoLP maps. The AoLP map is encoded by (cos 2ψ, sin 2ψ) to be contiguous in the representation space (30) . This encoding block contains six cascaded convolutional blocks and a zero convolutional layer. Each 2D convolution in the convolutional block has a 3×3 kernel. The zero convolutional layer consists of a kernel size of 1 and a stride of 1 with all weights initialized as zero at the start of training. The additional encoded polarization features are fused with The network is expected to focus on polarization cues in high-fidelity regions but not fit on unreliable information. Therefore, we propose the masked cross-attention block for the spatial interaction of polarization and PS features. The structure of the DoLP cross-attention block is illustrated in Fig. 3 . DoLP is activated by an exponential function to get the mask probability and then generate the attention mask. For the hierarchical feature fusion in Pol&PS Feature Extractor, we take the feature from PolFEM as the key K and value V , and the feature from PSPEM as the query Q. We then use the DoLP cross attention to integrate the fused features from two sources, which is given by:Aggregation Module K,V K,V K,V K,V Q Q Q Q Con vNeY j E = " > A A A C A X i c b V D L S s N A F J 3 U V 6 2 v q B v B z W A R X J V E i r o s C O I y o n 1 A E 8 p k O m 2 H T h 7 M 3 I g l x I 2 / 4 s a F I m 7 9 C 3 f + j Z M 2 C 2 0 9 c O F w z r 3 c e 4 8 f C 6 7 A s r 6 N 0 t L y y u p a e b 2 y s b m 1 v W P u 7 r V U l E j K m j Q S k e z 4 R D H B Q 9 Y E D o J 1 Y s l I 4 A v W 9 s e X u d + + Z 1 L x K L y D S c y 8 g A x D P u C U g J Z 6 5 o E b E B h R I t K r r O c C e 4 D U i Y R z m / X M q l W z p s C L x C 5 I F R V w e u a X 2 4 9 o E r A Q q C B K d W 0 r B i 8 l E j g V L K u 4 i W I x o W M y Z F 1 N Q x I w 5 a X T D z J 8 r J U + H k R S V w h 4 q v 6 e S E m g 1 C T w d W d + r 5 r 3 c v E / r 5 v A 4 M J L e R g n w E I 6 W z R I B I Y I 5 3 H g P p e M g p h o Q q j k + l Z M R 0 Q S C j q 0 i g 7 B n n 9 5 k b R O a / Z Z r X 5 T r z Z w E U c Z H a I j d I J s d I 4 a 6 B o 5 q I k o e k T P 6 B W 9 G U / G i / F u f M x a S 0 Y x s 4 / + w P j 8 A S M K l z w = < / l a t e x i t > F PolPS < l a t e x i t s h a 1 _ b a s e 6 4 = " U x G Z X h L Z M s m 7 9 c 9 L G d m c U X K / B j U = " > A A A B 7 H i c j V B N S 8 N A E J 3 U r 1 q / q l 4 E L 4 t F 8 F Q S K d V j w Y v H i q Y t t K F s t p N 2 6 W Y T d j d C C f 0 N X j w o 4 t U f 5 M 1 / 4 / b j o K L g g 4 H H e z P M z A t T w b V x 3 Q + n s L K 6 t r 5 R 3 C x t b e / s 7 p X 3 D 1 o 6 y R R D n y U i U Z 2 Q a h R c o m + 4 E d h J F d I 4 F N g O x 1 c z v 3 2 P S v N E 3 p l J i k F M h 5 J H n F F j J T + / n f b d f r n i V d 0 5 y N + k A k s 0 + + X 3 3 i B h W Y z S M E G 1 7 n p u a o K c K s O Z w G m p l 2 l M K R v T I X Y t l T R G H e T z Y 6 f k 1 C o D E i X K l j R k r n 6 d y G m s 9 S Q O b W d M z U j / 9 G b i b 1 4 3 M 9 F l k H O Z Z g Y l W y y K M k F M Q m a f k w F X y I y Y W E K Z 4 v Z W w k Z U U W Z s P q X / h d A 6 r 3 r 1 a u 2 m V m k c L e M o w j G c w B l 4 c A E N u I Y m + M C A w w M 8 w b M j n U f n x X l d t BEQUATIONE x T H D F W s B B s G 6 i G Z G h Y J 1 w f D f 3 O x O m D Y / V I 0 w T F k g y V H z A K Q E r B f 6 Q S E m q P o 1 i u O y X K 2 7 N X Q C v E y 8 n F Z S j 2 S 9 / + V F M U 8 k U U E G M 6 X l u A k F G N H A q 2 K z k p 4 Y l h I 7 J k P U s V U Q y E 2 S L o 2 f 4 w i o R H s T a l g K 8 U H 9 P Z E Q a M 5 W h 7 Z Q E R m b V m 4 v / e b 0 U B r d B x l W S A l N 0 u W i Q C g w x n i e A I 6 4 Z B T G 1 h F D N 7 a 2 Y j o g m F G x O J R u C t / r y O m l f 1 b z r W vx W M S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 3 d z v P K H S P J Y P Z p q g H 9 G R 5 C F n 1 F i p 2 R y U K 2 7 V X Y C s E y 8 n F c j R G J S / + s O Y p R F K w w T V u u e 5 i f E z q g x n A m e l f q o x o W x C R 9 i z V N I I t Z 8 t D p 2 R C 6 s M S R g r W 9 K Q h f p 7 I q O R 1 t M o s J 0 R N W OV Q x b L B a x 6 g R U o + A S W 4 Y b g Z 1 E I Y 0 C g Q / B + G b m P z y h 0 j y W 9 2 a S o B / R o e Q h Z 9 R Y q X n X L 1 f c q j s H W S V e T i q Q o 9 E v f / U G M U s j l I Y J q n X X c x P j Z 1 Q Z z g R O S 7 1 U Y 0 L Z m A 6 x a 6 m k E W o / m x 8 6 J W d W G Z A w V r a k I X P 1 9 0 R G I 6 0 n U W A 7 I 2 p G e t m b i f 9 5 3 d S E 1 3 7 G Z Z I a l G y x K E w F M T G Z f U 0 G X C E z Y m I J Z YF R + f i k r e N U M W y x W M S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 3 d z v P K H S P J Y P Z p q g H 9 G R 5 C F n 1 F i p 2 R 6 U K 2 7 V X Y C s E y 8 n F c j R G J S / + s O Y p R F K w w T V u u e 5 i f E z q g x n A m e l f q o x o W x C R 9 i z V N I I t Z 8 t D p 2 R C 6 s M S R g r W 9 K Q h f p 7 I q O R 1 t M o s J 0 R N W O 9 6 s 3 F / 7 x e a s J b P + M y S Q 1 K t l w U p o K Y m M y / J k O u k B k x t Y Q y x= " > A A A B 7 X i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B U 9 k t R T 0 W v H i s Y D + g X U o 2 z b a x 2 S Q k W a E s / Q 9 e P C j i 1 f / j z X 9 j 2 u 5 B W x 8 M P N 6 b Y W Z e p D g z 1 v e / v c L G 5 t b 2 T n G 3 t L d / c H h U P j 5 p G 5 l q Q l t E c q m 7 E T a U M 0 F b l l l O u 0 p T n E S c d q L J 7 d z v P F F t m B Q P d q p o m O C R Y D E j 2 D q p 3 Z e K p 2 Z Q r v h V f w G 0 T o K c V C B H c 1 D + 6 g 8 l S R M q L O H Y m F 7 g K x t m W F t G O J 2 V + q m h C p M J H t G e o w I n 1 I T Z 4 t o Z u n D K E M V S u x I W L d T f E x l O j J k m k e t M s B 2 b V W 8 u / u f 1 U h v f h B k T K r V U k O W i O O X I S j R / H Q 2 Z p s T y q S O Y a O Z u R W S M N S b W B V R y I Q S r L 6 + T d q 0 a X F X r 9 / V K o 5 b H U Y Q z O I d L C O A a G n A H T W g B g U dM S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 3 d z v P K H S P J Y P Z p q g H 9 G R 5 C F n 1 F i p 2 R y U K 2 7 V X Y C s E y 8 n F c j R G J S / + s O Y p R F K w w T V u u e 5 i f E z q g x n A m e l f q o x o W x C R 9 i z V N I I t Z 8 t D p 2 R C 6 s M S R g r W 9 K Q h f p 7 I q O R 1 t M o s J 0 R N W Of + N s s o U m H h g 4 n H N f c 7 x Y C o 2 2 / W 0 V V l b X 1 j e K m 6 W t 7 Z 3 d v f L + Q U t H i W K 8 y S I Z q Y 5 H N Z c i 5 E 0 U K H k n V p w G n u R t b 3 y V + e 0 H r r S I w l u c x N w N 6 D A U v m A U j X R / 1 + 8 h f 8 L U T z Q f T P v l i l 2 1 Z y D L x M l J B X I 0 + u W v 3 i B i S c B D Z J J q 3 X X s G N 2 U K h R M 8 m m p Z 8 b G l I 3 p k H c N D W n A t Z v O r p 6 S k 2 w n 8 S N l X o h k p v 7 u S G m g 9 S T w T G V A c a Q X v U z 8 z + s m 6 F + 6 q Q j j B H n I 5 o v 8 R B K M S B Y B G Q j F G c q J I Z Q p Y W 4 l b E Q V Z W i C K p k Q n M U vQ l t E c q m 7 E T a U M 0 F b l l l O u 0 p T n E S c d q L J 7 d z v P F F t m B Q P d q p o m O C R Y D E j 2 D q p 3 Z e K p 2 Z Q r v h V f w G 0 T o K c V C B H c 1 D + 6 g 8 l S R M q L O H Y m F 7 g K x t m W F t G O J 2 V + q m h C p M J H t G e o w I n 1 I T Z 4 t o Z u n D K E M V S u x I W L d T f E x l O j J k m k e t M s B 2 b V W 8 u / u f 1 U h v f h B k T K r V U k O W i O O X I S j R / H Q 2 Z p s T y q S O Y a O Z u R W S M N S b W B V R y I Q S r L 6 + T d q 0 a X F X r 9 / V K o 5 b H U Y Q z O I d L C O A a G n A H T W g B g U dEQUATIONwhere • denotes the spatial interaction between the query and the key and • denotes the matrix multiplication. To ensure network efficiency, We employ the efficient additive attention mechanism (44) with linear complexity regarding the number of pixels. The Mask(A, p) operation randomly masks the elements in the attention matrix A with a probability of p. γ(ρ) generates the mask probability by given the DoLP value ρ, which is defined as:EQUATIONin which the hyperparameter α modulates the decay rate of the mask probability as DoLP increases.We empirically set α to 3 in our implementation. V is multiplied by the normalized masked attention matrix and then added with Q to derive the fused feature. After getting F glb , we adopt two two-layer MLPs with ReLU activation functions to estimate the material type logits m and the surface normal orientation n in a per-pixel manner, respectively. We find that jointly estimating material segmentation and surface normals helps to improve the normal accuracy.Loss Functions To supervise normal estimation, we adopt the cosine similarity loss L normal to penalize the cosine distance between the ground truth and predicted normal vectors. The widely used cross-entropy loss L material is employed for supervising two-class material type predictions. We use coefficients λ n and λ m to balance the two loss functions in the training process. The total loss for supervising SfPUEL framework is denoted as follows: During the data generation, several augmentation approaches are employed to enrich the data distribution. For augmentation of object shape, we smooth the macro-structure of the mesh model with 3 different levels using Blender and then increase the model's fine-grained details by applying various displacements on the mesh. For augmentation of SVBRDF, we randomly change the value of diffuse albedo in HSV space and shift and scale the UV map of the object to generate richer texture on the object. We also scale the mean of the roughness map to mimic various material appearances. we obtain the "ground truth" normal map for each object image following the acquisition pipeline proposed in (45).EQUATION 
implementation details: 
 Our model is implemented with PyTorch (42) . We initialize PolFEM with the weights of SDM-UniPS (26) feature encoder for facilitating model training. We adopt the AdamW optimizer with parameters β 1 = 0.9, β 1 = 0.99, and weight decay of 0.05. We set the batch size of 8 and trained the framework for 50 epochs on our large-scale synthetic dataset. The initial learning rate is set to 1×10 -4 and is halved every 10 epochs. During the training stage, We randomly crop the input images with the resolution of 512×512 to patches of 128×128 for augmentation. The hyperparameters λ m and λ n are set to 0.1 and 1, respectively. All experiments are conducted on Ubuntu 20.04 LTS with four NVIDIA RTX 3090, where the training process takes about 40 hours. 
experiments: 
 To verify the effectiveness of SfPUEL, we compare with state-of-the-art SfP methods, where SfPW ( 30) is designed for scene-level normal recovery, and DeepSfP (4) performs normal estimation under natural illumination. We also include single-shot RGB-based surface normal estimation methods UNE (5), DSINE (6) , and the generation method One-2-3-45 (32) . We first compare our method with the baselines on our synthetic and real-world datasets. Then, we conduct ablation studies by removing each module one after another in SfPUEL to evaluate the effectiveness of our proposed method.For quantitative evaluations, we adopt six metrics: i.e., mean and median angle error, root-meansquare error (30) (lower denotes better results), and angular accuracy percentage (higher denotes better results) between our estimated and the GT surface normals. We also evaluate material predictions using the classification accuracy percentage ("Material Accuracy", higher denotes better). The number below each normal map represents mean angular error.As shown in Fig. 5 , our framework produces high-quality normal maps by taking single-view polarization images even if metallic and dielectric objects coexist in the scene. Our method also predicts reliable material labels on the pixel level. The material prediction accuracy is up to 97.8% as summarized in Table 1 .We evaluate the performance of our method compared to two learning-based SfP and four singleimage-based approaches. The quantitative results are summarized on 1,000 synthetic samples including dielectric and metallic objects, as listed in Table 1 . The qualitative comparisons are displayed in Fig. 6 . Our method outperforms all other approaches in both qualitative and quantitative evaluation. SfPW ( 30) is proposed to estimate surface normal of scene-level objects, so it produces good results on the Boat scene in Fig. 6 but cannot generalize well on Armadillo. DeepSfP (4) produces blurry normal maps, while our method could recover more satisfactory results with finer details. The single-image-based methods UNE (5) and One-2-3-45 (32) solely rely on the RGB information and generate a coarse shape, losing high-frequency details. The recent normal estimation approach DSINE (6) is capable of producing generally satisfactory results but lags behind our method in terms of detail recovery.To evaluate the generalization ability and performance of existing methods and SfPUEL, we utilize our real-world dataset and the 105 sets of images under 35 views of 3 objects released by PAN-DORA (15) as the benchmark for quantitative comparisons. Since no ground-truth normal is available in PANDORA (15) , we take the normal maps estimated by PANDORA (15) as the reference. Quanti- tative evaluation results are provided in Table 2 on the two datasets. We also visualize qualitative results of the surface normal estimation on our dataset, as shown in Fig. 7 4 .As shown in Fig. 7 , SfPW (30) fails to estimate the rough shape of the objects on real data. DeepSfP (4) and UNE (5) successfully produce the outline of the objects but rarely recover the geometric details, such as the texture of OWL and PINEAPPLE surfaces. DSINE (6) generates the comparable normal prediction but struggles to get the accurate shape of reflective objects like BIRD. Our method, which takes polarization images in a single view, outperforms the state-of-the-art SfP approaches and faithfully recovers normal maps on objects with different shapes and material types.We conduct ablative experiments on our real-world dataset to analyze the importance of each design in SfPUEL. The quantitative evaluation is listed in Table 3 and the visual results are shown in Fig. 8 .PS Priors First, we verify the consistency between SfP and PS tasks, i.e., the PS model could provide useful priors for SfP tasks, which inspires us to design the architecture of SfPUEL. We test SDM-UniPS ( 26) by taking as input 1) only one polarization image; 2) four different polarization images. The quantitative results are listed in "SDM-UniPS w/ 1 pol." and "SDM-UniPS w/ 4 pol." of Table 3 . When SDM-UniPS is fed only with one polarization image, the normal result is still plausible, as shown in 'SDM-UniPS/1pol." of Fig. 8 . This result suggests that the pretrained model could provide informative features for normal estimation solely from RGB information. As the number of input polarization images increases, the mean angular error of normal prediction decreases from 19.46 • to 15.73 • , indicating that the PS network SDM-UniPS ( 26) is generalizable to produce reasonable normal predictions from different polarization images, and its pretrained weights can potentially boost the performance of SfP.PolFEM Then we validate the necessity of introducing PolFEM to encode the polarization feature rather than directly finetuning PSPEM in Pol&PS Feature Extractor. As shown in "w/o PolFEM", the performance of finetuning PSPEM is worse than SfPUEL, since the training process affects the PS priors from the pretrained model.We validate the effectiveness of the DoLP-guided mask mechanism in the cross-attention block by removing the Mask operation, i.e., directly using the vanilla cross attention for polarization and PS feature fusion. Quantitative results in the "vanilla cross attn" row show that this strategy helps to enhance the generalization ability on the real data.Material Segmentation Our framework simultaneously estimates material segmentation and the normal orientation at each pixel. We retrain the framework disabling the material estimation MLP, and the quantitative results are listed in the "w/o material est" row. The performance degenerates especially on the metallic KETTLE, demonstrating the benefit of incorporating material segmentation.In PolFEM, we add the polarization encoder to complement polarization information from computed DoLP and encoded AoLP maps. After removing this module and retaining the model, we find angular error metrics marginally increase and two accuracy indices drop on the real-world dataset. The polarization encoder has a lightweight CNN with about 0.14M parameters, so we keep this structure to boost the performance of our framework.  
limitations: 
 Compared to UNE ( 5) and DSINE (6), our method additionally requires a mask as input to annotate object regions like previous SfP approaches DeepSfP (4) and polarNet (17) . We use Segment Anything Model (28) to generate the object mask in our experiments. In the data generation pipeline, we render the synthetic dataset using unpolarized environmental maps since large-scale polarized environment light data have not yet been collected. Resorting to polarimetric measurements, SfPUEL can produce appealing normal maps on dielectric and metallic objects with reflective surfaces, but the performance can degenerate as polarization information gets invalid. Surfaces with complicated micro-structures, e.g., rough stone and fabric, can depolarize the reflected light, where our method can fail to work. As shown in Fig. 9 , The DoLP of the two objects are near zero and the AoLP maps are noisy and less informative. The diffuse characteristics of rough sanded surfaces and fabric materials greatly mitigate the normal dependency on the polarization properties and degrade SfP performance, in line with previous studies (7; 35) . The over-exposed regions in images also affect polarization cues, which lead to artifacts in results such as estimated normals on the back of BIRD in Fig. 7 . 
conclusion: 
 This paper presents SfPUEL, an SfP framework to estimate surface normal and material segmentation from single-shot polarized images captured under unknown environment light. SfP encounters challenges of spatially varying mixed reflections under environment illumination. To handle this problem, we design a transformer-based framework to explore the global context features. Insightfully considering the consistency between SfP and PS, we propose integrating priors from PS pretrained models into the SfP task for the first time. In addition, considering the BRDF divergence of metallic and dielectric objects, we propose to jointly estimate per-pixel material types with normal vectors, which further improves normal predictions and extends our method to diverse material types. The synthetic and our collected real-world dataset including metallic and dielectric objects demonstrate that SfPUEL significantly surpasses existing methods.-Supplementary Material - 
 
 Appendix: We separate the material type into dielectric and metallic, due to polarization properties tightly correlated to these two material types. Polarimetric BRDFs (8; 24) are derived from Fresnel equations, and Fresnel analysis is also the main technique to distinguish dielectric from metallic materials. Early works (49) estimated dielectric/metallic types by polarization information, which insight us to categorize material as the two types.The difference between dielectric and metallic materials can be analyzed via the polarimetric BRDF (pBRDF) (8) . The Fresnel terms F T o(θ o ; η) and F R (θ d ; η) in the pBRDF are depend on the refractive index (RI) η (8). η is a real number for dielectric materials, while it is a complex number consisting of an imaginary part denoted as the extinction coefficient (EC) for metallic materials (11) . 10(c ), but the degree of linear polarization (DoLP) of the dielectric sphere is much higher than that of the metallic sphere. The differences in AoLP patterns and DoLP magnitudes can guide the dielectric/metallic material segmentation. This is also why we categorize materials as dielectric and metallic and introduce material segmentation to boost normal estimation.We initialize the Polarization Feature Extraction module (PolFEM) and the image-level attention module in Global Context Extractor with the pretrained weights from SDM-UniPS (26). To evaluate the impact of network initialization on framework performance, we initialized these two modules of SfPUEL with Xavier initialization (20) and trained the framework with the same strategy discussed in Sec. 4.3 of the main paper. We find that the network struggled to converge even after being trained for over 80 epochs on the synthetic dataset. It suggests that initializing SfPUEL with the pretrained weights from SDM greatly facilitates the training process.In our real dataset, polarization images and the ground truth normal maps of 6 objects are provided for quantitative evaluation. We acquire the GT normal maps following the guideline of (45) . We use EinScan-SP V2 SPECS Desktop 3D Scanner to scan the objects and generate the object meshes.The six objects in our dataset and the scanned meshes are displayed in Fig. 11 . We calibrate the polarization camera to get camera intrinsic parameters (52) , then conduct the image-mesh alignment to get the camera extrinsic parameters, and finally render the "ground truth" normal in Blender (12) .Figure 11 : Six objects in our dataset and the corresponding scanned meshes.To further validate SfPUEL on material estimation, we provide more results on synthetic and realworld data, as shown in Fig. 12 and Fig. 13 , respectively. We compare the model size (#Param) and test running time of the state-of-the-art methods (i.e., SfPW (30) , DeepSfP (4), UNE (5), DSINE (6) , and One-2-3-45 (32) ) and our model. The test time of each method is calculated by processing a single test sample with a resolution of 512, and these experiments are conducted on the same device (Ubuntu 20.04 LTS with an NVIDIA RTX 3090). The results are listed in Table 4 . One-2-3-45 has the most parameters and takes the longest time in the inference stage. The test time of our method is slightly longer than other single-shot-based methods since the current model has not been optimized for computational efficiency. Adopting advanced lightweight attention mechanisms like efficient additive attention (44) in Global Context Extractor may help to reduce our model's computation complexity. In the main paper, we display the normal predictions of SfPUEL on 4 objects compared to the state-of-the-art methods. In this section, we provide normal results on the rest two objects in Fig. 14 .In addition, we compare SfPUEL to PANDORA (15) , the multi-view 3D reconstruction method taking polarization images, as well as SfP (30; 4), 3D generation approach One-2-3-45 (32) , and single-image-based approaches (5; 6) on the real data released by (15) . The qualitative results are shown in Fig. 15 . Our method outperforms previous SfP and single-shot normal estimation approaches. Taking as input single-view polarization images, SfPUEL also produces comparable results against the multi-view method (15) . Moreover, Fig. 16 provides an additional qualitative evaluation on four more objects to show the generalization ability of our method. We tabulate the detailed structures of some SfPUEL modules in Table 5 . 51) is used for fusing hierarchical features from Image Encoder. In PolFEM, we propose to extract features directly from polarization properties using the polarization encoder, whose structure is We employ SDM-UniPS(26) to provide the PS priors since SDM-UniPS(26) and our method are based on the environment light setting.Fig.2displays only one DoLP cross-attention block just for better visualization.https://thinklucid.com/product/triton-5-mp-polarization-camera/Additional visualization results on our real-world dataset and the PANDORA(15) dataset are provided in the supplementary material. 


Decision: The authors made a comprehensive responses and finally the paper received three (all) accept recommendations. The area chair agreed with this recommendation.

Reviews: review: paper_summary: paper_summary: This paper tries to solve normal estimation tasks using a single image captured from a polarization camera under unknown environmental light. To handle unknown environmental light conditions, this paper adopted a deep learning-based strategy to take advantage of inductive bias from the training dataset. Compared with previous methods, this paper additionally utilizes a pre-trained photometric stereo network and material segmentation tasks. The network architecture generally follows the Transformer and is trained with the synthetic dataset created by the authors. The proposed method is evaluated on the six real-world objects and shows reasonably good results compared with the GT normal map. It also shows results comparable to the multi-view SfP method (PANDORA).
strength_weakness: strength_weakness: The proposed network seems to rely strongly on the photometric stereo network SDM-UniPS. It not only borrows its architecture but also adopts network weight for extracting the photometric stereo features. Such dependency on the specific network can be treated as a weakness. Also, refer to the first question, which includes the question about SDM-UniPS.

It is unclear why the BSDF type is separated into only two: dielectric and metallic (conductor). The authors provided examples and short statements but didn’t explain why they are different. This makes it weak to add the material segmentation tasks only for two types.
questions: questions: 1. It is unclear how polarization images can be treated as the photometric stereo input. The only theoretical reason I found it on the L42-44, both SfP and PS estimate the result of the same pixel position, with measurement under different conditions. Yet, PS usually assumes the difference in lighting condition, and SfP assumes the difference in polarization direction. Are there any more specific analysis for this? Besides, will it work if we replace SDM-UniPS with other PS networks and follow a similar training strategy?
2. Related to the second weakness, what makes the difference (e.g., the complex-valued IOR), and how is the difference observed (in both RGB and polarization domain) between dielectric and conductor material? To make the paper more self-contained and concrete, an explanation about this seems necessary.
3. Also, the following questions are related to the practicability of the method.
- This method also estimates the material segmentation, but the results for real data’s material segmentation are missing. Are there any reasons for excluding this?
- Since trained synthetic data consists of dielectric and conductor material, can this method deal with diffuse objects? Also, objects cannot be classified into simple dielectric or conductor in the real world—for instance, multi-layered materials or coated materials. What happens if we use the proposed method for these objects? Such limitations should be clearly stated, and a single statement in the Limitations section does not seem enough.
- It seems the method assumes the environment light is unpolarized. Since there is also a polarized light source, is it considered in the training dataset?
4. Is there any plan to make the training dataset and full test dataset publicly available? Such effort to collect datasets will be helpful for future research.
limitations: limitations: No potential negative social impacts.
paper_summary: paper_summary: This paper addresses Shape from Polarization under Unknown Environment Light (SfPUEL) from a single polarimetric image. Existing SfP methods have the ambiguity of surface normal caused by unknown illumination and materials and make some assumptions on reflection type or illumination. This paper introduces a novel SfP framework based on a transformer that considers the global context. This paper also proposes to combine SfP with pre-trained photometric stereo (PS) priors by using cross-attention based on DoLPs. For guidance to resolve the ambiguity by materials, the network outputs segmentation of dielectric and metallic materials along with surface normals. The experimental results show that the proposed method quantitatively and qualitatively outperforms the SOTA normal estimation methods using a single image or polarimetric image.
strength_weakness: strength_weakness: - Since photometric stereo inherently requires multiple images captured under different lighting conditions, the pre-trained PS model is not supposed to take polarimetric images under the same lighting conditions as the input. This would lead to unexpected behavior and make it unclear how generalizable the features extracted from the pre-trained PS model are.
- A polarized light source like a sunny sky affects the polarization of reflected light [20], especially for a specular dominant surface, such as a black dielectric surface and smooth metallic surface. Since the authors seem to create synthetic data from unpolarized environment maps, the proposed learning-based method cannot handle such cases.
questions: questions: - Since the proposed method only requires a single polarimetric image (and its mask), is it possible to capture real-world data more casually if GT is not acquired? If so, testing a wider range of real-world data can validate the generalizability of the proposed method qualitatively.
limitations: limitations: - As mentioned in Weakness, the proposed method cannot handle a specular dominant surface under polarized illumination.
paper_summary: paper_summary: This paper proposes SfPUEL, which estimates surface normals and material (metallic or dielectric) under unknown environmental light using a single polarization image. The proposed network integrates polarization information with photometric stereo priors using a photometric stereo feature extractor and a polarization feature extractor with DoLP cross-attention. By jointly estimating material segmentation and surface normals from the global context features, it helps predict surface normals and materials.
strength_weakness: strength_weakness: * There are no comparison results with other material segmentation methods.
* The method does not clearly show high-frequency, spatially-varying material examples, such as dielectric surfaces with dense strip patterns using metallic surfaces. Each object is composed of a single material only.
* Closely related to the limitation, which is beyond the scope of the paper, there is no consideration of appearance.
questions: questions: The paper is well-written, and the network architecture is described well in both the main and supplemental papers. The published code also helps to address any further questions.
limitations: limitations: The author addressed the following limitations:

1. Requires an object mask
2. Supports dielectric and metallic objects

I might add that this paper is also limited to normal and material estimation only. Given the normal and material components, future research could focus on estimating material appearance parameters.

Comments: title: Further comments？
comment: I appreciate the authors for the detailed rebuttal and for providing the extra results. My questions and concerns are properly answered, and there appear to be no significant technical problems. Thus, I will increase my ratings and recommend acceptance.
I really appreciate the author's additional results, which show realistic outcomes with complex materials on real datasets. There are no significant issues within the discussions to change the existing rating, so I will keep my current rating, recommended for acceptance.
I appreciate your additional discussions and experimental results that show the effectiveness of the proposed method. I keep my rating and recommend the acceptance of this paper.
We are looking forward to further discussions with the reviewers during the author-reviewer period. Thanks again for the reviewers' insightful comments and interest.

By the way, we also would like to further explain whether SDM-UniPS can be replaced by another PS network in our method. Most existing PS methods work under directional lights, as listed in the DiLiGenT10$^2$ benchmark [Ren et al. 2022], meaning that the image capture should be conducted in a darkroom. Considering SfP under unknown environment light, an alternative PS network to SDM-UniPS in our model should (1) work under unknown environment light and (2) effectively utilize variations between polarization images for normal estimation as verified in Fig. **I** and Table **I** of the attached PDF file. Currently, SDM-UniPS is the one satisfying the above requirements.

**Reference**

[Ren et al. 2022] DiLiGenT10$^2$: A Photometric Stereo Benchmark Dataset with Controlled Shape and Material Variation. Ren et al. CVPR 2022.