Learning to dehaze with polarization

1 Introduction . When taking photos in hazy environments , the visibility and color fidelity of recorded scenes are usually contaminated , because the captured images often contain a superposition of two unknown components : the transmitted light ( an attenuated fraction of original scene radiance ) , and the airlight ( ambient light scattered towards the viewer ) . It is highly ill-posed to separate them in a single hazy image as it requires estimating multiple unknowns from a single observation . Handcrafted priors from natural image statistics [ 21 , 15 , 1 ] have been wildly used to solve this problem . With the development of deep neural networks , learning-based methods ( e.g. , CNN-based [ 24 , 65 , 7 , 82 ] and GAN-based [ 79 , 5 ] ) have also been adopted to recover the haze-free images by extracting image features from a large amount of training data . However , these methods do not generalize well to real-world images , because they depend strongly on the image features extracted from training data and do not explicitly consider useful constraints from physical image formation models . For better generalization , multi-image dehazing methods have been proposed . They capture multiple images from different viewpoints [ 38 , 52 , 84 , 63 ] , weather conditions [ 58 , 55 , 56 , 57 ] , or polarization angles [ 77 , 78 , 53 , 81 , 76 , 25 , 54 ] . Although all of these multi-image dehazing methods can relieve the ill-posedness , polarization-based ones have their unique advantages , since they directly utilize the physical image formation model with less dependency on image features extracted from training data . Nowadays , multiple polarized images can be conveniently captured in a single shot using a ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . polarization camera such as Lucid Vision Phoenix polarization camera2 . However , these polarizationbased methods are not robust due to several issues : ( 1 ) . They are largely based on a strong assumption that the transmitted light is not significantly polarized , while it is not the case for real-world images since both transmitted light and airlight contribute to the polarization [ 13 , 29 ] . ( 2 ) . They usually require specific clues ( e.g. , sky regions [ 13 , 77 , 53 ] , similar objects [ 54 , 50 ] , known depth [ 54 ] ) to estimate the infinite airlight and the degree of polarization ( DoP ) , which significantly reduces their applicability since these requirements are not always met . ( 3 ) . They are optimization-based methods which do not make full use of semantic and contextual information in image features to handle the spatially-variant real-world scattering . In this paper , to enable the polarization-based dehazing methods to handle images captured in the wild more robustly , we propose a generalized physical formation model of hazy images , without assuming that the transmitted light is not significantly polarized , while considering the spatiallyvariant real-world scattering . Based on the physical model , we propose a robust polarization-based dehazing pipeline to extend their applicability by adopting deep learning to estimate the infinite airlight and the DoP of both transmitted light and airlight without the requirement of specific clues like sky regions , similar objects , etc . According to our dehazing pipeline , we design a neural network to perform the dehazing process : It first estimates the DoP of both transmitted light and airlight to solve the transmitted light , then predicts the infinite airlight to reconstruct the original scene radiance . Thanks to our learning-based pipeline , our method extracts image features from training data and use semantic and contextual information to refine the results , which is suitable for handling the spatially-variant real-world scattering . To summarize , this paper makes contributions by demonstrating : ( 1 ) . A generalized physical formation model of hazy images , taking into account the polarization effects of both transmitted light and airlight , along with the spatially-variant real-world scattering . ( 2 ) . A robust polarization-based dehazing pipeline without the requirement of specific clues , by adopting deep learning to estimate necessary physical parameters ( infinite airlight , DoP of both transmitted light and airlight ) . ( 3 ) . A neural network making full use of semantic and contextual information to handle the spatially-variant real-world scattering to improve the clarity of original scene radiance recovery . Experimental results show that our approach achieves state-of-the-art performance on both synthetic data and real-world hazy images . 2 Related work . Single-image dehazing . Single-image dehazing is a highly ill-posed problem because it requires estimating multiple unknowns ( the transmitted light and airlight ) from a single observation . Park et al . [ 64 ] estimated haze from the difference among the RGB channels . Some methods adopted an adaptive contrast enhancement strategy to maximize the local contrast of restored images [ 87 , 17 , 47 ] . Some works proposed several assumptions ( e.g. , the surface shading and transmission are locally uncorrelated [ 14 ] , the scene albedo and depth are independent [ 59 ] , both the scene albedo and transmission are constant inside each patch [ 86 ] ) or image priors ( e.g. , dark channel prior [ 21 , 89 , 49 , 37 ] , color attenuation prior [ 98 ] , non-local prior [ 1 , 43 ] , ellipsoid prior [ 19 ] , color-lines [ 15 ] ) to handle this problem . Recently , with the development of deep neural networks , learning-based methods have also been adopted to recover haze-free images by extracting image features from a large amount of training data . These learning-based methods could be divided into two groups : direct methods , which dehaze in an end-to-end manner using convolutional neural networks ( CNN ) [ 2 , 68 , 30 , 31 , 62 , 6 , 18 , 45 , 95 , 44 , 36 , 24 , 65 , 7 , 8 , 70 , 94 , 51 , 4 , 48 , 3 ] or generative adversarial networks ( GAN ) [ 93 , 69 , 35 , 92 , 66 , 33 , 34 , 79 , 5 , 9 , 12 , 85 , 10 , 82 ] , and indirect methods , which 2https : //thinklucid.com/product/phoenix-5-0-mp-polarized-model/ estimate image priors or their variants [ 88 , 74 , 91 , 46 , 20 ] first and then use them to dehaze . Although these methods have shown their effectiveness in a large variety of scenes , their generalization ability is still limited , since image priors are not always observed in the input and the image features extracted from synthetic training data often have a large domain gap with real-world ones . Multi-image dehazing . For better generalization and less ill-posedness , multi-image dehazing methods have been proposed . They use computational photography techniques to capture multiple images with conventional or unconventional cameras for acquiring extra information . Some methods used separately measured range data [ 60 ] and georeferenced digital terrain with urban models [ 28 ] to facilitate dehazing . Some works captured multiple images under different unknown weather conditions to recover scene depth maps for dehazing [ 58 , 55 , 56 , 57 ] . Some approaches take advantage of stereo vision to remove the effects of haze [ 38 , 52 , 84 , 63 ] by taking multiple photos from different viewpoints . Some methods fuse RGB images with NIR ( near-infrared ) ones to help dehaze [ 75 , 16 , 42 , 11 , 83 ] , because the scattering is significantly smaller in NIR than in visible light since NIR wavelengths are longer . Although these methods have better generalization ability , capturing such data is not easy since they require multiple shots and/or complicated imaging systems . Polarization-based dehazing . Recently , polarization-based methods have been proposed to solve the dehazing problem by capturing multiple polarized images at the same view with different polarization angles . These methods have their unique advantages : they directly utilize the physical image formation model without dependence on image features extracted from training data , and multiple polarized images can be captured in a single shot using a polarization camera . However , most of them are based on a strong assumption that the transmitted light is not significantly polarized [ 77 , 78 , 53 , 81 , 76 , 25 , 54 , 50 , 41 , 40 , 39 , 67 , 80 ] , while it is not the case for real-world images since both transmitted light and airlight contribute to the polarization [ 13 , 29 ] . Fang et al . [ 13 ] takes the polarization of transmitted light into consideration , however , it supposes that the depth of sky regions is approximately infinite and require sky regions to estimate the infinite airlight and the DoP , just like [ 77 , 53 ] , which significantly reduces the applicability since sky regions are not always available . 3 Method . In this section , we show the physical formation model of hazy images in Section 3.1 , demonstrate our polarization-based dehazing pipeline in Section 3.2 , and introduce our neural network in Section 3.3 . 3.1 Physical image formation model . As shown in Figure 1 ( top row ) , when taking photos in hazy environments , caused by atmospheric scattering , the captured image I = { I ( x , y , c ) } ( ( x , y ) is the pixel coordinate and c denotes the color channel index ) is composed of two components : the transmitted light T = { T ( x , y , c ) } ( an attenuated fraction of original scene radiance R = { R ( x , y , c ) } ) , which decreases with the scene depth z = { z ( x , y ) } , and the airlight A = { A ( x , y , c ) } ( ambient light scattered towards the viewer ) , which increases with z . According to [ 77 ] , the formation of a hazy image can be described as I = T + A = R · e−β·z + A∞ · ( 1− e−β·z ) , ( 1 ) where β = { β ( c ) } is the scattering coefficient , A∞ = { A∞ ( c ) } denotes the infinite airlight ( the airlight radiance corresponding to an object at an infinite distance , e.g. , the horizon ) , and · stands for element-wise multiplication . A synthetic example of its visualization can be found in Figure 1 ( bottom left ) . However , real-word scattering does not always satisfy such an ideal model , which means that both β and A∞ are not only dependent on wavelength , but also on the the size of the scattering particles [ 27 , 22 ] and angular scattering coefficient [ 78 ] . To encode such variations , we replace them with β = { β̄ ( c ) +N ( x , y , c ) } and A∞ = { Ā∞ ( c ) +N ( x , y , c ) } respectively , where ·̄ marks the mean value and N ( x , y , c ) denotes the spatially-variant turbulence . Assume for a moment that the illumination of any scattering particle comes from one direction , the light ray from the source to a scatterer and the line-of-sight from the camera to the scatterer define a plane of incidence ( PoI ) [ 77 ] , as shown in Figure 1 ( top row ) . We decompose I , T , and A into two components respectively : I‖ and I⊥ , T‖ and T⊥ , A‖ and A⊥ , where the subscript ‖ ( ⊥ ) means the component is parallel ( perpendicular ) to the PoI . The degrees of polarization ( DoP ) of I , T , and A are defined as P , I⊥ − I‖ I , PT , T⊥ − T‖ T , and PA , A⊥ − A‖ A , ( 2 ) respectively , where I = I⊥ + I‖ , T = T⊥ + T‖ , and A = A⊥ + A‖ . ( 3 ) Since the scattered light is partially polarized perpendicular to the PoI [ 77 , 27 , 22 , 13 ] , P , PT , and PA are not less than zero . Besides , although P is spatially-variant ( i.e. , P = { P ( x , y , c ) } ) , the distributions of PT and PA are not irregular : the values of PT are approximately uniform in the same semantic segment3 , while PA can be regarded as spatially-uniform , i.e. , PA = { PA ( c ) } , according to [ 13 ] . A synthetic example of their visualization can be found in Figure 1 ( bottom right ) . When we place a polarizer with polarization angle α in front of the camera , according to Malus ’ law [ 22 ] , the captured polarized image Iα can be calculated as Iα = I · ( 1− P · cos ( 2 ( α− θ‖ ) ) ) 2 , ( 4 ) where θ‖ = { θ̄‖ +N ( x , y , c ) } denotes the orientation of the polarizer for best transmission of the component parallel to the PoI . Similarly , the two components T and A at angle alpha can be calculated as Tα = T · ( 1− PT · cos ( 2 ( α− θ‖ ) ) ) 2 and Aα = A · ( 1− PA · cos ( 2 ( α− θ‖ ) ) ) 2 , ( 5 ) which satisfy Iα = Tα + Aα . Note that both T and A contribute to the polarization , and the polarization of T should not be ignored [ 13 , 29 ] . From Equation ( 4 ) and Equation ( 5 ) , we can derive the following equation : I · P = T · PT + A · PA , ( 6 ) which reveals that the relationship among I , T , and A are determined by P , PT , and PA. 3The polarization properties of transmitted light depend on material properties of scene objects ( e.g. , surface texture ) [ 29 ] , and objects in the same semantic segment often have similar material properties .

Summary: This paper proposes a neural network for image dehazing where the input is expected to contain polarization information (e.g. multiple polarized images of the exact same scene). The total irradiance and total degree of polarization can be obtained directly from the polarized input frames. The paper then follows the image formation model of [77] (and others) to identify key quantities to estimate: the infinite airlight color, and the degree of polarization of the ambient and scene light. Multiple neural networks are designed to estimate these components; they are coupled using closed-formed expressions derived from the image formation model to produce the final dehazed image. The networks are trained independently, then finetuned jointly on a synthetic dataset.