Learning to dehaze with polarization
Authors: Chu Zhou
Minggui Teng
Yufei Han
Chao Xu
Boxin Shi
Venue: NeurIPS 2021 Poster (2021)
Keywords: Computational photography
polarization-based image dehazing

Abstract: Haze, a common kind of bad weather caused by atmospheric scattering, decreases the visibility of scenes and degenerates the performance of computer vision algorithms. Single-image dehazing methods have shown their effectiveness in a large variety of scenes, however, they are based on handcrafted priors or learned features, which do not generalize well to real-world images. Polarization information can be used to relieve its ill-posedness, however, real-world images are still challenging since existing polarization-based methods usually assume that the transmitted light is not significantly polarized, and they require specific clues to estimate necessary physical parameters. In this paper, we propose a generalized physical formation model of hazy images and a robust polarization-based dehazing pipeline without the above assumption or requirement, along with a neural network tailored to the pipeline. Experimental results show that our approach achieves state-of-the-art performance on both synthetic data and real-world hazy images.

Learning to dehaze with polarization
Chu Zhou' Minggui Teng" Yufei Han" Chao Xu' Boxin Shi
Key Lab of Machine Perception (MOE), Dept. of Machine Intelligence, Peking University 2 Naa'll Eng. Lab for Video Technology, School of Computer Science, Peking University Institute for Artificial Intelligence, Peking University
"Beijing Academy of Artificial Intelligence
School of Info. and Comm. Eng., Beijing University of Posts and Telecommunications (zhou_chu, minggui_teng, shiboxin] Opku. edu. cn,
hanyufeibuppt. edu. en, xuchaoBcis, pku. edu, cn
Abstract
Haze, a common kind of bad weather caused by atmospheric scattering, decreases the visibility of scenes and degenerates the performance of computer vision algo- rithms. dingle-image dehazing methods have shown their effectiveness in a large variety of scenes, however, they are based on handcrafted priors or learned features, which do not generalize well to real-world images. Polarization information can be used to relieve its ill-posedness, however, real-world images are still challenging since existing polarization-base based methods usually assume that the transmitted light is not significantly polarized, and they require specific clues to estimate necessary physical parameters. In this paper, we propose generalized physical formation model of hazy images and a robust polarization- the above assumption requirement, along with neural network tailored to the pipeline. Experimental results show that our approach achieves state-of-the-art performance on both synthetic data and real-world hazy images.
dehazing pipeline without
Introduction
When taking photos in hazy environments, the visibility and color fidelity of recorded scenes usually contaminated, because the captured images often contain a superposition two unknown components: the transmitted light (an attenuated fraction of original scene radiance), and the airlight (ambient light scattered towards the viewer). It is highly ill ill-posed to separate them a single hazy image it requires estimating multiple unknowns from single observation. Handcrafted priors from natural image statistics [5 have been wildly used to solve this problem. With the development of deep neural networks, learning- baaeed methods (e.g., CNN-based (24 65 S2 and GAN-based
[79 5l) have also been adopted to recover the haze-free images by extracting image features from a large amount of training data. However, these methods do not generalize well real-world images, because they depend strongly on the image features extracted from training data and do not explicitly consider useful constraints from physical image formation models.
For better generalization, multi-ime imati-iimge dehazing methods have been proposed. They capture multiple images from different viewpoints (38]
52
84 [3] weather conditions [58 55
56 (5). polarization
angles 78) 53 81 76,
25 54 Although all of these multi-image dehazing methods can relieve
the ill-posedness, polarization-basse
ones have their unique advantages, since they directly utilize the physical image formation model with less dependency on image features extracted from training data. Nowadays, multiple polarized images can be conveniently captured in single shot using
Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

polarization camera such as Lucid Vision Phoenix polarization camer: However, these polarization- based methods are not robust due several issues:
(1). They are largely based strong assumption that the transmitted light is not significantly polarized, while it is not the case for real-world images since both transmitted light and airlight contribute to the polarization [13 29]
(2). They usually require specific clues (e.g., sky regions B
53 similar objects [54 (0].
known depth (54)) to estimate the infinite airlight and the degree of polarization (DoP), which significantly reduces their applicability since these requirements are not always met.
(3). They are optimization-baset methods which do not make full use of semantic and contextual information in image features to handle the spatially-variant real-world scattering.
In this paper, to enable the polarization-based dehazing methods to handle images captured in the wild more robustly, we propose a generalized physical formation model of hazy images, without assuming that the transmitted light not significantly polarized, while considering the spatially- variant real-world scattering. Based on the physical model, we propose a robust polarization based dehazing pipeline to extend their applicability by adopting deep learning to estimate the infinite airlight and the DoP of both transmitted light and airlight without the requirement of specific clues like sky regions, similar objects, etc. According to our dehazing pipeline, we design a neural network to perform the dehazing process: It first estimates the DoP of both transmitted light and airlight to solve the transmitted light, then predicts the infinite airlight to reconstruct the original scene radiance. Thanks to learning-base -ased pipeline, our method extracts image features from training data and use semantic and contextual information to refine the results, which is suitable for handling the spatially-varian real-world scattering.
To summarize, this paper makes contributions by demonstrating:
(1). A generalized physical formation model of hazy images, taking into account the polarization effects of both transmitted light and airlight, along with the spatially-variant real-world scattering.
(2). A robust volarization--asse dehazing pipeline without the requirement of specific clues, by adopting deep learning to estimate necessary physical parameters (infinite airlight, DoP of both transmitted light and airlight).
(3). A neural network making full use of semantic and contextual information to handle the spatially-variant real-world scattering to improve the clarity of original scene radiance recovery.
Experimental results show that our approach achieves state-of-the-an performance on both synthetic data and real-world hazy images.
2 Related work
Single-image dehazing. Single-image dehazing is highly ill-posed problem because it requires estimating multiple unknowns (the transmitted light and airlight) from single observation. Park et al. [64] estimated haze from the difference among the RGB channels. Some methods adopted an adaptive contrast enhancement strategy to maximize the local contrast of restored images 87 17 47).
Some works proposed several assumptions (e.g., the surface shading and transmission are locally uncorrelated [4], the scene albedo and depth are independent (59], both the scene albedo and transmission are constant inside each patch [66)) or image priors (e.g., dark channel prior [2 89
49 37,, color attenuation prior (98), non-local prior [ 43), ellipsoid prior [19], color-lines [5]
to handle this problem. Recently, with the development of deep neural networks, learning- based methods have also been adopted to recover haze-free images by extracting image features from a large amount of training data. These learning- learning-based methods could be divided into two groups: direct methods, which dehaze in an and-to-end manner using convolutional neural networks (CNN)
[2, 66, 30 31 62, 6 88 45 95
44
36
24
65
7 8
70
94
5
48 3 generative adversarial 10
4
networks (GAN) 93 69 35 92
66
33
34
79
9
S5
82
and indirect methods, which
5-0 mp- polpoaiiieed model/

---

estimate image priors or their variants [8 74 91 46 20 first and then use them to dehaze. Although these methods have shown their effectiveness in a large variety of scenes, their generalization ability is still limited, since image priors are not always observed in the input and the image features extracted from synthetic training data often have a large domain gap with real-world ones.
Multi-image dehazing. For better generalization and less ill ill-posedness, multi-image dehazing methods have been proposed. They use computational photography techniques to capture multiple images with conventional or unconventional cameras for acquiring extra information. Some methods used separately measured range data [60] and georeferenced digital terrain with urban models [28] to facilitate dehazing. Some works captured multiple images under different unknown weather conditions recover scene depth maps for dehazing [58
55
[66] [57]. Some approaches take 84
63 by taking multiple photos from different viewpoints. Some methods fuse RGB images with NIR (near-infrared) ones to help dehaze 775 16, 42
advantage of stereo vision to remove the effects of haze 38
52
11 33 because the scattering is significantly smaller in NIR than in visible light since NIR wavelengths are longer. Although these methods have better generalization ability, capturing such data is not easy since they require multiple shots and/or complicated imaging systems.
Polarization- based dehazing. Recently, polarization- based methods have been proposed solve the dehazing problem by capturing multiple polarized images at the same view with different polarization angles. These methods have their unique advantages: they directly utilize the physical image formation model without dependence on image features extracted from training data, and multiple polarized images can be captured in a single shot using polarization camera. However, most of them are based on a strong assumption that the transmitted light is not significantly polarized 39 67 80) while it is not the case for real-world images since both transmitted light and airlight contribute to the polarization 13 29]. Fang et al. [13] takes the polarization of transmitted light into consideration, however, it supposes that the depth of sky regions is approximately infinite and require sky regions to estimate the infinite airlight and the DoP, just like
77 78 53 76 25 54 50
77 53 which significantly reduces the applicability since sky regions are not always available.
3
Method
In this section, we show the physical formation model of hazy images in Section 3.1 polarization-based dehazing pipeline in Section 3.2 and introduce our neural network in Section 3.3
demonstrate our
3.1 Physical image formation model
As shown in Figure (top row), when taking photos in hazy environments, caused by atmospheric scattering, the captured image color channel index) is composed of two components: the transmitted light T = {T' y, c}} (an attenuated fraction of original scene radiance R = {R(x, y, c)}), which decreases with the scene depth z = {2 y)}, and the airlight A = {A(z, y, c)} (ambient light scattered towards the viewer), which increases with z. According to 77 the formation of hazy image can be described as
{1( y, c)} ((x, y) is the pixel coordinate and denotes the
= T+ R
3z 3 Ax (1-e-32))
(I)
where 3 = {B(c)} the scattering coefficient, Ax = (c)( denotes the infinite airlight (the airlight radiance corresponding to an object at an infinite distance, e.g., the horizon), and stands for element-wise multiplication. A synthetic example of its visualization can be found in Figure [ (bottom left). However, real-word scattering does not always satisfy such ideal model, which means that both B and Asc are not only dependent on wavelength, but also on the the size of the scattering particles [27] 22 and angular scattering coefficient (78). To encode such variations, we replace them with 3 = (:(cc) + ((x, y, c)} and Ax = { A (c) + N(x, y, <)} respectively, where: marks the mean value and (xx, y, c) denotes the spatially- variant turbulence. Assume for moment that the illumination of any scattering partiele comes from one direction, the light ray from the source to scatterer and the line-of-sight from the camera to the scatterer define a plane of incidence (Pol) 777. as shown in Figure [ (top row). We decompose I, T, and A into two components respectively: Ill and 1 T and T A and A where the subscript || (.L) means the component is parallel (perpendicular) to the Pol. The degrees of polarization (DoP) of I, T. and are defined as
Ill
T
T
PA A A
P
Pr
and
T
A
3

---

Illumination
(Normal of Poll
Infinite airlight A
Incident airlight
distarce
Polarizer
Airlight
Pol
Transmitted light
Atmospheric particles (scatterers)
Incident direction
Scene radiance
Camera
Depth
R
Almospheric scattering
P
Py
1 ef
A..
PA
Semantic segmentation map
Figure 1: Top row: An illustration of the atmospheric scattering and polarization; transmitted light (blue solid line) T is an attenuated fraction of original scene radiance R that decreases with the scene depth z; airlight A (red dashed line) is the ambient light scattered towards the viewer that increases with z; when placing a linear polarizer with polarization angle a front of the camera, the polarization component parallel to the plane incidence (Pol) best transmitted through the polarizer at a = 0 Bottom left: A synthetic example for visualizing the formation of a hazy image (see Equation ① for details). Bottom right: synthetic example for visualizing P, Pr, and PA (the DoP of I, T. and A, see Equation 2 for details), along with the semantic segmentation map.
respectively, where
I=I +II,
T= TT and A A AAl.
(3)
Since the scattered light is partially polarized perpendicular the Pol [77 27 22 13 P. Pr, and PA are not less than zero. Besides, although P is spatially-varian (i.e., P = {P(x. x y, c)}), the distributions of Pr. and PA not irregular: the values of Pr approximately uniform in the same semantic segmenf" while PA can be regarded as spatially-unfform, i.e., PA (c)(c)), according to
13 A synthetic example of their visualization be found in Figure T (bottom right).
When we place polarizer with polarization angle a front of the camera, according to Malus' law 22 the captured polarized image L, can be calculated
I I. (1 - cos(2(a 01)))
(4)
where 0 {0 + N(x,, y, c)} denotes the orientation of the polarizer for best transmission of the component parallel to the Pol. Similarly, the two components T and A at angle alpha can be calculated as
Tg T Pr cos(2(a ))))
and A A PA cos(2(a 1n)))
(5)
2
2
which satisfy I, = T, + A.. Note that both and A contribute to the polarization, and the polarization of T should not be ignored 13 29). From Equation (4) and Equation (5, we derive the following equation:
= T P + A P
(6)
which reveals that the relationship among I, T, and A are determined by P. Pr, and PA-
'The polarization properties of transmitted light depend on material properties of scene objects (e.g., surface texture) 29 and objects in the same semantic segment often have similar material properties.

---

Eq.
82 Refine f
P.
Transmittdd estimation
Precompute using lineer system from Eq. (8)
Original radiance reconstruction
1,00
Input
A
Output
Figure 2: We design a network tailored to our polarization-base dehazing pipeline (in Section 3.2)
which takes three polarized images Tac9 (i 1, 2, 3) captured at the same view with different polarization angles (i 2, 3) the input (along with the precomputed hazy image I and its DoP P using the linear system from Equation (8) ) and outputs the reconstructed original scene radiance R. It consists of two stages: transmitted light estimation and original scene radiance reconstruction. The first stage includes two subnetworks for estimating Pr, PA. and refining T. The second stage also includes two subnetworks for estimating Asc and refining R. C: denotes the coarse value calculated from Equation 7
3.2 colarization-based dehazing pipeline
We aim to restore the original scene radiance R using three polarized images 1, (i 1,2,3) captured at the same view with different polarization angles (i 2,) Eliminating A from Equation (T) and Equation (6). T and could be computed by the following two equations:
T A
(a) T Pr-PA and (b)
Ax - (I-
where Pz, PA. and A are required to be estimated, and can be directly calculated by I, 1, 2, 3).
Now first explain how calculate I and P using [ (i 1,2,3). Expanding Equation obtain (() denotes inner product)
I, where D = D2 = I.P. PPoos(20) and D I. I.P. P--ini)())))))
cos((a))
-sin(2)]
Di D2 Dzl):
Since Equation 8 has three unknowns (D,(i = 1, 3)), we can use Iaco (i 1, to obtain linear system that allows compute them. Then, we could calculate and P by I D2 and
/(D D
respectively.
Next, we only need to estimate three parameters Pr, P.. and A to reconstruct R. To alleviate the
dependency on specific clues (such sky regions [3 77 53 similar objects [54] (0), which are required by other polarization-be deep neural network that comprehensively explores physics and semantic features.
methods) for estimating these parameters, we choose to design
3.3 Polarization-base dehazing network
As shown in Figure 2 our network consists of two stages: transmitted light estimation and original scene radiance reconstruction.
Transmitted light estimation. As shown in the first stage of Figure it aims estimate the DoP of both transmitted light and airlight (Pr and PA) for solving the transmitted light T. So, it adopts
subnetwork 91 to estimate Pr and PA. then uses Equation (7) (a) to calculate T (the coarse value

---

of T)) However, we cannot directly feed T into the second stage since the numerical problem will occur when the denominator of Equation 7 (a) approaches zero, which often happens in pixels where Pr Re PA (the DoP of transmitted light and airlight are approximately the same). Besides, the estimated Py and PA by 91 are prone to be noisy which distorts the calculated T, because the apatially-variant turbulence is hard to learn due to its irregularities. So, we adopt another subnetwork 92 to refine T using semantic and contextual information extracted from Laco (i = 2,2,3) 3). In practice, we construct 9i using the U-Net architecture [7] since it works well on per-pixel estimation tasks such as semantic segmentation [ As for 92, we choose the autoencoder architecture [23] by virtue of its excellent context generalization ability for refining image contents.
Original scene radiance reconstruction. As shown in the second stage of Figure 2 aims to estimate the infinite airlight A. to reconstruct the original scene radiance R. So, it first adopts
subnetwork 93 to estimate so, then uses Equation (7) (b) to calculate R (the coarse value of R).
However, R also needs to be refined, because when the haze in some pixels is very thick and leaves little information of the transmitted light (T = 0), the numerator of Equation 7 (b) approaches zero, which leads to a wrong result that R zs So, similar to the first stage, we also adopt a subnetwork 91
to refine R. We also choose the U-Net architecture [71 for 93 and the autoencoder architecture 23 for g4-
4 Data preparation and network training
In this section, we first detail our synthetic dataset generation pipeline in Section 4.1 loss function and training strategy in Section
then show our
4.1 Synthetic dataset generation pipeline
is difficult to obtain pairwise hazy and clear images with three polarized observations at a large scale. Besides, getting the ground truth values of the DoP or infinite airlight is not feasible. So, we propose to generate a synthetic dataset for training our network. Since we require spatially-variant B and Ac to simulate real-world scattering, and need the semantic segmentation map S for generating reasonable p (see Section 3.1 for details about the properties of Py.), we cannot directly generate the polarized images from the hazy images in existing dehazing benchmarks [32
72 The
desired data source for generating our dataset should provide:
(1). clear image R with depth map z, from which we can calculate using Equation II by generating spatially-variant B3 and Axi
(2). semantic segmentation map S, from which we can generate reasonable Pr using Pr - ((S). where denotes function which randomly maps each semantic segment to value of Pr.
The Foggy Cityscapes-DBF dataset (72) meets the above two requirement so we use the provided R, and S to generate our synthetic dataset. In short, with z, R, and S available, our synthetic dataset generation pipeline could be described as
(1). Randomly generate /3 (in [0.01. 0.02), Axo (in [0.85, 0.95), and PA (in [0.05, 0.4)) to calculate T, A, and I using Equation [I]
(2). generate Pr from S using Pr- = (S) (in [0.025, 0.2)). then calculate P using Equation (
(3). randomly generate lI (in |-45", 45%), then use Equation (4) to calculate I, (i 1, 2, (i 2) are set to be 0%, 45", and 90" respectively).
The visualization of above mentioned parameters can be found in the bottom row of Figure I Note that for (3 = {B(c) + N(x, y, c)} and Ax = { (c) + N (I,, y, c)). we first generate their mean values (c) and A (c) for each channel, then add 5% Gaussian noise to make them spatially-variant
Besides, we also add 2% Gaussian noise to I (i = 1, 2,3.. To conform to the real-world scattering
"Although it does not directly provide z, it offers the transmittance (c (8-x) with known spatially scattering coefficient /3, so that we can can compute z by ourselves.
'The range of (3 is from Li et al. [32] with some adjustment (changing the sampling space from discrete to continuous), and the ranges of PA and Pr are from the statistics in Fang et al. [13]

---

images 1,0220
Hazy image
Original scene
P:31.73 N:8.973
P:23.96 N:0.885
GDN
BPP
FFA
HardGAN
MSBDN
P:22.40 M:8.924
56 N:8.934
P:31.23 M:8.973
P:26.98 M:8.339
P:28.62 N:0,919
Polarized images La
Hazy image!
Original scone
Ours P:33.11 M:8.982
SPCVE
P:19.64 N:8.652
GDN
BPP
FFA
HardGAN
MSBON
P:38.47 8:8.964
P:28.95 N:0.958
P:38.48 N:8.967
P:29.64 M:0.961
P:38.72 M58.664
images 1,22
Hazy image
Original scene radiance
Ours P:38.35 N:8. 965
P:28.45 N:0.719
SPCVE
GDN
BPP
FFA
HardGAN
MSBDN
P:28.69 M58.952
P:27.84 N:8.944
P:79.84
P:27.74 M:8.345
P:28.75 N:0.968
Figure 3: Qualitative comparisons on synthetic data among our method, representative polarization- based dehazing algorithm SPCVE (54) which also takes three polarized images as the input, and
five state-of-the-art learning-ban based dehazing methods including GDN 44], BPP [82], FFA (65), HardGAN [5), and MSBDN 7 which take single hazy image as the input. Quantitative results evaluated using PSNR (P) and MS-SSIM (M) are displayed below each image.
77.. ensure that B(r) 3(g) < ((b) and Pa(r) Pa(g) Pa(b). The images resized and randomly cropped to 240 240 patches during the training process, and cropped to 496 x 240 patches for tes
Loss function and training strategy
Loss function. The total loss function of our network is L b Lg, + >2. Lyz + Ax Lgs + Lg.. where Cg (i = 1, 1, 3, 4) define the loss of our four subnetworks, Each of them could be described as
Our training (test) images are generated from the training (test) images of the Foggy Cityscapes-DBi DDBF dataset
72

---

Table 1: Quantitative evaluation results on synthetic data among our method, a representative
polarization- based dehazing algorithm SPCVE [54] (taking three polarized images as the input), and five state-of-the-an bearning-based dehazing methods including GDN A4 BPP 82 FFA 65 HardGAN [5] and MSBDN (taking a single hazy image as the input).
Ours
SPCVE 54
GDN 44
BPP 82
FFA 65
HardGAN [
MSBDN 7
PSNR
28.32
15.94
26.54
24.93
26.84
26.22
26,94
MS-SSIM
0.951
0.521
0.928
0.915
0.934
0.928
0.932
Lg 2 + L2, where L1 and La denote the I.1 and Tz2 loss respectively. As(i 1,2,3,4 4) are empirically set to be 1.0, 1.0, 2.0, 2.0 respectively.
Training strategy. We implement our network using PyTorch on an NVIDIA 2080Ti GPU and apply a two-phase training strategy. First, to ensure a stable initialization of the training process, we train our two network stages independently for 400 epochs. ADAM optimizer (26) is used with an initial learning rate 5 x 10-4 for the first 300 epochs, and linear decay to 2.5 x 10 in the next 100 epochs. Then, we finetune the entire network in end-to-end manner for another 300 epochs, keeping the learning rate to 5 x 10 44 Instance normalization [90] are added during training.
5 Experiments
5.1 Evaluation on synthetic data
We compare our results to a representative polarization- based dehazing algorithm SPCVE [54] which also takes three polarized images as the input and five state-of-the-art beamnin-bbssed dehazing methods including GDN [44] BPP 82] FFA
[65] HardGAN (5), and MSBDN 7 which take 54 assumes that the transmitted light not significantly polarized (Pr = 0), and uses optimization to estimate PA and A... while our method takes into account the polarization effects of transmitted light and adopts deep learning to estimate Pr, PA. and Ago. All of these learning- boaning-bbasedaee methods are re- re-trained on our dataset using R and I (the calculated hazy image from 1. (i 1, 2, 3) using the linear system from Equation (8) Note that comparing with learning- bbsed dehazing methods might be a bit unfair because of the difterence in types of input data (ordinary image vs. polarized image), and conduct such a comparison show the advantage of using polarized images over image-only approaches.
single hazy image as the inpu SPCVE
Visual quality comparisons of dehazed results are shown in Figure f Compared to the polarization- based dehazing algorithm SPCVE (54), our method can dehaze robustly with fewer artifacts; com- pared to the learning- based methods, our method performs better in recovering details. Taking the sky region (green box) in the first group of Figure 3 as an example, SPCVE [54] suffers severely from noise, and the learning- based methods yield bad pixels (shown black streaks in the sky). This is because in our synthetic dataset we not only simulate the polarization effects of airlight but also transmitted light, and add spatially-varian ignores the polarization effects of the transmitted light and does not consider semantic and contextual information to refine the results, and the bearning-based methods [44 artifacts for the pixels with large spatially-varian turbulence. To evaluate the results quantitatively, image quality metrics including PSNR and MS-SSIM (multi-scale
turbulence to the scattering process, while SPCVE [54]
82 65 7 are prone to
we adopt two frequently- SSIM). Results are shown in Table I (also below corresponding examples in Figure B Our model consistently outperforms the volarization-basee and learning-based dehazing methods on all metrics.
"Note that the code of SPCVE [44] is not available and the demonstrated results are based on our own implementation. We directly provide ground truth Axe to our implementation as its upper bound performance. also owing to that SPCVE [54) requires similar objects or known depth to estimate Ase. which are not always available in our scenes.
We should not use the ground truth I as the input of these learning- based methods since we could only get i during the inferring phase of real data; and if we use I to re-train them, their results will be degenerated due to the large domain gap between and (caused by the noise in T_.o (i 1,2,3)). Additional synthetic results can be found in the supplementary material.
8

---

Hazy image
Ours
SPOVE
GDN
Polarized images L,00
BPP
FFA
HardGAN
MSBON
Hazy image
Ours
SPOVE
GDN
Polarized images (2223)
H
BPP
FFA
HardGAN
MSBINN
Figure 4: Qualitative comparisons on real data. See the caption Figure 3 for explanation. All dehazing results are white-balas white-balanced to the similar color appearance and multiplied by factor of 1.25 for better visualization. Please zoom-in for better details.
Polarized image pm
Polarized image (a))
Polarized image [اي)
PA
Estimated
Polarized image 4eu
Polarized image La)
Polarized image [ا:)
Estimated P4
Estimated Py
Figure The polarized images side by side with the estimated Pa and Pr real data.
5.2
Evaluation on real data
We use the Lucid Vision Phoenix polarization camera (RGB) to capture real data. The polarization camera can take four images with different polarization angles (0%, 45" 900, and 135%) at a single shot. We use three of them (0", 45% and 90%) as the input to our method and SPCVE [54), and
calculate the hazy image from the polarized images I, (i 1, 2,3) 3) using the linear system from
Equation (8) the input dearning-based methods (GDN 44 BPP 82 FFA (65 HardGAN (5),
and MSBDN (7).. Visual quality comparisons of dehazed results are shown in Figure 00 Our method able to generate clearer and brighter images than those by the saate-of-theaass polarization based and lcarning- based methods. For example, the color of the buildings (red box) in the first group of Figure correctly recovered by our method, while other methods suffer from color distortion
Additional real results can be found in the supplementary material.

---

Table 2: Quantitative evaluation results of ablation study.
PSNR
MS-SSIM
Ignoring the polarization effects of the transmitted light Neglecting the spatially-van
27.63
0,943
real-world scattering
27.86
0,948
Directly estimating T and R
27.27
0.945
Removing (91)
21.55 (21 (21.28
0.740 (0.903)
Removing both 92 and 91
17.04
0.662
Our final model
28.32
0.951
Ours
Foggy image
Polarized images 1,z2
Polarized images J,n2.
Misty image
Ours
case fog (on top of mountain)
case mist around image)
Figure 6: Two failure cases (fog and mist) in which our method shows degenerate performance.
artifacts which dim the results. For better visualization, we also show the polarized images side by side with the estimated P and Pr in Figure 5 We can see that the distributions of PA and P7 satisfy the ones mentioned Section 3.1 which demonstrates the rationality of our motivation.
5.3 Ablation study
To verify the validity each model design choice, we conduct series ablation studies and show comparisons in Table 2 comparing with a model that ignores the polarization effects of the transmitted light (by taking P as zero) and model that neglects the spatially-varie -variant real-world scattering (by taking Pr, PA. and Agc as spatially-unifone -unfform parameters). From the results we can see that our model more generalized and reasonable. We further verify the contribution of our dehazing pipeline which estimates three parameters (Pr, PA. and Ax) to solve and R by comparing with a model that directly estimates dehazing pipeline is better than directly estimating T and R since these
We first show the effectiveness of our physical image formation model by
T and R. We find that parameters are casier to learn than T and R. Then, we demonstrate the necessity of the refinement subnetworks (g2 and (a) by removing 92. 94. and both of them. We could tell that without the refinement subnetworks"" I1]] the performance degenerates rapidly, while still outperforms the existing polarization-bart based (also optimization-bant basedd dehazing algorithm SPCVE [54] (see Table I for the performance of SPCVE (54)) thanks to our generalized physical image formation model.
6 Conclusion
We presented learning- solution which leverages the properties of polarized light for image dehazing. To handle the images captured in the wild, we proposed a generalized physical formation model of hazy images, introduced robust polarization based dehazing pipeline, and designed neural network tailored the pipeline, showing satte-of-th-asl performance. Our solution extended the applicability of polarization- infinite airlight and the DoP of both transmitted light and airlight without the requirement of specific clues (e.g., sky regions, similar objects), while considering the spatially-varian real-world scattering.
dehazing methods by adopting deep learning to estimate the
Limitations. Since method is based on the physical image formation model of hazy images, may fail in situations which does not conform to the model, such as fog or mist. As shown in Figure method shows degenerate performance on those images, because fog and mist caused by suspension of water droplets, while haze is a suspension of extremely small particles (other than water droplets) in the air. As future work, plan to extend our model support other situations.
Synthetic results without refinement be found in the supplementary material.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by National Key R&D Program of China (2020AAA01052000, and National Natural Science Foundation of China under Grant No. 62136001, 62088 102, 61872012, 61876007.
References
[1] Dana Berman, Tali Treibitz, and Shai Avidan. Non-local image dehazing, In Proc. of Computer Vision and Pattern Recognition, pages 1674-1682, 2016.
[2] Bolun Cai. Xiangmin Xu. Kui Jia, Chunmei Qing. and Dacheng Tao. DehazeNet: An end-to-end system for single image haze removal. IEEE Trunsactions on Image Processing. 25(11):5187551 2016.
[3] Dongdong Chen, Mingming He. Qingnan Fan, Jing Liao, Liheng Zhang, Dongdong Hou, Lu Yuan, and Gang Hua. Gated context aggregation network for image dehazing and deraining. In Proc. of Winter Conference on Applications of Computer Vision, pages 1375-1383, 2019.
[4] Shuxin Chen, Yizi Chen, Yanyun Qu, Jingying Huang, and Ming Hong. Multi-scale adaptive dehazing
network. In Proc. of Computer Vision and Pottern Recognition Workshops, 2019.
[5] Qili Deng, Ziling Huang, Chung-Chi Tsai, and Chia- Wen Lin. HardGAN: A haze-aware representation distillation GAN for single image dehazing. In Proc. of Europaan Conference on Computer Vision, pages
722-738, 2020.
[6] Zijun Deng, Lei 7hu, Xiaowei Hu, Chi- Wing Fu, Xuemiao Xu, Qing Zhang, Jing Qin, and Pheng-Ann Heng. Deep multi-model fusion for single-image dehazing. In Proc. of International Conference on
Computer Vision, pages 2453-2462, 2019.
[7] Hang Dong. Jinshan Pan. Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming- -ssan Yang. Multi-scale boosted dehazing network with dense feature fusion. In Proc. of Computer Vision and Pattern Recognition, pages 2157-2167, 2020.
[8] Jiangxin Dong and Jinshan Pan. Physics-based feature dehazing networks. Proc. of European Conference on Computer Vision, pages 188-204, 2020.
[9] Akshay Dudhane, Kuldeep M Biradar, Prashant W Patil, Praful Hambarde, and Subrahmanyam Murala. Varicolored image de-hazing. In Proc. of Computer Vision and Pattern Recognition, pages 4564-4573, 2020.
[10] Akshay Dudhane, Harshjeet Singh Aulakh, and Subrahmanyam Murala. RI-GAN:
An end-to-end network for single image haze removal. In Proc. of Computer Vision and Pattern Recognition Workshops, 2019.
[11] Frederike Dümbgen, Majed El Helou, Natalija Gucevska, and Sabine Süsstrunk. Near-infrared fusion for photorealistic image dehazing. Electronic Imuging, 2018( 16)::21-111 2018.
[12] Deniz Engin, Anil Gene, and Hazim Kemal Ekenel. Cycle-Dehaze: Enhanced CycleGAN for single image dehazing. In Proc. of Computer Vision and Pattern Recognition Workshops, pages 825-833, 2018.
[13] Shuai Fang. XiuShan Xia, Xing Huo, and Wen Chen. Image dehazing using polarization effects of objects and airlight. Optics Express, 22(16):1553311933 2014.
[14] Raanan Fattal. Single image dehazing. ACM Transactions on Graphics (Proc. of ACM SIGGRAPH).
27(3):1-9, 2008
[15] Raanan Fattal. Dehazing using color-lines. ACM Transactions on Graphics (Proc. of ACM SIGGRAPH).
34(1):1-14, 2014.
[16] Chen Feng. Shaojie Zhuo Xiaopeng Zhang, Liang Shen, and Sabine Süsstrrnkk. Near-infrared guided color image dehazing. In Proc. of International Conference on Image Processing, pages 2363-2367, 2013.
[17] Adrian Galdran, Javier Jazquez-Corral David Pardo, and Marcelo Bertalmio. Enhanced variational image
dehazing. SIAM Journal Imaging Sciences, 83):1519-1546, 2015.
[18] Yosef Gandelsman, Assaf Shocher, and Michal Trani "Double-DIP": Unsupervised image decomposition via coupled deep-image-prrone In Proc. of Computer Vision and Pattern Recognition, pages 11026-1 1035,
2019.
[19] Kristofor B Gibson and Truong Q Nguyen. An analysis of single image defogging methods using a color
ellipsoid framework. EURASIP Journal on Image and Video Processing. 2013(I):1-14, 2013.
[20] Alona Golts, Daniel Freedman, and Michael Elad. Unsupervised single image dehazing using dark channel prior loss. IEEE Transactions on Image Processing, 29:2692-2701,, 2019.
[21] Kaiming He, Jian Sun, and Xiaoou Tang. Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(12))2341-2333 2010.
[22] Eugene Hecht et al. Optics, volume 5. Addison Wesley San Francisco, 2002.
[23] Geoffrey Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504 5077-665544577,7 507, 2006.
[24] Ming Hong. Yuan Xie, Cuihua Li, and Yanyun Qu. Distilling image dehazing with heterogeneous task imitation. In Proc. of Computer Vision and Pattern Recognition, pages 3462-3471, 2020.
[25] Ran Kaftory. Yoav Y Schechner, and Yehoshua Y Zeevi. Variational distance deeendeed image restoration. In Proc. of Computer Vision and Pattern Recognition, pages 1-8, 2007.
11

---

[26] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.69688 2014.
[27]
GP Können. Polarized light in nature. CUP Archive, 1985.
[28]
Oliver Deussen, Matt Uyttendacle, and Dani Lischinski. Deep Photo: Model-based photograph enhancement and viewing. ACM Transactions on Graphics (Proc. of ACM SIGGRAPH), 27 2008.
Johannes Kopf, Boris Neubert, Billy Chen, Michael Cohen, Daniel Cohen-Or,
[29] Meredith K Kupinski, Christine L, Bradley, David J Diner, Feng Xu, and Russell A Chipman. Angle of linear polarization images of outdoor scenes. Optical Engineering. 58(8)00,24,,, 2019.
[30] Boyi Li, Xiulian Peng, Zhangyang Wang. Jizheng Xu, and Dan Feng. AOD-Net: All-in-one dehazing network. In Proc. of International Conference Computer Vision. pages 4770-4778, 2017.
[31] Boyi Li, Xiulian Peng, Zhangyang Wang. Jizheng Xu, and Dan Feng. End-to-end united video dehazing and detection. In Proc. of the AAAI Conference on Artificial Intelligence, 2018
[32] Boyi Li. Wenqi Ren. Dengpan Fu, Dacheng Tao, Dan Feng. Wenjun Zeng. and Zhangyang Wang. Bench- marking single-image dehazing and beyond. IEEE Trunsactions on Image Processing, 28( 28144452-5 2018.
[33] Chongyi Li, Chunle Guo, Jichang Guo, Ping Han, Huazhu Fu, and Runmin Cong. PDR-Net: Perception- inspired single image dehazing network with refinement. IEEE Transactions on Multimedia, 2203)700--716 2019.
[34] Lerenhan Li, Yunlong Dong, Wenqi Ren, Jinshan Pan, Changxin Gao, Nong Sang, and Ming- Hiung Yang. Semi-supervised image dehazing. IEEE Transactions on Image Processing, 29:2766-2779, 2019.
[35] Runde Li, Jinshan Pan, Zechao Li, and Jinhui Tang. Single image dehazing via conditional generative adversarial network. In Proc. of Computer Vision and Pattern Recognition, pages 8202-8211, 1. 2018.
[36] Yuman Li. Qiguang Miao. Wanli Ouyang. Zhenxin Ma, Huijuan Fang, Chao Dong, and Yining Quan. LAP-Net: Level-aware progressive network for image dchazing. In Proc. of International Conference on Computer Vision, pages 3276-3285, 2019.
[37] Yu Li, Robby T Tan, and Michael Brown. Nighttime haze removal with glow and multiple light colors. 2015.
In Proc. of International Conference on Computer Vision, pages
[38] Zhuwen Li, Ping Tan, Robby T Tan, Danping Zou, Steven Zhiying Zhou, and Loong-Fah Cheong. Simultaneous video defogging and stereo reconstruction. In Proc. of Computer Vision and Puttern Recognition, pages 1988-4997,20 2015.
[39] Jian Liang, Liyong Ren, Haijuan Ju, Wenfei Zhang, and Enshi Qu. Polarimetric dehazing method for dense haze removal based distribution analysis of angle of polarization. Optics Express, 23(20):261462661 2015.
[40] Jian Liang, Liyong Ren, Enshi Qu, Bingliang Hu, and Yingli Wang. Method for enhancing visibility of hazy images based on polarimetric imaging. Photonics Research, 2(1):38 44, 2014.
[41] Jian Liang. Li- Yong Ren, Hai-Juan Ju. En-Shi Qu, and Ying-Li Wang. Visibility enhancement of hazy images based on a universal polarimetric imaging method. Journal of Applied Physics, I 116(17)1731077
2014.
[42] Jian Liang, Wenfei Zhang, Liyong Ren, Haijuan Ju, and Enshi Qu. Polarimetric dehazing method for visibility improvement based on visible and infrared image fusion. Applied Optics, (5(29):8221-8226, 2016.
[43] Qi Liu, Xinbo Gao, Lihuo He, and Wen Lu. Single image dehazing with depth- aware non-local total variation regularization. IEEE Transactions on Image Processing, 27(10)55178 -191, 2018
[44] Xiaohong Liu, Yongrui Ma. Zhihao Shi, and Jun Chen. GridDehazeNet: Attention blase multi-scale network for image dehazing. In Proc. of International Conference on Computer Vision, pages 7314-7323, 2019.
[45] Xing Liu, Masanori Suganuma, Zhun Sun, and Takayuki Okatani. Dual residual networks leveraging the potential of paired operations for image restoration. In Proc. of Computer Vision and Pattern Recognition, pages 7007-7016, 2019.
[46] Yang Liu, Jinshan Pan, Jimmy Ren, and Zhixun Su. Learning deep priors for image dehazing. In Proc. of International Conference on Computer Vision, pages 2492-2500, 2019.
[47] Raúl Luzón Gonzzlzz,,,,,, Juan L Nieves, and Javier Romero. Recovering of weather degraded images based on RGB response ratio constancy, Applied Optics, 54(4): 12111222222111,,,,21111111111111 2015.
[48] Kangfu Mei, Aiwen Jiang, Juncheng Li, and Mingwen Wang. Progressive feature fusion network for realistic image dehazing. In Proc. of Asian Conference on Computer Vision, pages 203-215, 2018.
[49] Gaofeng Meng, Ying Wang, Jiangyong Duan, Shiming Xiang, and Chunhong Pan. Efficient image dehazing with boundary constraint and contextual regularization. In Proc. of International Conference on Computer Vision, pages 617-624, 2013.
[50] Daisuke Miyazaki, Daisuke Akiyama, Masashi Baba, Ryo Furukawa, Shinsaku Hiura, and Naoki Asada. Polarization- based dehazing using two reference objects. In Proc. of International Conference on Computer Vision Workshops, pages 852-859, 2013.
[51] Peter Morales, Tzofi Klinghoffer. and Seung Jae Lee. Feature forwarding for efficient single image dehazing. In Proc. of Computer Vision and Pattern Recognition Workshops, 2019.
12

---

[52] Jeong-You Yun Na and Kuk-Jin Yoon. Stereo vision aided image dehazing using deep neural network. In Proc. of the Ist Workshop and Challenge on Comprehensive Video Understanding the Wild, pages 13-19. 2018.
[53] Einav Namer and Yoav Y Schechner. Advanced visibility improvement based on polarization filtered images. In Polarization Science and Remote Sensing II, volume 5888, page 588805, 2005.
[54] Einav Namer, Sarit Shwartz, and Yoav Y Schechner. Skyless polarimetric calibration and visibility enhancement. Optics Express, 17(2))472-44)
2009.
[55] Srinivasa G Narasimhan and Shree K Nayar. Chromatic framework for vision in bad weather. In Proc. of Computer Vision and Puttern Recognition, volume 1, pages 598-605, 2000.
[56] Srinivasa G Narasimhan and Shree K Nayar. Vision and the atmosphere. International Journal of Computer Vision, 48(3):233-254 254,244,, 2002.
[57] Srinivasa G. Narasimhan and Shree K. Nayar. Contrast restoration of weather degraded images. IEEE Transactions on Puttern Analysis and Machine Intelligence, 25(6):713 (1-774,,, 2003.
[58] Shree K Nayar and Srinivasa G Narasimhan. Vision in bad weather. Proc. of International Conference
on Computer Vision, volume 2, pages 820-827, 1999.
[59] Ko Nishino, Louis Kratz, and Stephen Lombardi. Bayesian defogging. International Journal of Computer Vision, 98(3):263-277)) 2012.
[60] John P Oakley and Brenda L Satherley. Improving image quality in poor visibility conditions using a physical model for contrast degradation. IEEE Transactions on Image Processing, 7(2):167-179, 1998.
[61]
Ozan Oktay, Jo Schlemper. Loic Le Folgoe. Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Berhhard Kainz, Ben Glocker, and Daniel Rueckert. Attention
U-Net: Learning where to look for the pancreas. arXiv preprint arXiv: /00,0999 2018.
[62]
Jinshan Pan, Sifei Liu, Deqing Sun, Jiawei Zhang, Yang Liu, Jimmy Ren, Zechao Li, Jinhui Tang, Huchwan Lu, Yu-Wing Tai, et al. Learning dual convolutional neural networks for low-level vision. In Proc. of Computer Vision and Pattern Recognition, pages 3070-3079, 2018.
[63] Yanwei Pang, Jing Nic, Jin Xie, Jungong Han, and Xuelong Li. BidNet: Binocular image dehazing without explicit disparity estimation. In Proc. of Computer Vision and Pattern Recognition, pages 5931-5940 2020.
[64] Dubok Park, David K Han, Changwon Jeon, and Hanseok Ko. Fast single image de-hazing using character- isties of RGB channel of foggy image. IEICE Transactions on Information and Systems, 96(8):1793-1)) 2013.
[65] Xu Qin, Zhilin Wang, Yuanchao Bai, Xiaodong Xie, and Huizhu Jia. FFA-Net: Feature fusion attention network for single image dehazing. In Proc. of the AAAI Conference on Artificial Intelligence, pages
11908- 11915, 2020.
[66] Yanyun Qu, Yizi Chen, Jingying Huang, and Yuan Xie. Enhanced pix2pix dehazing network. In Proc. of
Computer Vision and Pattern Recognition, pages 8160-8168, 2019.
[67] Yufu Qu and Zhaofan Zou. Non-sky polarization-base dehazing algorithm for non-specular objects using polarization difference and global scene feature. Optics Express, 25(21):25004-2500 2017. [68] Wenqi Ren, Si Liu, Hua Zhang, Jinshan Pan, Xiaochun Cao, and Ming- -suann Yang. Single image dehazing via multi-scale convolutional neural networks. In Proc. of European Conference on Computer Vision, pages 154-169 101.
2016.
[69] Wenqi Ren, Lin Ma. Jiawei Zhang, Jinshan Pan, Xiaochun Cao, Wei Liu, and Ming-Hsusn Yang. Gated fusion network for single image dehazing. In Proc. of Computer Vision and Pattern Recognition, pages
1553-3261,20 2018
[70] Wenqi Ren. Jinshan Pan. Hua Zhang, Xiaochun Cao, and Ming Hsuan Yang. Single image dehazing via multi-scale convolutional neural networks with holistic edges. International Journal of Computer Vision, :28(1):240-259 2020.
Ming-Hsuan
[71] Olaf Ronneberger, Philipp Fischer, and Thomas Brox U-Net: Convolutional networks for biomedical image segmentation. Proc. of International Conference on Medical Image Computing and Computer Assisted Intervention, pages 244-241,222 2015.
[72] Christos Sakaridis, Dengxin Dai, Simon Hecker, and Luc Van Gool. Model adaptation with synthetic and real data for semantic dense foggy scene understanding, In Proc. of European Conference on Computer Vision, pages 687-704, 2018
[73] Christos Sakaridis. Dengxin Dai, and Lue Van Gool. Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision, 176(9):733999,, 2018
[74] Sanchayan Santra, Ranjan Mondal, and Bhabatosh Chanda. Learning a patch quality comparator for single image dehazing. IEFE Transactions on Image Processing, 259))4598-46 2018.
[75] Lex Schaul, Clément Fredembach, and Sabine Süssstunkk Color image dehazing using the near-infrared. In Proc. of International Conference on Image Processing, pages 1629-1632, 2009.
[76] Yoav Y Schechner and Yuval Averbuch. Regularized image recovery in scattering media. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 29(9):1655 (199:1655-1660 1660,
2007. 2007.
[77] Yoay Y Schechner, Srinivasa G Narasimhan, and Shree K Nayar. Instant dehazing of images using polarization. In Proc. of Computer Vision and Pattern Recognition, volume 1, pages 1-1, 2001.
13

---

[78] Yoav Y Schechner, Srinivasa G Narasimhan, and Shree K Nayar. Polarization-base vision through haze. Applied Optics, 42(3):51 2003.
[79] Yuanjie Shao, Lerenhan Li, Wenqi Ren, Changxin Gao, and Nong Sang. Domain adaptation for image dehazing. In Proc. of Computer Vision and Pattern Recognition, pages 2808-2817, 2020.
[80] Linghao Shen, Yongqiang Zhao, Qunnie Peng, Jonathan Cheung- Wai Chan, and Seong G Kong. An iterative image dehazing method with polarization. IEEE Transactions on Multimedia, 21(5):1993-1107, 2018.
[81] Sarit Shwartz, Einav Namer, and Yoav Schechner. Blind haze separation. In Pruc. of Computer Vision and Pattern Recognition, volume 2. pages 1984-1991, 2006.
[82] Ayush Singh, Ajay Bhave, and Dilip K Prasad. Single image dehazing for variety of haze scenarios using back projected pyramid network. In Proc. of European Conference on Computer Vision Workshops, pages 166-181, 2020.
[83] Chang-Iwan Son and Kiao-Ping Zhang. Near-infrared fusion via color regularization for haze and color distortion removals. IEEE Transactions on Circuits and Systems for Video Technology. 28(11):3 1311133311-31126 2017.
[84] Taeyong Song. Youngjung Kim, Changjae Oh, and Kwanghoon Sohn. Deep network for simultaneous stereo matching and dehazing. In Proc. of British Machine Vision, page 5, 2018.
[85] Patricia L Suárez, Angel D Sappa, Boris X Vintimilla, and Riad I Hammoud. Deep learning based single image dehazing. In Pruc. Computer Vision and Puttern Recognition Workshops, pages 1 1169-1176, 2018. [86] Matan Sulami, Itamar Glatzer, Raanan Fattal, and Mike Werman. Automatic recovery of the atmospheric light in hazy images. In Pruc. of International Conference Cumputational Photogruphy, pages 1-11, 2014.
[87] Robby Tan. Visibility in bad weather from a single image. In Proc. of Computer Vision and Pattern Recognition, pages 1-8, 2008.
[88] Ketan Tang. Jianchao Yang, and Jue Wang. Investigating haze-relevant features a learning framework for image dehazing. In Proc. of Computer Vision and Pattern Recognition, pages 2995-3000, 2014. [89] Jean-Philige Philippe Tarel and Nicolas Hautiere. Fast visibility restoration from a single color gray level image. In Proc. of International Conference on Computer Vision. pages 2201-2208, 2009.
[90] Dmitry Ulyanov, Andrea Vedaldi. and Victor Lempitsky. Instance Normalization: The missing ingredient for fast stylization. arXiv preprint 160 08022, 2016.
[91] Dong Yang and Jian Sun. Proximal Dehaze-Net Net:: A prior learning-bast -based deep network for single image dehazing. In Proc. of European Conference Computer Vision, pages 702-717, 2018.
[92] Xitong Yang. Zheng Xu, and Jiebo Luo. Towards perceptual image dehazing by thysics-based disentangle- ment and adversarial training. In Proc. of the AAAI Conference on Artificial Intelligence, 2018.
[93] He Zhang and Vishal M Patel. Densely connected pyramid dehazing network. In Proc. of Computer Vision and Pattern Recognition. pages 3194-3203, 2018.
[94] He Zhang. Vishwanath Sindagi, and Vishal M Patel. Multi-scale single image dehazing using perceptual pyramid deep network. In Proc. of Computer Vision and Pattern Recognition Workshops, pages 2018.
[95] Jing Zhang and Dacheng Tao. FAMED-Net: A fast and accurate multi-scale end-to-end dehazing network. IEEE Transactions on Image Processing, 29:72 84, 2019.
[96] Yanfu Zhang, Li Ding, and Gaurav Sharma. HAZERD: An outdoor scene dataset and benchmark for single
image dehazing. In Proc. of International Conference on Image Processing, pages 3205 (005-3209,20 2017.
[97] Shiyu Zhao, Lin Zhang. Shuaiyi Huang, Ying Shen, and Shengjie Zhao. Dehazing Evaluation: Real-world benchmark datasets, criteria, and baselines. IEEE Transactions on Image Processing, (6697-6662., 2020. [98] Qingsong Zhu, Jiaming Mai, and Ling Shao. A fast single image haze removal algorithm using color attenuation prior. IEEE Transactions on Image Processing, 24(1 1 3:33222-333, 2015.
14


Reviews:
[{"review_id": "u6ccqCMuIf", "reviewer": "Reviewer_bb8S", "summary": "The paper presents a method for polarization-based dehazing of images. Using a camera with a polarization filter array, multiple polarization images can be captured in a single snapshot. These images are used to estimate an unpolarized image $I$ and a degree-of-polarization image $P$. A series of neural networks are trained on synthetic data to estimate the polarization and radiance of the ambient and transmitted scene light conditioned on $I$ and $P$. The results of the dehazed estimated scene radiance are compared with other methods for dehazing using polarization images or a single unpolarized image. Quantitatively, the approach appears to offer roughly 1-2 dB in improvement over previous methods in terms of PSNR. Qualitatively, the method appears to generate improved results on captured data as well.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper seems to be well written and the method is clearly explained. The results convincingly demonstrate that accounting for polarization in the dehazing task can improve image quality. While the results do not offer an overwhelming improvement over previous techniques for single image dehazing, the improvement is significant. I think the method could inspire future work on polarization-based dehazing and inverse scattering methods.\n\nI have just a few comments which I note below.\n\nI was surprised by the poor performance of the SPCVE method. In their original paper the method appears to offer a good deal of qualitative improvement over the original images whereas the results shown in Figs. 3 and 4 for SPCVE are almost worse than the input images in some cases. What are the main reasons for this difference? Are there situations in which SPCVE is more competitive with the proposed approach?\n\nPart of the motivation for the method is that regions with the same material properties often reflect light with similar polarization properties. I think seeing these properties in captured data would help motivate the method. For example, I would be curious to see the polarization images side by side (rather than in the stacked format of Figure 3, which obscures their differences). I'm also curious to see the intermediate results of the network, such as the predicted degrees of polarization (these could perhaps be included in the supplemental).\n\nHow are the simulation parameters described in section 4.1 chosen?\n\nHow do the predicted degree of polarization images in the captured data compare to the simulated dataset?\n\nFig. 2 typo \\hat{T}->R after Eq. (7)(b)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact appear to be adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "mc8mZPVPjiy", "reviewer": "Reviewer_RnLz", "summary": "This paper proposes a method for image dehazing by polarization-based learning pipeline. It is based on a physical formation model which is considering the spatially variant real-world scattering and taking into account the polarization effects of both transmitted light and airlight. Experiments demonstrate that the proposed method can dehaze robustly with fewer artifacts, and performs better in recovering details.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is a very interesting approach to utilize polarization information of both transmitted light and airlight as a feature for image dehazing. By introducing spatially-variant turbulence N(x,c) from polarization information, the effect of light scattering in the air can be well reflected spatially, and the dehazing results seem to be better. Although the input image data is different, the dehazing results from the proposed method are better than the results from other comparison methods and I believe that the proposed method is able to utilize the polarization information.\n\nOn the other hand, I am afraid that there may be some assumptions in the scene where the proposed method can be applied as the scenes used in the evaluation experiments are similar. I think polarization information can be very useful, but it can also lead to erroneous results due to noise effects. (e.g. the haze in some pixels is very thick and leaves little information of the transmitted light.) It would be better to conduct evaluation experiments of the proposed method for various scenes (depth, normals, colors). I feel that the efficacy of refinement subnetworks (g2 and g4) as well as some networks for polarization information is significant, so it would be good to do more evaluation of refinement subnetworks.\n\nThe paper is well-written and well-structured. There are only a few typos (e.g. ‘dehzaing ’ in Section 6 Conclusion) that can be addressed in another proofreading.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author fully addresses the physical image formation model of hazy images, and mentions that it does not apply to such as fog or mist. The use of polarization information as a feature will have a significant impact as polarization cameras become more widely used.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hour"}, "extra_text": {}}, {"review_id": "kasfeMVC6Ny", "reviewer": "Reviewer_N2H7", "summary": "This paper presents dehazing algorithm using characteristics of polarized images. It first introduces generalized physical formation model of polarized hazy images, regarding spatially-variant real-world scattering. Taking three differently polarized hazy images as input, the proposed algorithm estimates and refines degrees of polarization (DoP) of the transmitted light T and airlight A, and utilizes these predictions to finally obtain original scene radiance R. By doing so, this paper was able to show outstanding performances, especially on real-world images due to its consideration on spatially-variant DoP and global airlight.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The physical formulation and the dehazing pipeline that were proposed in this paper are reasonable, clearly described, and properly executed. The paper is well-written and was easy to follow. While I am satisfied overall, I have a few questions.\n\n1. In refining process for both T and R, the authors claimed that the refinement was done by ‘using semantic and contextual information extracted from I(1, 2, 3)’. While removing these refining subnetworks does drop performance severely (table 2), it seems like the subnetworks are simply using I(1, 2, 3) as inputs and no other technical consideration using architecture or loss function was given. Please elaborate on why do the authors think that this kind of simple process can boost so much of a performance.\n\n2. On top of the mention from the paper, ‘comparing with learning-based dehazing methods might be a bit unfair because of the difference in types of input data’, I think it also restricts practicality and applicability. The most commonly used image capture device is an RGB camera, and these ‘learning-based methods’ can be applied to any hazy image while the proposed algorithm cannot. I want to know the authors’ opinion on why the polarization-based dehazing approaches should be researched.\n\nMinor comments: In figure 2, bottom block, initially estimated R is marked as T^.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I appreciate the authors for mentioning the limitation regarding the generality of the proposed hazy image formation model, for other image degradation weather conditions.", "review_text": "", "extra_scores": {"ethical_concerns": ".", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "OqLTATlngE", "reviewer": "Reviewer_WrNr", "summary": "This paper proposes a neural network for image dehazing where the input is\nexpected to contain polarization information (e.g. multiple polarized images of\nthe exact same scene). The total irradiance and total degree of polarization\ncan be obtained directly from the polarized input frames. The paper then\nfollows the image formation model of [77] (and others) to identify key\nquantities to estimate: the infinite airlight color, and the degree of\npolarization of the ambient and scene light. Multiple neural networks\nare designed to estimate these components; they are coupled using\nclosed-formed expressions derived from the image formation model to produce the\nfinal dehazed image. The networks are trained independently, then finetuned\njointly on a synthetic dataset.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written, consistent and produces good result. Despite its\nclaims, there is not much novelty in the image formation model. Like many\n'physically-based' deep learning image restoration approaches, the proposed\ntechnique replaces heuristic measurements of physical quantities (e.g. a patch\nof sky to estimate the airlight) by neural network estimators.\n\n\n### Strengths\n\n+ I found the paper clear and easy to follow.\n+ The description of prior art appears complete and relevant.\n+ The derivation is sound (although not technically a contribution of this\nwork).\n+ Synthetic test results are convincing and outperform previous single- and\nmulti- frame baselines.\n\n\n### Weaknesses\n\n- The complete derivation of section 3.1 is from [77], this should be better acknowledge.\n- One the claimed contributions is a generalized hazy image formation model,\nbut it is not clear how this paper generalizes the ideas of [77]. The spatially\nvarying noise terms N(x, c) are introduced but never used in the derivation.\n- The real-world results (Fig4) are not convincing. The image suffers from severe\ngrid artifacts, corrupted boundaries, ghosting around buildings. This makes\nme question the \"robustness\" property claimed in the intro (2), it is not evident\nat all that synthetic training data + the physical models are sufficient to\nmake the method practical in the real-world.\n- How do the result degrade visually without the \"refinement\" networks? It\nwould be helpful to have some extra real/synthetic images in supplemental.\n\n\n### Misc\n\n- 268: dehzaing -> dehazing\n- 28: although [...] however -> one suffices\n- 41: for enabling -> to enable\n- paragraph l.41 and l.53 of the intro read as a bit redundant. Maybe a contribution list is not  needed,\nor the previous paragraph can be shortened?\n- 123: divide -> split, decompose\n- 134: consider using different notations for the spatial variation terms N(x, c).\nI found it confusing to see it appear in theta, beta and A_\\inf\n- 201: Is Gaussian noise really a good model for the beta and alpha_inf\nspatial variations? This choice seems a bit ad hoc\n- The results from SPCVE seem very noisy\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper shows and discuss the limitations of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "None that struck me.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The four reviewers thought this paper was above threshold for acceptance. They all found the idea useful and interesting. The author response also helped to clarify some issues raised by the reviewer.
Decision: Accept (Poster)