Image dehazing method via a cycle generative adversarial network

IET Image ProcessingVolume 14, Issue 16 p. 4240-4247 Research ArticleFree Access Image dehazing method via a cycle generative adversarial network Chao-Yue Zhao, Chao-Yue Zhao College of Computer Science and Engineering, Shandong University of Science and Technology Qingdao 266590, People's Republic of ChinaSearch for more papers by this authorRui-Sheng Jia, Corresponding Author Rui-Sheng Jia jrs716@163.com College of Computer Science and Engineering, Shandong University of Science and Technology Qingdao 266590, People's Republic of China Shandong Province Key Laboratory of Wisdom Mine Information Technology, Shandong University of Science and Technology, Qingdao, 266590 People's Republic of ChinaSearch for more papers by this authorQing-Ming Liu, Qing-Ming Liu College of Computer Science and Engineering, Shandong University of Science and Technology Qingdao 266590, People's Republic of ChinaSearch for more papers by this authorHong-Mei Sun, Hong-Mei Sun College of Computer Science and Engineering, Shandong University of Science and Technology Qingdao 266590, People's Republic of China Shandong Province Key Laboratory of Wisdom Mine Information Technology, Shandong University of Science and Technology, Qingdao, 266590 People's Republic of ChinaSearch for more papers by this authorHai-Bin Sun, Hai-Bin Sun College of Computer Science and Engineering, Shandong University of Science and Technology Qingdao 266590, People's Republic of China Shandong Province Key Laboratory of Wisdom Mine Information Technology, Shandong University of Science and Technology, Qingdao, 266590 People's Republic of ChinaSearch for more papers by this author Chao-Yue Zhao, Chao-Yue Zhao College of Computer Science and Engineering, Shandong University of Science and Technology Qingdao 266590, People's Republic of ChinaSearch for more papers by this authorRui-Sheng Jia, Corresponding Author Rui-Sheng Jia jrs716@163.com College of Computer Science and Engineering, Shandong University of Science and Technology Qingdao 266590, People's Republic of China Shandong Province Key Laboratory of Wisdom Mine Information Technology, Shandong University of Science and Technology, Qingdao, 266590 People's Republic of ChinaSearch for more papers by this authorQing-Ming Liu, Qing-Ming Liu College of Computer Science and Engineering, Shandong University of Science and Technology Qingdao 266590, People's Republic of ChinaSearch for more papers by this authorHong-Mei Sun, Hong-Mei Sun College of Computer Science and Engineering, Shandong University of Science and Technology Qingdao 266590, People's Republic of China Shandong Province Key Laboratory of Wisdom Mine Information Technology, Shandong University of Science and Technology, Qingdao, 266590 People's Republic of ChinaSearch for more papers by this authorHai-Bin Sun, Hai-Bin Sun College of Computer Science and Engineering, Shandong University of Science and Technology Qingdao 266590, People's Republic of China Shandong Province Key Laboratory of Wisdom Mine Information Technology, Shandong University of Science and Technology, Qingdao, 266590 People's Republic of ChinaSearch for more papers by this author First published: 23 February 2021 https://doi.org/10.1049/iet-ipr.2020.0928Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Aiming at the problems of colour distortion and residual dehazing in the existing image dehazing methods when processing outdoor images, a method of image dehazing based on the cycle generative adversarial network is proposed. Taking the cycle generative adversarial network as the overall framework of the model, firstly, the neural network is trained to obtain the mapping relationship between haze images and haze-free images. Secondly, to accelerate the convergence speed of network training, using residual structure to improve network stability and reduce parameters. Then, a new loss function is proposed, fusing the Wasserstein distance into adversarial loss and cycle consistency loss, which reduces the deviation between the generated dehaze image and the real haze-free image, and alleviating the problems of colour distortion and haze removal residue. Finally, use the optimised bounded ReLU (BReLU) activation function instead of the original activation function to improve the transmittance reflecting the haze depth information. The experimental results demonstrated that, compared with the comparison method, the proposed method improves the peak signal-to-noise ratio, structure similarity, information entropy, and average gradient, and has an achieved better performance in dehazing. 1 Introduction The haze is a common natural weather phenomenon, as illustrated in Fig. 1. In the haze scene, various aerosol particles suspended in the air cause large scattering and refraction to the light, as a result, the image quality obtained by various imaging systems is seriously degraded, and the visual effect of the image is seriously affected. The primary degradation is that the image colour is severely attenuated, the colour saturation and contrast are reduced, and the details are not prominent enough. This is not conducive to the extraction and recognition of image features and has a severe impact on the fields of aerial photography, intelligent driving, urban monitoring, satellite remote sensing, video monitoring, target recognition, field exploration, etc. [[1]]. Therefore, how to dehaze the outdoor degraded image obtained in the haze scene to improve the quality of image visualisation has important practical significance and application prospects [[2]]. Fig. 1Open in figure viewerPowerPoint Visual effect of hazy weather The traditional single image dehazing methods are mainly divided into two categories: image enhancement methods-based on a non-physical model [[3]-[7]] and image restoration methods-based on a physical model [[8]-[10]]. Among them, the image enhancement method based on the non-physical model mainly enhances the overall or local characteristics of the degraded image, improves the saturation and contrast of vision, to improve the quality of an image to a certain extent to achieve the purpose of dehazing. For example, the histogram equalisation method [[5]], the Retinex [[6]] method, the homomorphic filtering method, the wavelet transforms method [[7]], etc. The formation of haze has a certain physical theory on the basis, its influence on image quality is also improved with the increase of distance. Although this kind of method can enhance the definition of the image, it does not study the physical model of haze image degradation, make it easy to distort some image information. The physical model-based image restoration method is based on the physical model of atmospheric scattering. With a set of constraints based on prior knowledge, clear images are obtained by solving the physical model, the degradation mechanism of haze images is studied, and a universal degradation model is established. In 2009, He et al. [[10]] proposed a dark channel prior (DCP) algorithm to estimate the global atmospheric light and the initial transmission image. After that, a soft matting or guided filtering [[11]] is used to optimise the image to recover the haze-free image. However, this method overestimates the haze concentration, resulting in the overall brightness of the image being dimmed after dehazing, and distortion often occurs in the sky area. Recently, more and more researchers apply the deep learning method to the field of image dehazing [[12]-[16]]. For example, Cai et al. [[12]] have designed DehazeNet. DehazeNet uses the haze imaging model and a large number of clear images to generate haze images, and a convolutional neural network [[13], [14]] is used to learn the characteristics of haze images to estimate the transmissivity map. Obtain the algorithm has achieved a good dehazing effect. However, due to the limitation of selecting a patch-based training data, there are some phenomena of colour distortion and an incomplete dehazing in some haze image processing. Similar to DehazeNet, Ren et al. [[15]] proposed a single image dehazing method via multi-scale convolutional neural networks (MSCNNs), which is composed of a rough network and a refined network. The former estimates the rough transmissivity map, and the latter refines it locally to get the final transmissivity map. In the process of transmission learning, the two deep learning algorithms ignore the difference between the structure information and texture information of the haze image, which leads to blindness in the learning process, and the pool operation used in the model will cause information loss and other problems. In 2017, Li et al. [[16]] proposed an image dehazing model designed based on the reconstituted atmospheric scattering model, called All-in-one dehazing network (AOD-Net). AOD-Net realises end-to-end training, which directly generates clear images from the haze images, rather than relying on any single and intermediate parameter estimation steps. Compared with the previous algorithm, the dehazing effect is improved, but still exists colour distortion and other problems. Since 2014, generative adversarial nets (GANs) [[17]] have become a research hotspot in deep learning. Scholars began to use generative adversarial networks to fill and repair images. In 2017, Isola et al. [[18]] proposed an image-to-image translation with conditional adversarial networks (Pix2Pix). Pix2pix can transform haze image and haze-free image by training the pairs of the haze image and the haze-free image, but it is very difficult to get the natural haze-free image corresponding to the natural haze image in the real scene. To solve the above problems, our paper designs an image dehazing method based on the cycle generative adversarial network. The non-paired haze image and haze-free image are put into the cycle generative adversarial network for training, which solves the problem that the real paired image set is challenging to obtain. To solve the problem of slow network convergence, a residual structure is added to the generated network, which stabilises the training process and speeds up the network convergence. In addition, we optimised the loss function and the activation function to reduce the deviation between the generated dehaze image and the real haze-free image, thereby reducing image distortion and dehazing residual problems, and finally making the image colour more natural and dehazing thorough. 2 Related work 2.1 Physical model of atmospheric scattering We use the physical model of atmospheric scattering to simulate the formation process of haze degraded image. The physical model of atmospheric scattering is demonstrated in Fig. 2, which is an essential basis for restoring clear images [[19], [20]]. As illustrated in Fig. 2, the atmospheric scattering is mainly divided into two types: one is that the sunlight encounters haze when passing through the atmosphere, scatters and diffuses, and some of the solar radiation cannot reach the ground. Also the other is the attenuation effect of the reflected light from the target object when passing through the haze layer. In Fig. 2, the attenuation effect of the reflected light from the target object passing through the haze layer, the view formed by the scattering effect of the sunlight by the haze and the scattering effect of the object itself cause the contrast of the scene image captured by the visual imaging system to decrease. The attenuation of the colour and resolution is obvious, and the imaging result is ambiguous, thus forming the haze image. The mathematical formula can be expressed as I x = J x t x + A 1 − t x (1) Fig. 2Open in figure viewerPowerPoint Physical model of atmospheric scattering Among them, I x is the haze image obtained for the imaging equipment, J x is the haze-free image to be recovered, A is the global atmospheric light value, t x is the scene transmittance, J x t x is the incident light attenuation model, which describes the light weakening process from the scene point to the observation point; A 1 − t x is the atmospheric light imaging model, which describes the influence of various kinds of light in the surrounding environment on the light received by the observation point due to the scattering effect of atmospheric particles. In addition, t x can be expressed as t x = exp − β d x (2) where β is the medium extinction coefficient, d x is the distance from the object to the imaging system, i.e. the scene depth. To clarify the process of restoring a clear image J x , formula (1) is transformed into the following formula: J x = I x − A 1 − t x t x (3) For formula (3), it is known that the condition is a haze image I x , and the haze-free image J x needs to be solved. According to algebraic knowledge, this is an ill-posed equation, which can only be solved on the basis of the prior information of scene transmittance t x and global atmospheric light A. 2.2 CycleGAN In the field of image dehazing, the traditional neural network training needs enough positive and negative samples, i.e. the same scene with haze image and without haze image, and this kind of data is challenging to obtain. Hence, researchers use a large number of synthetic haze images as training samples. The synthetic haze map has a strong dependence on data one-to-one matching, which makes a series of deep learning models have limited adaptability. In 2017, the unpaid image-to-image translation using cycle consistent alternative networks (CycleGANs) [[21]] proposed to solve this problem. CycleGAN realises training with non-paired haze images and haze-free images by connecting two unidirectional GANs head to tail to form a cycle. As illustrated in Fig. 3, (a) the model contains two mapping functions G: X→Y and F: Y→X and associated adversarial discriminators DY and DX. To further regularise the mapping, we introduce two-cycle consistency losses to realise the transformation from one domain to another and then back to where we started. (b) Forward cycle-consistency loss: x→G(x)→F(G(x)) ≃ x, and (c) backward cycle-consistency loss: y→F(y)→G(F(y)) = y. Fig. 3Open in figure viewerPowerPoint Schematic of CycleGAN In this paper, the Cycle-GAN principle is applied to the field of dehazing, the defocusing problem caused by depth is not considered, and the transformation from haze image to haze-free image is realised by using non-paired image training, which reduces the dependence on data matching, but there are still problems of unstable training and image distortion. To solve the problem of slow network convergence, uses the residual structure to stabilise the training process and accelerate the convergence speed. Secondly, we use the loss function combined with the Wasserstein distance which reduces the deviation between the generated dehaze image and the real haze-free image, and use the optimised BReLU (bounded ReLU) activation function instead of the original activation function to improve the transmittance reflecting the haze depth information, and finally, make the image dehazing more thorough and the colour more self-contained. 3 Proposed method 3.1 Proposed network To better retain the image colour and solve the problem of unstable training while dehazing the image, we propose an image dehazing method based on the cycle generative adversarial network, the network consists of two unidirectional GANs connected head to tail, sharing two generators, which are recorded as G and F, respectively. G and F each have a discriminator, which are recorded as DX and Dy, respectively. As illustrated in Fig. 3, the network workflow is mainly divided into two parts: the transformation from haze to clear image and the transformation from clear to haze image. In the conversion process, G and F are used to generate haze-free image and haze image, respectively. Dy and Dx are used for adversarial training, judging the authenticity of a haze-free image and clean image, and promoting generator G to generate the high-quality target domain image. Through continuous confrontation training, adjusting the weight parameters, the generated image is clearer and the dehaze is more thorough, to realise the transformation from the haze image to the haze-free image. The overall network structure through the cycle loss Lcyc between X and F(G(X)) and Y and G(F(Y)), makes the two generative adversarial networks to be able to cooperate and optimise, so that the generator G can learn the distribution characteristics of the haze-free image more fully, and achieve a better dehazing effect. 3.1.1 Generator The generators G and F are one of the essential parts of the network model. They are responsible for generating haze images and haze-free images. As illustrated in Fig. 4, the generator used in this paper consists of encoding, transformation and decoding. Transformation: Image feature conversion is carried out by combining the various features of the image. The conversion process adopts the residual structure, which consists of nine residual blocks and a residual block is composed of two convolution layers. In the residual structure, part of the input data is directly added to the output, ensuring that the input data information of the previous network layer directly acts on the subsequent network layer, so that the deviation of the corresponding output from the original input is reduced, and the characteristics of the original image are retained. The structure of the Resnet module is shown in Fig. 5. Fig. 4Open in figure viewerPowerPoint Structure of the generator. The convolutional layers are represented as '(number of filters) × (kernel size) × (kernel size) × (stride)' Encoding: using a convolutional neural network to extract features from the input image. The conversion process uses a residual structure, the encoder is composed of three rolls up layers, three batch normalisation layers, and three BReLU activation layers Fig. 5Open in figure viewerPowerPoint Structure of residual block Decoding: The work of restoring low-level features from the feature vector is completed by using the deconvolution, and finally, the generated image is obtained. The decoder consists of two anti-convolution layers, two batch normalisation layers, two BReLU activation layers, and one hyperbolic tangent activation layer. 3.1.2 Discriminator The discriminators DX and Dy are another important part of the network model. They are responsible for judging the haze-free image and the real haze-free image. Its structure is illustrated in Fig. 6. The discriminator consists of five convolution layers, four BReLU activation layers, and three batch normalisation layers. Compared with the texture information of the original image, the texture information of the enhanced image can be clearer by using PatchGANs [[22]] in the Pix2Pix discriminator. Fig. 6Open in figure viewerPowerPoint Structure of discriminator In the network model of this paper, except that no BReLU activation function is used in the decoder, the rest uses the BReLU activation function [[23]], the function expression is BReLU x = 0 x < 0 x 0 ≤ x ≤ n n x > n (4) The value of n can be adjusted, we call it bound bounded ReLU-n, and we usually use BReLU-6. Compared with the traditional sigmoid function, BReLU is easier to learn and optimise and faster to operate. Compared with the ReLU function, BReLU avoids the phenomenon of complete loss of negative axis information, so we chose leaky ReLU as the activation function. Also, the accuracy of transmittance can be improved by using the BReLU activation function so that the image can be dehazed more thoroughly, and the image can be clearer after dehazing. 3.2 Loss function Our goal is to learn mapping functions between two domains X and Y given training samples x i i = 1 N , where x i ∈ X and y j j = 1 N , where y j ∈ Y. We denote the data distribution as x ~ pdata(x) and y ~ pdata(y). As illustrated in Fig. 3a, we can know the mapping relationship, to further regularise the mappings, we introduce two types of terms: adversarial losses for and cycle consistency losses. 3.2.1 Adversarial loss We apply adversarial losses [[17]] to match the distribution of generated images to the data distribution in the target domain, we express the objective as L GAN G , D Y , X , Y = E y ∼ p data y log ⁡ D Y y + E x ∼ p data x log 1 − D Y G x (5) where G tries to generate images G(x) that look similar to images from domain Y, while Dy aims to distinguish between translated samples G(x) and real samples y. G aims to minimise this objective against an adversary D that tries to maximise it, i.e. min G max D Y L GAN G , D Y , X , Y . We introduce a similar adversarial loss for the mapping function F: Y→X and its discriminator DX as well: i.e. min F max D X L GAN F , D X , Y , X . To make the loss more stable during training and generates higher quality results, we replace the negative log-likelihood objective by a least-squares loss [[24]]. So, for the loss L GAN G , D , X , Y , we train the G to minimise E x ∼ p data x D G x − 1 2 and train the D to minimise E y ∼ p data y D y − 1 2 + E x ∼ p data x D G x 2 . To make dehazing more thorough, the image after dehazing is more natural, the Wasserstein distance is introduced into the loss function of the discriminator as the regular term [[25], [26]], which is defined as follows: W P r , P g = inf γ ∼ ∏ P r , P g ⁡ E x , y ∼ γ x − y (6) where Pr is the real distribution and Pg is the generated distribution, ∏ P r , P g is the set of all joint distributions of Pr and Pg. X is the real sample, Y is the generated sample, for each joint distribution, get X and Y from sampling x , y ∼ γ. In the process of discriminator optimisation, the loss function of the model generator is approximately equivalent to minimising the JS distance between Pg and Pr. In the gradient descent method, the JS distance cannot provide sufficient gradient for network training, so the Wasserstein distance is used as a regular term to guide network training. Compared with a JS distance, the Wasserstein distance has an excellent smoothing property, which can effectively solve the problem of gradient disappearance. At the same time, Wasserstein distance is also used as a training index to judge whether the training process meets the requirements. During training, the generator minimises the Wasserstein distance between Pr and Pg. That is to say, make the generated distribution approach the real delivery, and finally make the generated haze-free image more real, which is defined as follows: L GAN G , D Y , X , Y = E y ∼ p data y D y − 1 2 + E x ∼ p data x D G x 2 + α E x ∼ p x ∧ ∇ x D x p − 1 2 (7) where the third term to the right of the equal sign is the gradient penalty term. P x ^ is subject to the distribution of x ^ , ∇ x D x p is the p norm of the gradient of the discriminator D, P = 2 is taken and the default value of a super parameter is 10 in this paper. See the following formula: x ^ = ε x r + 1 − ε G x g (8) where xr represents the real sample, xg represents the generated sample, and the parameters ε obey the uniform distribution on (0, 1). 3.2.2 Cycle consistency loss We apply the cycle consistency loss to prevent the learned mappings G and F from contradicting each other. As illustrated in Figs. 3b and c, there are two cycle loss functions in our network model, which are forward cycle loss and backward cycle loss: x → G x → F G x ≃ x(9) y → F y → G F y ≃ y(10) Adopt L1 loss, i.e. L cyc G , F = E x ∼ p data x F G x − x 1 + E y ∼ p data y G F y − y 1 (11) 3.2.3 Full objective Our full objective is L G , F , D X , D Y = L GAN G , D Y , X , Y + L GAN F , D X , X , Y + λ L cyc G , F (12) where the first two terms on the right side of the equation are confrontation loss, the last one is cycle loss and λ is the proportional coefficient. Our ultimate goal is to solve G ∗ , F ∗ , i.e. G ∗ , F ∗ = arg min G , F max D X , D Y ⁡ L G , F , D X , D Y (13) 3.3 Algorithm flow In this paper, the RMSProp algorithm is selected to optimise the parameters of the network model to achieve a better demisting effect. This model uses the idea of confrontation training. Firstly, we fixed the generator, discriminator, to train neritic times, and limits the parameters of the discriminator to [−c, c] each time. Then we set discriminator, training generator. In the training process, batch processing is adopted. The specific parameter setting and training process are illustrated in Fig. 7. 4 Experiments Our experiments are all carried out on an Intel (R) Core (TM) i5-8300h CPU @ 2.3 GHz, 8 GB RAM computer, and accelerated by an NVIDIA GTX 1060 GPU. The neural network model is built with the Tensorflow framework, MATLAB R2017a, and JetBrains PyCharm 2017.1x64. In the process of training, we use the stochastic gradient descent (SGD), set the initial learning rate α = 0.0002, set the clipping parameter c = 0.01, and set the number of iterations of the model to 200, then save the network model every five iterations. Fig. 7Open in figure viewerPowerPoint Algorithm flow 4.1 Dataset In this paper, the natural haze images and natural haze-free images are all from the Internet screening, then use the Brenner gradient function to calculate the Brenner value of the image, images with a value below 250 are classified as a haze image, and the rest as a haze-free image. After screening, 1000 natural haze images and 1000 haze-free images are selected. 700 natural haze images, and 1000 natural haze-free images are selected as training sets, and the remaining 300 natural haze images are chosen as Test set I. To verify the effectiveness of the method in this paper, the composite haze images in the O-hazy data set are added as Test set II for testing, and the O-hazy data set comes from the NTIRE 2018 challenge on the single-image dehazing data set. The selected pictures are cropped to the 256*256 size required by the model, the haze images in the O-hazy data set are all obtained from the natural haze-free image through artificial hazing, some data set images are illustrated in Fig. 8. 4.2 Evaluation index In the experiment, we use the following five indices indexesto evaluate the image reconstruction algorithm: (i) peak signal-to-noise ratio (PSNR) [[24]], (ii) structural similarity (SSIM) index [[27]], (iii) information entropy (IE) [[28]], (iv) average gradient (AG), and [[28]] (v) algorithm time. The evaluation indices (1) and (2) are used to verify the effectiveness of the model in this paper. PSNR is commonly used to evaluate the quality of an image after compression compared with the original image. The higher the PSNR [[24]] is, the smaller the distortion after compression is. SSIM [[27]] is a kind of full-reference image quality evaluation index. The larger the value of SSIM [[1], [29]], the smaller the image distortion. The evaluation indices (3) and (4) are used to verify the effectiveness of the model for natural haze images. The IE reflects the amount of information contained in the image, the larger the entropy, the clearer the image. The AG indicates the rate of the change of the tiny detail contrast of the image, the larger the average gradient, the clearer the image level. Fig. 8Open in figure viewerPowerPoint Some images of data set 4.3 Results and analysis In order to verify the dehazing effect of our method, we compare this method with other image dehazing methods, including Image dehazing method (CLAHE [[29]]) based on image enhancement, Image dehazing method based on image restoration (DCP [[10]]) and image dehazing method based on deep learning (Dehazenet [[12]], AOD-Net [[16]], CycleGAN [[21]]). The experiments are carried out on the synthetic haze image Test set II and the natural haze image Test set I, respectively. The experimental results are illustrated in Table 1. As illustrated in Table 1, we use three evaluation indexes: PSNR/dB, SSIM, and operation time to compare the dehazing effect of synthetic haze images. The results in Table 1 show that our method has the best dehazing impact on the synthetic haze image data set, PSNR and SSIM are higher than other comparison methods, and our methods operation time is lower than other methods. For the natural haze image data set, we use three evaluation indexes: IE, AG, and operation time to compare the dehazing effect. The results in Table 1 show that the IE and AG values of the image after dehazing, our method is hig