Image defogging based on amended dark channel prior and 4‐directional L <sub>1</sub> regularisation

IET Image ProcessingVolume 15, Issue 11 p. 2454-2477 ORIGINAL RESEARCH PAPEROpen Access Image defogging based on amended dark channel prior and 4-directional L1 regularisation Yuliang Yang, Corresponding Author Yuliang Yang 714788505@qq.com orcid.org/0000-0002-4354-6869 School of Mechanical Engineering, Sichuan University, Yihuan road, Chengdu, China Shaanxi Institute of Technology, No.8 Renmin road Huyi District, Xian, China Correspondence Yuliang Yang, School of Mechanical Engineering, Sichuan University, Yihuan road, Chengdu, 610065, China. Email: 714788505@qq.comSearch for more papers by this authorWei Long, Wei Long School of Mechanical Engineering, Sichuan University, Yihuan road, Chengdu, ChinaSearch for more papers by this authorYanyan Li, Yanyan Li School of Mechanical Engineering, Sichuan University, Yihuan road, Chengdu, ChinaSearch for more papers by this authorXiaoqiu Shi, Xiaoqiu Shi School of Manufacturing Science and Engineering, Southwest University of Science and Technology, Mianyang, ChinaSearch for more papers by this authorLin Gao, Lin Gao School of Information Engineering, Hubei Minzu University, Enshi, ChinaSearch for more papers by this author Yuliang Yang, Corresponding Author Yuliang Yang 714788505@qq.com orcid.org/0000-0002-4354-6869 School of Mechanical Engineering, Sichuan University, Yihuan road, Chengdu, China Shaanxi Institute of Technology, No.8 Renmin road Huyi District, Xian, China Correspondence Yuliang Yang, School of Mechanical Engineering, Sichuan University, Yihuan road, Chengdu, 610065, China. Email: 714788505@qq.comSearch for more papers by this authorWei Long, Wei Long School of Mechanical Engineering, Sichuan University, Yihuan road, Chengdu, ChinaSearch for more papers by this authorYanyan Li, Yanyan Li School of Mechanical Engineering, Sichuan University, Yihuan road, Chengdu, ChinaSearch for more papers by this authorXiaoqiu Shi, Xiaoqiu Shi School of Manufacturing Science and Engineering, Southwest University of Science and Technology, Mianyang, ChinaSearch for more papers by this authorLin Gao, Lin Gao School of Information Engineering, Hubei Minzu University, Enshi, ChinaSearch for more papers by this author First published: 26 April 2021 https://doi.org/10.1049/ipr2.12233AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract The dark channel prior (DCP) algorithm has been widely used in the field of image defogging because of its simple theory and clear restoration result. However, the DCP algorithm has significant limitations. This study clarifies the relationship between halo artfacts and the size of the dark channel patch of the DCP algorithm and analyses the reason why the colour of close-range white objects appears distorted in the restored images. An amended DCP method is then proposed to solve these problems, utilising a locally variable weighted 4-directional L1 regularisation and a corresponding parallel algorithm to optimise the transmission. A deep neural network, 4DL1R-net, is then trained to further enhance the processing speed. Extensive experiments demonstrate that this method is effective. The proposed method can obtain clear details, maintain the natural clarity of images, and achieve significant improvements over state-of-the-art methods. 1 INTRODUCTION In recent years, the quality of outdoor digital images has been seriously affected because the occurrence of foggy or hazy weather has increased. In many fields, including highway traffic monitoring, intelligent driving, and surveillance systems of train stations, this problem is more obvious and affects the safety of human lives and property. Moreover, foggy images affect the performance of many high-level computer vision tasks, such as object detection and recognition. Defogging is, thus, becoming an increasingly desirable technique for both computational photography and computer vision tasks, and it has been extensively studied. Many studies [1-37] have made significant progress in the single-image defogging field. The early approaches focused on image enhancement [6-9], whereas the later approaches developed handcrafted features based on the statistics of clear images, such as Colour-Lines [15], DCP [16], and boundary constraint [29], methods. In the field of computer vision, the model most widely used to describe the formation of a hazy image is as follows: I x = J x t x + A 1 − t x , (1)where I(x) is the foggy image, and J(x) is the theoretically clear image. t ( x ) = e − β d ( x ) represents the transmission under foggy conditions and attenuates exponentially with the depth of field of the scene. A is the value of atmospheric light. Equation (1) indicates that image defogging is an ill-posed inverse problem without the knowledge of J(x), A, and t(x). Thus, various image priors have been explored to evaluate transmission for single-image defogging tasks [15-32]. Each of these approaches has advantages and disadvantages. Among them, the dark channel prior (DCP) method is simple and effective and can obtain high-quality restored images; thus, it is widely used in practical applications. The theory is as follows: min y ∈ Ω ( x ) ( min C ∈ { R , G , B } I C ( y ) A ) = min y ∈ Ω ( x ) ( min C ∈ { R , G , B } ( J C ( y ) A ) ) × t ( x ) + 1 − t ( x ) , (2)where min y ∈ Ω ( x ) ( min C ∈ { R , G , B } ( I C ( y ) A ) ) and min y ∈ Ω ( x ) ( min C ∈ { R , G , B } ( J C ( y ) A ) ) are, respectively, the minimum values of the dark channels of I(x) and J(x) in the local patch, Ω ( x ) . In most local image regions that do not cover the sky, some pixels very often have considerably low intensity in at least one colour (RGB) channel [16]. In other words, min y ∈ Ω ( x ) min C ∈ { R , G , B } J C ( y ) A is close to zero. Thus, transmission t(x) is approximately expressed as t x = 1 − min y ∈ Ω x min C ∈ R , G , B I C y A . (3)Although the DCP algorithm is undoubtedly successful, we found, through several experiments, that it has several shortcomings. First, halo artfacts occur because of the different sizes of patch Ω ( x ) . Completely eliminating this phenomenon using soft matting, guided filters, or other transmission optimising methods is difficult. Second, this method will fail if there are white close-range objects in the foggy image. Third, the recovered images are usually much darker than the original images. We analyse the causes of these problems and the relevant improvement methods in Section 3. Recently, some scholars proposed the use of convolutional neural networks (CNNs) [31-37] to automatically learn feature representations for foggy images. These algorithms [31-37] have shown exciting results because of CNN's strong learning ability. However, these scholars focused on improving the structure of CNNs and neglected the fact that a significant number of natural foggy images and corresponding clear samples is almost impossible to obtain. During training, a large number of artificial foggy images were used. These images are synthesised by using clear images of sunny weather; their atmospheric light value is far from the actual atmospheric light values found in foggy weather. Some scholars used indoor data sets [31, 32]; however, the illumination of indoor images is uneven because of local light sources and is completely different from the atmospheric light in outdoor foggy weather. Moreover, owing to the limitations of current sensor technology, there is no reasonable way to simulate fog concentration distribution through the image depth of field in all outdoor datasets. These problems lead to the unsatisfactory generalisation performance of the current CNNs on natural foggy images, especially dense foggy images. Based on this analysis, we integrate the advantages of conventional methods and of CNNs to solve such problems. We propose an amended dark channel prior (ADCP) to improve the DCP algorithm and a locally-variable weighted 4-directional L1 regularisation to optimise the transmission and train a lightweight CNN, named 4DL1R-net (4-directional L1 regularisation), to further enhance the processing speed. Extensive experiments show that our method is effective and can achieve significant improvements over state-of-the-art methods. Our contributions are summarised as follows: We propose an ADCP algorithm to amend the dark channel value pixel-wise. This method is efficient and novel. We utilise the 4-directional L1 regularisation to optimise the transmission map and propose an approximation algorithm and a parallel method to solve this complex variational problem. To our knowledge, this is the first time that the 4-directional L1 regularisation has been applied in image edge preserving filtering. This is the first study in which the parallel method is effectively employed to solve the 4-directional variational problem. We train a deep learning network (4DL1R-net) to further enhance the computation speed; thus, our processing speed is the highest among those of current defogging algorithms (without using GPUs). 2 RELATED WORK In this section, we briefly review some important defogging algorithms. At present, the mainstream image defogging algorithms can be roughly divided into three categories. The first type of method is based on image enhancement [6–9, 18, 38-41]. These methods can be represented by the Retinex method [6–8, 38–41]. This method has a fast processing speed and a better enhancement effect overall on dark original images, but its restored image has colour distortion and image desaturation. Tan et al. [9] proposed a local contrast-maximising method based on a Markov random field under the prior condition that the local contrast of the clear image is much higher than that of the hazy image. This method is essentially the contrast stretching enhancement algorithm, and its result is prone to over-saturation of the restored image. The second type of method is based on the Atmospheric Scattering Model [4, 5, 13-16], which has been adopted by many scholars. The ideas of Narasimhan and Nayar [10-12] are ground-breaking here and primarily based on this model. Their restored images are bright and have higher contrast, but it is necessary to obtain images of the same scene under different weather conditions for comparison. It is very difficult to acquire such images under different weather conditions. Tarel et al. [13] used a median filter to estimate the dissipation function; however, the median filter cannot preserve the objects’ edges well, resulting in a small amount of fog remaining in the depth mutation of the restored image. Fattal [14, 15] proposed the theory of defogging by colour-Lines, which requires images with high colour contrast; hence, it is only suitable for mist images. He et al. [16] obtained the dark channel rule using the statistics of a large number of outdoor images, and proposed the dark channel prior (DCP) method. Meng et al. [29] proposed an effective regularisation defogging method to restore defogged images by exploring the inherent boundary constraint, but this approach is time-consuming because it requires multiple iterations. The third type of method is based on data-driven approaches [31-43]. Some scholars have proposed algorithms based on machine learning [42, 43] or deep learning (CNNs) [31-37], and these algorithms have been highly successful. In [42], Qingsong Zhu et al. adopted the colour attenuation prior and used a supervised learning method to learn the depth information. They then estimated the transmission map using depth information and finally obtained the restored image. In [43], Ketan Tang et al. investigated a variety of haze-relevant features in a regression framework based on the random forest algorithm and used a large number of synthetic hazy image patches to train this regression framework. Recently, deep learning methods [31-37] have been adopted in image defogging. Inspired by haze-relevant features, Cai et al. [31] proposed a trainable end-to-end CNN named DehazeNet for estimating the transmission map, with specially designed feature extraction layers. Ren et al. [32] proposed a multi-scale convolutional neural network (MSCNN) for the transmission estimation of foggy images. This work includes two parts: a coarse-scale network to learn a preliminary transmission map and a fine-scale network to refine this map. Li et al. [35] proposed AOD-Net to avoid separately estimating the transmission and atmospheric light values. The AOD-Net can directly generate a restored image through a lightweight CNN. More recently, Yang D et al. [33] proposed a deep learning network based on a dark channel prior and a multi-iteration algorithm. It can be applied to many situations such as underwater image enhancement and night-time image defogging. However, a guided filter is still used in post-processing, which leads to low processing efficiency. Ren W et al. [36] proposed a gated fusion network, which uses three inputs from an original hazy image by applying white balance (WB), contrast enhancement (CE), and gamma correction (GC). They adopted a multi-scale approach to avoid halo artfacts. This training process is time-consuming and easy to overfit. Liu et al. [37] proposed GridDehazeNet, which is a type of mesh structure consisting of pre-processing, backbone, and post-processing. This structure has excessive amounts of layers and blocks, and has difficulty achieving real-time processing results. These methods [32, 34-37] use CNNs to learn a mapping from an input hazy image to a transmission or haze-free image, but they do not consider haze-related priors to constrain the mapping space compared with traditional methods. 3 ADCP AND 4-DIRECTIONAL L1 REGULARISATION METHOD In this section, we analyse the problems existing in DCP and propose the amended dark channel prior. We propose a locally-variable weighted 4-directional L1 regularisation to optimise transmission and utilise the parallel method to solve this complex variational problem. 3.1 Limitations of DCP 3.1.1 Halo artifacts The causes of halo artifacts for various defogging methods differ, and thus there are different ways to eliminate them. Regarding the DCP algorithm, we know that the halo artifacts are related to the size of the patch in the dark channel map. After many experiments, we found that using a guided filter cannot eliminate these halo artifacts in many foggy scenes, especially for those scenes where the colour difference between close-range objects and the background is obvious and the size of path Ω(x) is larger than 5 × 5. The samples are shown below in Figures 1 and 2. FIGURE 1Open in figure viewerPowerPoint Halo artifacts from different sizes of patch Ω(x) in the dark channel maps The images in row (I) are the restored images; those in row (II) are the dark channel maps. (a) Size of the patch is 20 × 20, (b) size of the patch is 10 × 10, and (c) size of the patch is 1 × 1 FIGURE 2Open in figure viewerPowerPoint Halo artifacts for different sizes of patch Ω(x) in the dark channel map. The images in row (I) are the restored images; those in row (II) are the dark channel maps. (a) Size of the patch is 20 × 20, (b) size of the patch is 10 × 10, and (c) size of patch is 1 × 1 The halo artifacts of the DCP algorithm can be divided into two types: (1) These images are mainly close-range objects. If the size of patch Ω(x) in the dark channel map is larger, the background portion, which should have a high brightness value, will take the minimum brightness value of the Ω(x) region near the edges of the objects. The patch is the Ω(x) region in Equation (2). Therefore, the background is regarded as the object, and the final result is that there is an expansion phenomenon around the close-range objects in the dark channel map. The degree of expansion depends on the size of patch Ω(x). Because the background or sky portions are regarded as objects, the high value of this portion is replaced by the low value of the objects in the dark channel map. Therefore, the pixel value of this portion exceeds the normal range for standard images (RGB) when image restoration calculation is executed. Therefore, the colour around the objects is destroyed and a white colour (no colour) appears, namely halo artifacts. Specific examples are shown in Figure 1. (2) There is a simultaneous close-range and distant view in many images. When the colour difference between the close-range objects and the distant view is obvious and the size of the patch Ω(x) is larger, the recovery images can be expected to produce halo artifacts. To be exact, the image regions that should be the residual fog around close-range objects appears as halo artifacts. If the size of the patch is larger than 1 × 1 when calculating the dark channel map, there will be an object boundary in the patches near the edges of the object, namely, there will be two different objects in the same patch. As shown in Figure 2, if the close-range view is a dark object, its grey value will be very low in the dark channel map, and then the edge of the distant object of the same patch, Ω(x), will also take this grey value (far lower than its own dark channel value).Then, the distant object is mistaken for a close-range object in the dark channel map, as shown in Figure 2(a,b). Because t ( x ) = e − β d ( x ) , the d(x) value of the distant object or background near the boundary is equal to the d(x) value of the close-range darker object in the same patch, Ω(x). This d(x) value of the distant object is far less than the real value, or even close to 0. Transmission t(x) of a distant object or background near the boundary will be close to 1. In this case, according to Equation (1), there will be J(x)≈I(x), which means that the portion near edges is not processed or the processing is considerably insufficient. Thus, there is residual fog around the boundary of the close-range objects, appearing like halo artifacts. In Figure 2, it can be easily seen that there is a fringe of halo artifacts around the edges of the close-range leaves in columns (a) and (b), which is actually unprocessed fog. This phenomenon disappears completely as the size of Ω(x) decreases to 1 × 1. Figure 3 shows a comparison of local magnification of two different halo artifacts, and the two patch sizes are both 20 × 20. We can observe that the outer edge of the Tiananmen has no colour in the first picture. The value of this part exceeds the normal range of RGB colours and is ultimately shown in white. The second picture shows that the edge of the leaves has a pale halo of fog with little processing. FIGURE 3Open in figure viewerPowerPoint Comparisons of the different halo artifacts effects (The sizes of the two patches are both 20 × 20) 3.1.2 Invalidation of the white close-range objects When the scene objects are inherently similar to the atmospheric light and no shadow is cast on them, the dark channel prior is invalid [16]. When a white object is shown in a long-range perspective, such as the sky region in a foggy image, the dark channel theory will be invalid. At this time, its transmission ( t ( x ) = e − β d ( x ) ) of the sky still tends to zero due to its long-distance ( d ( x ) → ∞ ), so the first term on the right of Equation (2) is still close to zero and the approximate value of t(x) can be obtained by Equation (3). This is why the DCP method in the sky area can still be used even if it becomes invalid. In Figure 4(a), the sky region of the restored image is very clear, but the dark channel value does not tend to zero in the sky region. If a white object is shown in close-range in the foggy image, the DCP will be seriously invalid and the colour of the restored image will be greatly distorted. Because the first term on the right of Equation (2) does not tend to zero, Equation (3) does not hold in this circumstance. In Figure 4(b), we can see that the colour of marble varies seriously due to this effect. This is also the reason why images restored by DCP would be darker than those processed by other methods. FIGURE 4Open in figure viewerPowerPoint Comparison of DCP restored images (patch size is 1 × 1): (a) the long-range respective (sky) is not affected even if DCP theory becomes invalid. (b) The colour of white marble is distorted seriously 3.2 ADCP algorithm Based on the above analysis, we know that the dark channel values of the close-range white objects are not close to zero, but the relatively large values and halo artifacts are related to the patch size of the dark channel map. Therefore, we propose the amended dark channel prior to solve these problems in a pixel-wise fashion. The specific steps are as follows. For convenience in Equation (2) we replace min y ∈ Ω ( x ) ( min C ∈ { R , G , B } ( I C ( y ) ) with I ∇ and replace min y ∈ Ω ( x ) ( min C ∈ { R , G , B } ( J C ( y ) ) with J ∇ , namely: I ∇ = J ∇ · t ( x ) + A · ( 1 − t ( x ) ) . (4) The size of the path Ω ( x ) is 1 × 1 in Equation (4), so we first eliminated the halo artifacts in our restored images. Then we establish the energy functional equation to amend the values of J ∇ (dark channel) as follows. arg min J ∇ ∑ P I ∇ − J ∇ · t ( x ) − A · ( 1 − t ( x ) ) 2 + γ · φ I ∇ · J ∇ 2 . (5) In Equation (5), when γ · φ ( I ∇ ) → ∞ , J ∇ will be the theoretical dark channel. When γ · φ ( I ∇ ) → 0 , Equation (5) changes back to Equation (4). The " φ ( I ∇ ) " can be adaptively adjusted to align the value of J ∇ with the actual situation. If the value of I ∇ is very small, this indicates that this pixel accords with the dark channel prior, then the φ ( I ∇ ) value will be very large, and the dark channel prior becomes valid and can be expected to be effective. If I ∇ is large, it suggests that the pixel may be a close-range white object or other high-brightness object, because I ∇ is the dark channel value of the original foggy image and the RGB three-channel values of white objects are all higher, and the dark channel J ∇ will not equal zero. In other words, the dark channel prior is invalid for these pixels. Then, the value of φ ( I ∇ ) will be automatically adjusted to a small value at this time, and J ∇ will be amended greatly. Therefore, the dark channel value J ∇ can be amended pixel-wise through Equation (5), and the final result is more in line with the actual situation. In our experiments, we design several functions φ ( I ∇ ) to adjust the weights in Equation (5). We found φ ( I ∇ ) = − 0.7 − 0.1 I ∇ is easy and suitable, and the γ = 5 . Take the derivative of Equation (5) with respect to J ∇ , then take it to zero. The following equation can be obtained. J ∇ = I ∇ − A · ( 1 − t ( x ) ) t ( x ) − γ · φ I ∇ . (6) The transmission achieved by the DCP can be used as the initial transmission t(x). To decrease the computational complexity, we can select a random matrix of size h × w (size of the foggy image) as the initial transmission. By substituting the initial transmission t(x) into Equation (6), we can obtain the new J ∇ . Then, we put this J ∇ into Equation (4) to update the new amended t(x). This method is efficient and does not require iteration. Figure 5 shows a comparison of restored images between DCP and ADCP, where both patch sizes are 1 × 1. As shown in Figure 5(d), the amended values of J ∇ are not equal to zero in white or other brightness regions, but in the DCP algorithm, the values of this map should be zero regardless of the path size of the dark channel. Figure 5(d) shows noise because the initial value of t(x) is a random matrix. After the subsequent optimisation of the transmission map, this noise does not affect the final restored images. In our experiments, we used many different initial t(x) maps, but there was almost no difference among the final restored images. This phenomenon demonstrates that our method is valid. FIGURE 5Open in figure viewerPowerPoint Comparison between DCP and ADCP: (a) Foggy images, (b) restored images by DCP (patch size is 1 × 1), (c) restored images by ADCP (the size of patch is 1 × 1), and (d) the amended dark channel map J ∇ (if using DCP, this map should be all dark no matter what the path size of the dark channel is.) 3.3 Optimisation of transmission For image defogging, the edge information is critical because it can distinguish objects with different depths of field. We argue that various parts of the same object have the same depth of field in images. Therefore, the relatively weak internal texture and other details of objects are noise and interference for the purpose of calculation of the transmission map. In the transmission map, we hope for the main edges of objects to be clear, the useless details to be restrained, and the transmission values to be kept as consistent as possible in each object. In [16, 17], soft matting and guided filtering are used mainly to smooth and remove weaker details while preserving the main edges of an image, but their ability to remove residual fog is insufficient. It is difficult to balance the smoothing texture and preserve the main edges for guided filtering or soft mating. At present, there are many algorithms that have similar functions, such as Tikhonov regularisation, TV regularisation [44], gradient L0 optimisation [45], WLS filtering [46], RTV filtering [47], and bilateral filtering [48]. Most of these algorithms are used in image denoising or other fields. Our experiments showed that a great capability of texture removal is necessary for transmission optimisation. Thus, the filter or optimising algorithm requires a stronger smoothing ability while preserving the general edges. It is obvious that the requirements of transmission are not satisfied by the algorithms above, and these algorithms have high computational complexity. This paper proposes a locally variable weighted 4-directional L1 regularisation and wavelet fusion to optimise the transmission, and adopted an approximate solution to solve the 4-directional variational problem. In Figure 6(b), the x, y, x45, and y45 four gradient directions are shown. Our energy equation is regularised from four directions at the same time. Compared with the previous method [16, 17, 44-48], our smoothing and edge-preserving ability are both doubled in the same patch. Namely, our method has an edge-preserving capability and smoothing ability in the x, y, x45, and y45 axes at the same time, which is conducive to processing irregular natural objects in foggy images. Our computational complexity is significantly lower than the ordinary variational problem [44, 47] because an approximate algorithm is used. Finally, more clearly restored images can be obtained. To our knowledge, this is the first time that 4-directional L1 regularisation has been applied in the image edge preserving filter and image defogging field, and it is the first time that an approximate method has been effectively used to solve the 4-directional variational problem. FIGURE 6Open in figure viewerPowerPoint Gradient directions: (a) 2 gradient directions (x, y); (b) 4 gradient directions (x, y, x45 and y45) Sakurai et al. [49] first proposed the 4-directional total variation for the anisotropic case, but a complete mathematical proof was not provided (it was admitted as such in [49]). Fan Liao et al. [50] provided the complete mathematical proof for both anisotropic and isotropic cases of the 4-TV model, and they adopted fast gradient projection (FGP) methods [51] to solve the 4-TV model. This method is still time-consuming because it requires many iterations, and its convergence becomes slower with the increasing iterations. This is a common phenomenon in the gradient projection method. Unlike TV [44, 49–51] or context regularisation [29], our method does not need multi-iteration and adopts parallel computation so that it has a fast computational speed. 3.3.1 4-directional L1 regularisation First, we build the energy functional equation for the transmission map as follows: arg min t ∼ ∑ P t ∼ P − t P 2 + λ · W P · ∂ t ∼ ∂ x P + ∂ t ∼ ∂ y P + W 45 P ∂ t ∼ ∂ x 45 P + ∂ t ∼ ∂ y 45 P , (7)where subscript P denotes the location of one pixel in the transmission map, t ∼ P the final optimised transmission value, and t P is the input transmission. The data term, ( t ∼ P − t P ) 2 , is a fidelity term preventing the input and output from deviating wildly. The second term, λ · [ W P · ( | ∂ t ∼ ∂ x | P + | ∂ t ∼ ∂ y | P ) + W 45 P ( | ∂ t ∼ ∂ x 45 | P + | ∂ t ∼ ∂ y 45 | P ) ] , is the regularisation term and can smooth the result by minimising