A remote‐sensing image enhancement algorithm based on patch‐wise dark channel prior and histogram equalisation with colour correction

IET Image ProcessingVolume 15, Issue 1 p. 47-56 ORIGINAL RESEARCH PAPEROpen Access A remote-sensing image enhancement algorithm based on patch-wise dark channel prior and histogram equalisation with colour correction Fayaz Ali Dharejo, Computer Network Information Center, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Haidian, Beijing, ChinaSearch for more papers by this authorYuanchun Zhou, Computer Network Information Center, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Haidian, Beijing, ChinaSearch for more papers by this authorFarah Deeba, Computer Network Information Center, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Haidian, Beijing, ChinaSearch for more papers by this authorMunsif Ali Jatoi, Department of Biomedical Engineering, Barrett Hodgson University, Karachi, Sindh, PakistanSearch for more papers by this authorYi Du, Computer Network Information Center, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Haidian, Beijing, ChinaSearch for more papers by this authorXuezhi Wang, Corresponding Author wxz@cnic.cn Computer Network Information Center, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Haidian, Beijing, China Correspondence Xuezhi Wang, Computer Network Information Center, Chinese Academy of Sciences, Beijing 100086, China. Email: wxz@cnic.cnSearch for more papers by this author Fayaz Ali Dharejo, Computer Network Information Center, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Haidian, Beijing, ChinaSearch for more papers by this authorYuanchun Zhou, Computer Network Information Center, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Haidian, Beijing, ChinaSearch for more papers by this authorFarah Deeba, Computer Network Information Center, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Haidian, Beijing, ChinaSearch for more papers by this authorMunsif Ali Jatoi, Department of Biomedical Engineering, Barrett Hodgson University, Karachi, Sindh, PakistanSearch for more papers by this authorYi Du, Computer Network Information Center, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Haidian, Beijing, ChinaSearch for more papers by this authorXuezhi Wang, Corresponding Author wxz@cnic.cn Computer Network Information Center, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Haidian, Beijing, China Correspondence Xuezhi Wang, Computer Network Information Center, Chinese Academy of Sciences, Beijing 100086, China. Email: wxz@cnic.cnSearch for more papers by this author First published: 04 December 2020 https://doi.org/10.1049/ipr2.12004Citations: 4AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract The object identification within an image captured during rough weather conditions (such as haze, fog) poses difficulty due to the reduction of an image. The rough weather conditions lead not only to the variation of the image's visual effect but also to the disadvantage of post-processing of an image. Furthermore, it causes inconvenience of all types of instruments that rely on optical imaging, such as satellite remote-sensing systems, aerial photo systems, outdoor monitoring systems, and object identification systems, respectively. Hence, the improvement and restorement of the visual effects and enhanced post-processing are needed. This research introduces a new image enhancement approach for image dehazing based on dark channel prior and piecewise linear transformation; also, the histogram equalisation technique, i.e. contrast limited adaptive histogram equalisation is applied. A dark channel prior is well known for its simplicity and productivity. In this work, the dark channel prior to a new angle is analysed in the first step, where average patch sizes are estimated for the computation of haze densities. Furthermore, the sky is approximated up to 5–10% of the hazy images, which has a good effect in removing the haze from the image. Using the dark channel, the proposed algorithm significantly boosted the effects of the dark images as well as reduced the influence of haze and noise. Eventually, for colour correction, the piecewise linear transformation technique is applied, which enhances the colour close to the original image. Experimental results demonstrate that the proposed method significantly improves the visibility of the algorithm on dark remote-sensing images as well as on hazy natural images. 1 INTRODUCTION Image dehazing technique is essentially a way to minimise or even eliminate interferences through special techniques, to produce adequate visual effects and gain more valuable information. It is always challenging to remove the artefacts and noise [1] from the dehaze image. A variety of dehazing algorithms have been proposed over the last few years. They can be broadly categorised into two groups: multiple frames and single frames. Different image-based dehazing procedures also use a range of images, such as different constraint-based [2, 3], polarisation-based [4], and depth methods [5, 6]. Recently, for image-based dehazing, certain types of priors and assumptions are adopted. For instance, Tan [7] proposed a new approach based on the significant observation that in the hazy image, the local contrast was higher than the hazy image [7]. Considering that the transition and the surface shading of objects were statistically independent, Fattal et al. removed the haze using a separate analytical component [8]. Subsequently, he suggested estimating the transmission using the colour line model in a small patch [9]. Berman et al. [10] demonstrated that it was a haze line in the red, green and blue (RGB) space, proposing that the haze colour cluster-free image is a non-local prior. Based on the prior colour reduction, Zhu et al. [11] developed and monitored the model parameter in a linear scene-depth model for the hazy input image. For a single-image dehazing, the dark channel prior was suggested. The preliminary data is focussed on measurements which often contain very low intensity pixels in local patches of hazardous images [12]. For the transmission calculation, Bui et al. used colour ellipsoids, which are statistically calibrated for the haze of RGB pixel clusters [13]. A system for optimising transmission is also proposed in [14]. In a decomposition-based approach for an edge, preservation is proposed [15]. Using the internal relationship between gamma correction and the atmospheric dispersion mechanism, Ju et al. proposed a dehazing mechanism [16]. Zhu et al. [17] suggested a new approach for estimating transmissions based on finding dark pixels rather than the dark channel. As a luminance reconstruction framework, Liu et al. [18] reformulated the problem of hazel removal. For transmission evaluation, strength and saturation are used [19]. The transmission is determined only by employing saturation in [20]. Image dehazing is an ill-posed problem (i.e. it has no unique solution) of image processing. Various optimisation and machine learning techniques are required to solve this problem, i.e. assuming that there are numerous images in a scene. After ideal performances of the neural network models for image classification [21], now they are also providing better outstanding performances for single-image dehaze, such as the recently proposed method called all-in-one dehazing network (AOD-Net) [22], which is used to recover the final fog-free image through a new central variable without evaluating the transmitted image with the limitation of the weak overall effect. The current single-image denoising algorithm can remove haze to certain extent in the image, but the high-quality fluid images cannot be obtained. Compared with the existing single-image defogging methods, PFNet has the best performance in the resident (13) dataset. FFNet [23] can restore most outdoor blurred images as expected, but when it is applied to indoor blurred images, it will produce unwanted artefacts. Our recently proposed method [24] also has a good effect on remote-sensing (RS) images in the hazy environment. However, it is not good at dealing with the dark environment. A variety of dehazing algorithms, including multiscale CNN [12], DehazeNet [25], AOD-Net [22] etc., are being proposed with the rapid advancement of artificial intelligence (AI) technology. The more detailed study of various outer imagery dehazing methods can be found in [26]. Generally, the dark channel prior approach for its simplicity and efficiency is considered as one of the best methods for removing haze. However, it has other inherent difficulties in reality. As mentioned in [27], for example, the dark channel often underestimates the transmission when an object exhibits a similar grey intensity to atmospheric light, which in turn over-saturates and makes the scene radiance unnatural. In optical RS images, clouds are also an inevitable barrier to the effective observation of sensors [28]; therefore, it is essential to recover the original information for further post-processing to recover the original information in order to use it safely. In RS image enchantment colour and contrast are challenges to always improve, such as Imtiaz et al. [29] proposed methods capable of handling images of grey and colour channels. The existence of a light-scattering effect in the RS image will cause the phenomenon of dark (haze) in the image. It is noted that the pixels for an image are termed as dark pixels where a very low intensity is observed for at least one colour (RGB) channel. In order to solve the problem of the dark channel, the defog algorithm dark channel prior (DCP) is used to remove the fog/haze in the image. Before DCP, haze density is the most crucial factor affecting the quality of the dark image, so the haze density is calculated by estimating the patch size. The image blocks that are more suitable for DCP are observed, and then a primary denoising algorithm DCP method is employed to eliminate fog in a dark environment. It is found that the DCP results darkened the background and the whole image. Therefore, to overcome the shortcomings of the DCP method, here the technique of combining DCP and clean technology is introduced [30], and finally the colour correction piecewise linear transformation (PWLT) technology is applied because it can recover the original colour well. By combining DCP and histogram equalisation, we can recover high-quality, non-fog images, and then use colour correction technology, PWLT, to process the results, which is the final output image. The final output image can not only eliminate the haze freely but also restore the original colour of the image. The remainder of the paper is structured as follows. Section 2 reflects the proposed approach, Section 3 describes the experiment and outcomes, and finally, the conclusion and future research is presented. 2 METHODOLOGY Real-time dehazing is used in fast, robust visibility for computer vision applications such as AI robots, auto-pneumatic vehicles, and RS technology. The DCP results will be analysed by histogram equalisation to eliminate the drawbacks of the DCP algorithm., i.e. the contrast limited adaptive histogram equalisation (CLAHE) technique [3]. Then, the obtained image is segmented into small blocks to estimate the haze density. The optimal patch size is then feedback to the DCP method, which gives the excellent results. Furthermore, to achieve original colour use, PWLT [31] processes the colour correction method for the output results. In the image-processing sense, the histogram is the procedure in which any intensity value occurs in the image; the histogram equalisation is the technique that increases the dynamic range of the histogram of the image. Histogram plots show multiple times (frequencies) of each image-intensity value. A histogram plot shows the position of the image globally, providing valuable details for the enhancement of contrast. Low contrast means the image uses less intensity, and high contrast means the image uses several distinct intensity values. The histogram of a digital image is the discrete function, f, and is defined as H ( r k ) n k , where r k is the Kth intensity value and n k is the number of pixels in f with intensity r k . As hazy images lose contrast and colour, even in minor changes it is necessary to evaluate the differences between the two images with a histogram. Four specific types of images and their respective histograms exist – bright, dark, low contrast, and strong contrast. We have used the histogram equalisation technique (CLAHE) because it is a process in which the histogram is evenly distributed to increase the contrast in an image. Our method has two objectives: the first objective is to restore a dark RS image when a dark environment affects the RS image, the other objective is dehydration because fog is also a factor that affects the image quality of RS. Finally, our main goal is to restore the original image using RS dark image. Histogram equalisation has an adverse effect to increase visual graininess in a low-contrast image. The overall flow of our method is shown in Figure 1. FIGURE 1Open in figure viewerPowerPoint Proposed approach block diagram Image dehazing is a poorly caused two-dimensional ill-posed problem in signal or image reconstruction. Its purpose is to restore an unknown image of a hazy image. The model can be expressed as follows: I x = J x t x + A 1 − t x , (1)where I (x) is the observation image, J (x) is the clearer image we need to estimate, A is the global atmospheric light, and t(x) is the medium transmission: J x = I x − A max t x , t 0 + A . (2) 2.1 Haze density estimation matrix Haze density is the most important factor in dehumidification application, so because of weather, haze density is not constant. To calculate the haze density, we transform RGB colour mage into I(x), i.e. hue, saturation, and value (HSV) colour model. Then, we get the distance D(x) of each pixel in the IHSV(x) model. d x = C − S H S V x , (3)where C is the brightness in the HSV colour model, and its effect is such that the shorter the distance, the brighter the pixel, which indicates the maximum value of density in all d(x) [12]. RGB are all connected to colour luminance (which is called intensity), meaning that colour details cannot be isolated from lighting. HSV or hue value is the separation from colour details of image luminance. It helps the research or includes image/frame luminance. For cases where representation of colour plays an integral role, HSV is often used. The key colour properties that allow us to distinguish between colours are HSV. Colour is a crucial element of photography when colour is successful, as it can attract the viewer's eye towards the composition of the image and can affect the mood and emotion of the image. For a certain colour, grey is characterised by saturation between 0% and 100%. Reducing the part to zero leads to more grey and a faded effect. It defines the brightness or intensity of the colour, from 0% to 100%, where 0 is totally dark and 100 is the brightest, and shows the most colour: H a z e d e n s i t y x = 1 − d x m à x d x . (4) 2.2 Patch-size estimation In our method, patch size is the main contribution of DCP technology. The fixed patch size is selected for the formation and transmission of dark channels. In general, the larger the patch, the better the dark channel, but due to the larger patch size, the halo effect near the edge may become more substantial, which will prevent the premise of DCP from achieving uniform transmission. Similarly, if we choose a smaller patch size, there will be an excessive colour enhancement in the output [32]. Experiments have shown that if the input image is not clear (hazy), small patch sizes will lead to better output results. Therefore, the haze density has a significant influence on the selection of the patch size, but the fixed patch size used by most methods cannot reflect the best parameter-handling process. We calculated the average haze density through the haze density matrix. We experiment with different noise (blurred) images to find the average of varying coloured patch sizes. The results are shown in Table 1. TABLE 1. Image analysis of average haze density with various patch sizes Patch size Image 1 Image 2 Image 3 Low hazy image Medium hazy image Heavy hazy image 2 × 2 0.0214 0.0191 0.0112 4 × 4 0.0295 0.0139 0.0107 8 × 8 0.0247 0.0147 0.0118 16 × 16 0.0205 0.0135 0.0105 Optimum patch size 4 × 4 2 × 2 4 × 4 2.3 Haze removal method based on DCP There are many recent methods for RS image enhancement. Still, dark channel prior methods are commonly used to generate images from noise and natural haze-free images in the dark environment. We use this technology to process dark RS images because bad weather and light scattering can cause haze in the image, resulting in noisy dark particles. However, the DCP method eliminates it well. Although DCP states that in many cases, local areas appearing in the background of an RS image usually have very low intensity in the RGB colour in the form of at least one channel. This colour is called Z d a r k ( x ) , which represents the dark at x aisle: Z d a r k x = min c ∈ R , G , B min y ∈ Ω x Z c y . (5) In Equation (5) Z c is the RGB channel, dark-channel operators are the coordinates y and Z(y) that extract a minimum RGB colour channel in the local Ω ( x ) patch centred at x. If x does not belong to a local area, Z d a r k ( x ) indicates a lower value approaching 0, so it is called the dark channel. S(x) is the intensity of the image, it is a mixture of atmospheric light, so we can say that S(x) is usually brighter than Z ( x ) and has a lower value that is medium transmission t(x). It is worth noting that compared to Z ( x ) , the dark channel of J ( x ) has a higher value, which helps to eliminate haze. Firstly, the global atmospheric light A is estimated, then the average transmission t ∼ is calculated by dividing Equation (1) to Ac: min C ∈ R , G , B min y ∈ Ω x S C y A C = t ∼ x min C ∈ R , G , B min y ∈ Ω x Z C y A C + 1 − t ∼ x , (6)where t ∼ ( x ) is the coarse transmission map based on a patch and the arguments S C ( y ) A C and Z C ( y ) A C are the element-wise divisions. If W(x) is set to large patch size (e.g. 16 × 16), then it should tend to be 0 [9], so the equation can be min C ∈ R , G , B min y ∈ Ω x Z C y A C = 0 . (7) By combining Equations (3) and (4), we have t ∼ x = 1 − min C ∈ R , G , B min y ∈ Ω x S C y A C . (8) In addition, He [12] proposed a parameter called w, which is used in Equation (5) to keep the haze in the image to a minimum to see the depth of the image, so we have a medium t ∼ transmission: t ∼ x = 1 − ω min C ∈ R , G , B min y ∈ Ω x S C y A C , (9)where w is the rate of depth removal considered for human perception. To optimise medium transmission t ∼ and find accurate medium transmission t, a soft matching algorithm is applied: Z x = J x − A max t x , t 0 + A (10)where t ∼ ( x ) is the patch-based transmission map, t ( x ) is the refined transmission map, A is atmospheric light, and t 0 is set to be 0.1 in order to avoid the division by a small value. The DCP improves the spatial resolution of the transmission map t(x) along with object wheels by measuring the maximisation of the space within a dark path image. Therefore, by manipulating images, He et al. refined the map [12]. The processing of images therefore requires high computational costs and cannot be done in real time. Further research proposed a pixel-wise DCP [29] and a combination of original patch-wise DCP in a flat and pixel-wise region around the edge region Ibrahim [35]. Therefore, Z C ( y ) A C (Equation (6)) cannot be 0 by setting the correct patch size, as shown in Table 1, instead of constant patch size. So, in the proposed method Z C ( y ) A C (Equation (6)) has a small value, which ranges from 0 to 1. 2.4 Limitations in the DCP process In general, a digital image is made up of two parts. The first part is the foreground part, which is the main content (information). The second part is the background part, which has nothing to do with the main content (except the main content). Therefore, DCP technology has major flaws in terms of background enhancement. If there is noise (haze) in a little background with low contrast, DCP will have an adverse effect on the dark environment with low contrast because DCP will only reduce the contrast of the local area, as shown in Figure 2(a). The local area is regarded as the image background of the scene with light and dark and mist shadow. For the second problem of DCP, some direct attenuation can be obtained from Equation (1) [6], which is also non-linear with the image strength. Therefore, DCP underestimates foreground image fading, so the output contrast of DCP is lower than that of the fog-free image. The disadvantage discussed here is that the haze is eliminated. Still, when the background area of the haze-free image is large, and the contrast is low, DCP will produce a weak visual effect and reduce the contrasting foreground. Due to the influence of external light, DCP does not restore the primary colour well, so even some traditional methods based on DCP cannot restore the primary colour [12, 33, 34]. FIGURE 2Open in figure viewerPowerPoint (a) shows the gives the color and contrast results where, first shows the Low-contrast hazy image second is DCP output, and third is proposed output; (b) analysis of three different results: first DCP, second prior to PWLT, and third with PWLT 2.5 Sky region estimation In the previous methods, the transmission time is a fixed value (0.1 s), and the transmission value is (t0). It is very important to select the sky region in the hay image. The sky region (t0) is a part of the haze image. To remove the artefact of the sky region, it must be processed. Furthermore, when a fixed value is used, the sky region is shaped because the transfer map is not smooth. However, DCP may not work when the scene looks substantially like air light. For example, it has been found that in experiments, sky regions tend to degrade due to severe noise and colour distortion. In our research work, we took this into account and assumed that t0 is calculated from the given input image. We assumed that the sky area is 5–10% of the hazy vision, so we used the same method to calculate (t0) based on 5–10% of the hazy part via calculating the patch size. 2.6 Air light estimation Air light is an approximation that contains dense fog from hazy images. This air is thought to come from the observer's view. We calculate it by creating a dark channel twice the size of the original patch, in order to eliminate the trap of improper calculation of air light due to the existence of strong external light source. Finally, after estimating the dark channel, assuming that 0.2% of the brightness value of the dark channel is taken as the average value, the air light is calculated. From Equation (1), we assume that the transfer t(x) in a local patch is constant: mi n y ∈ Ω x I c y A C = t mi n y ∈ Ω x J c y A C + 1 − t x . (11) From Equation (5) we have Z d a r k ( x ) = mi n c mi n y ∈ Ω ( x ) J C ( y ) . (12) By combining (11) and (12), we have mi n c mi n y ∈ Ω x Z c A C = 0 . (13) By combining Equations (12) and (13), we have t x = 1 − mi n c mi n y ∈ Ω x Z c A C . (14) A constant parameter is introduced ω , 0 < ω < 1 due to some existence of particles even in clear atmosphere: t x = 1 − ω ∗ mi n c mi n y ∈ Ω x Z c A C . (15) 2.7 Scene radiance To restore the brightness of the scene, we have selected the brightest 0.1% pixels in the dark channel, since these pixels are hazy, the final reconstructed image is calculated as J c x = I C x − K c max t x , t 0 + K c , (16)where ambient light ( K c ) is estimated to be the average of these light pixels in the tri-colour channel and to eliminate the noise t0= 0 when t(x) is close to 0. 2.8 Colour correction For accurate colour correction, we applied the Buchsbaum technique Ibrahim [35] colour correction model through PWLT transformation. This assumption implies that the average colour is achromatic, equal to 128 of the 8 bits of the image. The method is modified, the colour of segmentation transformation is proposed, and the average value of the image is extended to 128: M c − M mean c M min c − 128 M min c − S mean c + 128 , M mean c ≤ 128 M c − M mean c M max c − 128 M max c − M mean c + 128 , M mean c > 128 . (17) Based on the average value, we should determine which colour direction is suitable for low contrast and low-intensity values. This is due to the fast absorption of the red wavelength, and because it is necessary to reconstruct the original colour, the following improvements can be made: M C R C = M c − λ M mean c − 128 , P c < 0.4 , M c − M mean c M min c − 128 M min c − S mean c + 128 , M mean c ≤ 128 M c − M mean c M min c − 128 M min c − S mean c + 128 , M mean c > 128 (18) As shown in Figure 2(b), the original colour is not well recovered for scene radiance output before PWLT (it shows whitening), so we applied PWLT for colour correction, which is the most important part of the image dehazing, as many images are similar to the colour of the object and where it is located. So, on the contrary, it can give us better contrasting features. 3 EXPERIMENTAL ANALYSIS We have implemented our algorithm on the NWPU-RESISC45 [25] dataset concerning dehazing and dark images to evaluate the performance of different algorithms quantitatively. We compared our method with Ibrahim [35], adaptive gamma correction with weighting distribution (AGCWD) [26] method, dark channel prior (DCP) technique [12], color attenuation prior (CAP) method [11], gated fusion network (GFN) method [36], and the proposed method for qualitative visual assessment and quantitative assessment, respectively. The experiment was performed on a PC with 8 GB RAM on MATLAB 2019b. When handling with dense, hazy images, a moderate β is required. β = 1.0 is more than adequate in most of the situations. For the best results, we have picked Ω = 0. 85, t0= 0.1, and for colour correction λ = 0.1. The image shown in Figure 3(a) is composed of different sizes, so in order to respond quickly, we reduce its size to 256 × 256 pixels. Haze density is sensitive to the dehazing proposed method; therefore, it is very conscious that we compute the haze density first, as shown in Table 2, by choosing the average patch size. While it was seen during experiments that if the patch size is randomly taken as earlier methods, then the dehazing is not much effective as shown in results from Figure 3(b). Our main goal is to design the algorithm for RS dark satellite images. From the NWPU-RESISC45 dataset, we applied both types of image hazy (airport image) and dark images to our method. FIGURE 3Open in figure viewerPowerPoint (a) Input images with correspond histogram from NWPU-RESISC45, (b) comparison of proposed method with correspond histogram from NWPU-RESISC45 [25] ((a) Ibrahim [35] method, (b) AGCWD method [26], (c) DCP method[12], (d) CAP method [11], (e) GFN method [36], and (f) proposed method) TABLE 2. Comparative analysis of PSNR and SSIM of different methods RS images Original Ibrahim [35] AGCWD [26] DCP [12] CAP [11] GFN [36] Proposed airport-650 (Image.1) 9.37/ 0.721 11.91/0.810 13.3/0.814 16.86/0.725 18.36/0.831 22.65/0.845 26.98/0.878 commercial_area_699 (Image.2) 11.25/0.809 11.31/0.811 12.82/0.814 14.63/0.724 15.47/0.831 23.78/0.8