Low‐light image enhancement based on exponential Retinex variational model

IET Image ProcessingVolume 15, Issue 12 p. 3003-3019 ORIGINAL RESEARCH PAPEROpen Access Low-light image enhancement based on exponential Retinex variational model Xinyu Chen, Xinyu Chen orcid.org/0000-0002-5560-8297 School of Computer Science and Technology, Shandong Technology and Business University, Yantai, China Institute of ZhongKe Network Technology, Yantai, ChinaSearch for more papers by this authorJinjiang Li, Corresponding Author Jinjiang Li lijinjiang@gmail.com orcid.org/0000-0002-2080-8678 School of Computer Science and Technology, Shandong Technology and Business University, Yantai, China Co-innovation Center of Shandong Colleges and Universities: Future Intelligent Computing, Shandong Technology and Business University, Yantai, China Correspondence Jinjiang Li, School of Computer Science and Technology, Shandong Technology and Business University, Yantai, China. Email: lijinjiang@gmail.comSearch for more papers by this authorZhen Hua, Zhen Hua orcid.org/0000-0003-1638-2974 Institute of ZhongKe Network Technology, Yantai, China Co-innovation Center of Shandong Colleges and Universities: Future Intelligent Computing, Shandong Technology and Business University, Yantai, ChinaSearch for more papers by this author Xinyu Chen, Xinyu Chen orcid.org/0000-0002-5560-8297 School of Computer Science and Technology, Shandong Technology and Business University, Yantai, China Institute of ZhongKe Network Technology, Yantai, ChinaSearch for more papers by this authorJinjiang Li, Corresponding Author Jinjiang Li lijinjiang@gmail.com orcid.org/0000-0002-2080-8678 School of Computer Science and Technology, Shandong Technology and Business University, Yantai, China Co-innovation Center of Shandong Colleges and Universities: Future Intelligent Computing, Shandong Technology and Business University, Yantai, China Correspondence Jinjiang Li, School of Computer Science and Technology, Shandong Technology and Business University, Yantai, China. Email: lijinjiang@gmail.comSearch for more papers by this authorZhen Hua, Zhen Hua orcid.org/0000-0003-1638-2974 Institute of ZhongKe Network Technology, Yantai, China Co-innovation Center of Shandong Colleges and Universities: Future Intelligent Computing, Shandong Technology and Business University, Yantai, ChinaSearch for more papers by this author First published: 15 June 2021 https://doi.org/10.1049/ipr2.12287AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Aiming at the problems of residual noise, low contrast, and limited detail information caused by low-light images, this paper proposes a new Retinex variational model. According to Retinex theory, it is necessary to estimate the illumination and reflectance components decomposed from the original image. In order to better maintain the edge information, texture richness, and prevent artefacts, the exponential forms of local variation deviation and total variation are used as illumination prior and reflectance prior, respectively, and mixed norms are used to constrain them, so as to deal with the illumination information and texture details of the image more effectively, and then use the bright channel prior to improve the colour reproduction sense of the original image, thereby constructing the objective function, and finally using the alternating iterative optimization method to find the optimal solution to the proposed model. Experiments show that compared with other existing image enhancement methods, the method proposed here improves the contrast of the image, overcomes the phenomenon of halo artefacts and colour distortion, is more consistent with human vision, and produces better results in terms of quantitative performance. 1 INTRODUCTION Due to the influence of insufficient light such as night, cloudy weather, rain and fog, and backlight shooting, the collected images are prone to problems such as low dynamic range, loss of detail, and serious noise, which cause the degradation of image quality and limit the performance of computer vision system [1-3]. In order to be able to provide high-quality images for subsequent image analysis and processing, image enhancement of low-light images is required. With the development of digital image processing technology, many low-light image enhancement algorithms have emerged. Based on the spatial domain, it is a relatively simple and easy to understand algorithm. Commonly used methods include histogram equalization [4], linear spatial filtering, and non-linear spatial filtering. Histogram equalization methods are mainly divided into two categories: global histogram equalization methods and local histogram equalization methods. Global histogram equalization is to turn the greyscale histogram of the original low-light image as a whole into a uniform distribution of the entire greyscale range, which is a simple processing method, but can result in the loss of detailed information, excessive contrast enhancement in local areas, colour distortion, and other problems. As early as 1976, Ketcham [5] proposed a local histogram equalization, the main idea of the algorithm is to divide the low-light image into several small blocks, and then each sub-block of the histogram equalization, to achieve the effect of enhanced local area contrast, however, it is prone to edge artefacts. The brightness-preserving bi-histogram equalization (BBHE) proposed by Kim [6] and the dualistic sub-image histogram equalization (DSIHE) proposed by Wang et al. [7] are both improved algorithms based on local histogram equalization, which improve the brightness and detail parts, but the enhanced image lacks a sense of unity and brings certain side effects [8], such as excessive amplification of the image noise. The physical model-based approach is based on the property that each pixel in the original low-light image is inverted to resemble the characteristics of a foggy image [9]. The idea of defogging is adopted to enhance low-light images. Tang et al. [10] proposed a method to reduce the strong light area, suppress the halo, and improve the non-local denoising method to enhance the image, but it results in over-saturation. The physical lighting model for low-light image degradation proposed by Yu et al. [11] uses a Gaussian surrounding function and iterative estimation of the model parameters to restore low-light images and reproduce the colour consistency of the image, but cannot enhance local detail areas. Xiao et al. [12] used guided filtering and median filtering to estimate atmospheric light and obtain edge information, and then used gamma correction to enhance the image brightness. This method is susceptible to uneven ambient light. Low-light image enhancement algorithms based on machine learning have entered a period of rapid development due to the advent of the era of big data. Kin et al. [13] proposed the LLNet framework, which used the stacked sparse denoising autoencoder (SSDA) method to identify the signal characteristics of the low-light original image, and used two modules for contrast enhancement and denoising learning, but the encoder has limited processing power and is only suitable for small-sized images. K. Zhang et al. [14] trained a convolutional neural network (CNN) denoiser and integrated it into a model-based optimization method to deal with the inverse problem in low-level vision; the algorithm can produce good Gaussian denoising, but its robustness is poor. Tai et al. [15] proposed a deep recursive residual network (DRRN) approach, with a very deep network architecture and both global and local approaches to residual learning in order to alleviate the difficulty of training the network; however, the computational cost is high, a large amount of training data is required, and reconstruction artefacts are prone to occur. Dong et al. [16] proposed a single-image super-resolution (SR) deep learning method, which significantly improves the speed and performance; however, the method is less flexible in dealing with denoising and de-blurring. The Retinex-based method is an application based on the simulation of the human visual system. The Retinex theory was proposed by Land [17] in 1971, and then Land proposed a path comparison method, but the path selection and starting position are uncertain, which affects the estimation of illumination, and the computational complexity is high. Later, the single-scale Retinex (SSR) algorithm, the multi-scale Retinex (MSR) algorithm, and the multi-scale Retinex with colour restoration (MSRCR) based on centre-surround appeared one after another, but the image enhancement effect still has problems, such as the appearance of halos and unnatural colours. A new variational model based on Retinex theory is proposed. The original low-light image is decomposed into reflectance and illumination. For the reflectance and illumination regularization terms, the exponential form of total variation (TV) and exponential form of local variation deviation (LVD) methods are, respectively, applied, and the bright channel prior is added. Finally, an alternating iterative optimization algorithm is used to solve the model. As shown in Figure 1, an example of using this method for low-light image enhancement is given. It can be seen that the original low-light image has low brightness and can give people less information. The enhanced image can meet the basic needs of the human eye and computer vision system. The illumination image and the reflectance image effectively reflect the image Illumination spatial distribution and texture information. The experimental results demonstrate that the method enhances the brightness and contrast of the original low-light image, while improving the detailed information of the image and maintaining the original naturalness of the image. From the subjective and objective analysis, the effect is better than the existing methods. FIGURE 1Open in figure viewerPowerPoint An example of the application of the Retinex variational model approach proposed: (a) original low-light image, (b) enhancement, (c) illumination, (d) reflectance 2 RELATED WORK 2.1 Retinex model The Retinex model decomposes an image into illumination and reflectance components. The relationship between the two is shown in the following formula: I = L · R , (1)where I denotes the original image captured by the human eye or camera, L is the illumination component reflecting the intensity of ambient light, R is the reflectance component reflecting detailed information about the object itself, and ‘.’ denotes pixel-level multiplication, which expresses the product relationship between the illumination and reflectance components. The main idea of Retinex theory is to estimate and remove the illumination component to obtain the reflectance component, and get the image closest to the ground truth. From a mathematical perspective, this is actually an ill-conditioned problem. The traditional low-light image enhancement algorithms based on Retinex theory mentioned in the introduction all have halo artefacts. Therefore, in 2003, Kimmel et al. [18] first proposed a Retinex variational model and used the prior knowledge of reflectance images and Bayesian theory to add a penalty term to better deal with the situation of uneven lighting. Zhang et al. [19] proposed a global sparse gradient guided variational retinex model based on variational Retinex, which uses global sparse regularization to estimate the gradient field of the illumination image, and because of the assumption that the illumination is smooth, the guide vector field term is introduced. This algorithm can effectively improve the image enhancement effect. In the Retinex variational model proposed by Jin et al. [20], the piece-wise constant of reflectance is taken into account, and TV regularization term is introduced, which helps to reduce image noise and improve image quality. Febin et al. [21] proposed another Retinex variational model, which uses a non-local frame to represent reflectance and intensity changes to preserve more texture details and boundary regions. The model is based on perception and is more efficient in enhancing and denoising the remote sensing image data. The above algorithm can achieve good low-light image enhancement compared to the traditional Retinex method, but there is still some room for improvement. In the existing Retinex-based variational models, the problem of estimating the illumination component and the reflectance component is transformed into a secondary optimization problem with a closed solution. The objective function of the previous method can be reduced to the following formula: arg min R , L ∥ R · L − I ∥ 2 2 + A ( ∇ R ) + B ( ∇ L ) . (2) The first of these terms is the fidelity term, which is a squared error term for calculating the new image versus the original image in terms of illumination and reflectance. Based on the two assumptions that reflectance is a piece-wise constant and illumination is spatially smooth, the second term proposed represents the spatial regularization term of the reflectance component R, and the third term represents the spatial regularization term of the illumination component L. The second and third terms reflect a priori information about the image to be reconstructed. According to some existing algorithms in low-light image enhancement based on the Retinex variational model, L 1 and L 2 norms [22-24] etc. can be used to constrain the illumination and reflectance components for more effective estimation. 2.2 Structure and texture separation According to Retinex theory, the original image can be viewed as a composition of reflectance component and illumination component, while reflectance captures the texture details in the image, illumination captures the structure in the image and expresses the main properties of the image. Therefore, the original image can be decomposed into a texture layer and a structural layer [25]. The texture layer can reflect the details of the image, such as wood grain and leaf veins, so that the image meets the human eye visual realism, and the structural layer reflects the image to express the information of the illumination direction, such as the image of the greyscale value and colour composition. Structural texture decomposition is to preserve the image edge and texture while maintaining the structural characteristics, and most of the previous methods are based on assumptions about the characteristics of the structure and texture. For example, Xu et al. [26] assumed that the texture has the characteristic of ‘oscillation’ on a small scale and used a joint bilateral filter to decompose the structure and texture. Belyaev et al. [27] is based on the TV model and considers the curvature change on the level set in the weight of the regularization term to maintain the weak edge structure. The TV model was first proposed by Rudin et al. [28] in 1992. It estimates the texture degree, reduces the excessive smoothness of the image edge, and has a good effect on noise suppression. Bolun et al. [29] proposed a method to maintain the LVD of the structure, and obtained the variance correlation of the gradient change, which can be used as the basis for the division of structure and texture according to the strength of the correlation, where structure tends to be more strongly correlated. In the theory of the Retinex model, changes in reflectance can lead to large gradients and changes in illumination lead to smaller gradients. Thus, in dealing with the estimation of structure and texture, Xu et al. [30] introduced the exponential form of the local derivative, where the exponential form of TV can be written as | ∇ L | γ ; the exponential form of LVD is | ∇ L ¯ | γ = | 1 | Ω | ∑ Ω ∇ L | γ , where Ω is a local patch of size r × r (we set r = 3), and in [30], it is also proposed that the structural edges of the image are better extracted when the exponent γ is greater than 1, and the texture details of the image are better extracted when the exponent γ is less than 1. As can be seen in Figure 2, there is a visual difference between the enhanced image without using the exponential form and using the exponential parameters proposed here, in the second row, we can see that using the exponential form of the illumination image extracts the structural part of the low-light image more precisely, smoothing the light space more and avoiding unnecessary texture information; while in the third row, using the exponential form of the reflectance image extracts the texture of clouds and buildings more completely. Therefore, in the first row, we can see that the overall tonality of the enhanced image without the exponential form gives a flatter feeling, while the enhanced image with the exponential form gives a stronger contrast and a richer sense of hierarchy to the cloud area of the sky and building area. It can be concluded that the appropriate exponential parameters can improve the contrast and naturalness of the image. We will specifically analyse the effect of the exponential parameter size in subsequent experiments to select the best value. FIGURE 2Open in figure viewerPowerPoint Comparison of the images obtained without using the exponential form and those obtained with using the exponential form: (a), (c), and (e) are the enhanced, illumination, and reflectance image without exponent; (b), (d), and (f) are the enhanced, illumination, and reflectance image with exponent, respectively 2.3 Bright channel prior The dark channel prior is mostly used for the algorithm of image dehazing [31-34]. First, the model of a fogged image is as follows: I ( x ) = J ( x ) t ( x ) + A ( 1 − t ( x ) ) , (3)where I(x) is the luminance value at pixel x in the original fog image, J(x) is the luminance value of the fog-free image to be recovered, t(x) is the transmittance, and A is the atmospheric light. In the foggy area, the brightness of the dark channel is higher. The value of the dark channel can be used to describe the concentration of the fog. The dark channel prior can be used to estimate the value of the transmittance t(x) in the fog imaging model for image restoration. The calculation formula is as follows: J dark ( x ) = min c ∈ { r , g , b } min y ∈ Ω ( x ) J c ( y ) , (4)where J dark ( x ) represents the dark channel image, J c ( y ) represents one of the three colour channels of the image, and Ω ( x ) is the neighbourhood centred on the pixel x. According to the dark channel prior, some pixel points must exist in the outdoor non-sky area captured in good fog-free weather. In a colour channel of red, green, blue (RGB), its intensity value is very low and tends to zero, that is, J dark ( x ) → 0 . The bright channel prior [35] is a new thinking inspired by the dark channel prior, which refers to the maximum intensity value of some pixels in a small area of a well-exposed image that is always close to 1. By analogy with the dark channel, the bright channel of an image J can be defined in the following way: J bright ( x ) = max c ∈ { r , g , b } max y ∈ Ω ( x ) J c ( y ) , (5)where J bright ( x ) represents the bright channel image, J c ( y ) represents one of the three colour channels of the image, and Ω ( x ) is the neighbourhood centred on x. The value of J bright ( x ) can be used to determine whether an image is well exposed or not, and the bright channel prior can be used to adjust the local exposure of the image [36] to recover the detail of a well-exposed image, which we can apply to the enhancement of low-light images. 3 PROPOSED METHOD 3.1 Proposed model Based on the discussion and analysis in the related work in Section 2, we propose a new variational Retinex model, as shown in the following formula: min R , L ∥ R · L − I ∥ 2 2 + λ l · A ( ∇ L ) + λ r · B ( ∇ R ) + λ b L − I B 2 2 , (6)where λ l , λ r , and λ b are positive parameters, ∇ represents the gradient operator in the horizontal direction (x) and vertical direction (y), the first term is a data fidelity term that minimizes the distance between the R and L product and the observed image I as close as possible. The second term is the illumination regularization term, which guarantees the spatial smoothness of illumination. The third term is the reflectance regularization term, which guarantees the segmental smoothness of the reflectance. The fourth term is the bright channel prior, which is used to constrain the distance between the illuminance and the bright channel of the image, where we define I B as I B = max c ∈ { r , g , b } ( max y ∈ Ω ( x ) ( I c ( y ) ) ) , and when we calculate the bright channel of the input low-light image, we use the guided filter [37, 38], and refine the edges to prevent texture distortion and edge degradation. The L 1 norm is currently a more widely used regularization term. Its shape is very sharp and has sparse features. It is easy to directly set unimportant features to zero. The L 1 norm has a certain range of application. The calculation efficiency on sparse vectors is low, and data with larger values is prone to biased estimates, and in some cases the desired approximate results cannot be obtained. Compared with the L 1 norm, the L p norm ( 0 < p < 1 ) has a better estimation effect for larger values in the sparse solution, but the L p regularization problem itself is non-convex and non-smooth, and there are many ways to solve it, such as the K-nearest neighbour-based prediction scheme mentioned in [39], but most of them are slightly more complex. The L 2 norm has a more severe penalty for larger values, so that it can be estimated more effectively, and a model optimization problem with smaller parameters can be constructed. It can be applied to multiple types of datasets, reduce data migration, and enhance anti-disturbance capability. In recent years, it has also been found that mixing norms to form an objective function with several regularization terms gives better results than using a single norm as a regularization term. Therefore, we rewrite the approach of using norms, we use the L 2 norm to constrain reflectance and the L p norm to constrain illumination. (Figure 3 is a comparison between the L 2 norm and the mixed regularization term of the L 2 norm and L p norm. From the illumination and reflectance images in the third and fourth columns, we can find that the algorithm using the mixed regularization term can better capture the illumination and texture details, make the contrast between light and shadow at the edges more distinct, and make the objective objects in the enhanced images more three-dimensional, for example, the jointed part of the hand of the statue is portrayed more in line with the human sensory system.) For the non-convexity and non-smoothness due to the L p norm, we use the iteratively re-weighted least squares (IRLS) method [40] to solve the problem. It is an effective linear least approximation method. In addition, we use exponential LVD and TV as the illumination prior and reflectance prior to maintain the smoothness of the structure and the details of the texture. The formulas of the second and third terms are defined as follows: From ∥ ∇ L ∥ p p = ( ∇ L + ε ) p − 2 ∥ ∇ L ∥ 2 2 , Let A ( ∇ L ) = u ∥ ∇ L ∥ 2 2 = | ∇ L ¯ | γ L + ε p − 2 ∥ ∇ L ∥ 2 2 = 1 | Ω | ∑ Ω ∇ L γ L + ε p − 2 ∥ ∇ L ∥ 2 2 . (7) Then let B ( ∇ R ) = v ∥ ∇ R ∥ 2 2 = | ∇ R | γ R + ε − 1 ∥ ∇ R ∥ 2 2 , (8) ∥ · ∥ 2 represents the L 2 norm, and ∥ · ∥ p represents the L p norm, γ L and γ R denote their respective exponential parameters. In the second term for the proposed variational model, since IRLS will be used to solve the problem of minimizing the L p norm [41, 42], it is similar to the treatment in the paper [43], where the L p norm is replaced by the L 2 norm to obtain ∥ ∇ L ∥ p p = ( ∇ L ) p − 2 ∥ ∇ L ∥ 2 2 ; after that, to avoid zero denominator at the fraction, we add ε to deform it and obtain ∥ ∇ L ∥ p p = ( ∇ L + ε ) p − 2 ∥ ∇ L ∥ 2 2 ; furthermore, the introduction of the LVD in exponential form eventually leads to the definition of the illumination regularization term as A ( ∇ L ) = ( | 1 | Ω | ∑ Ω ∇ L | γ L + ε ) p − 2 ∥ ∇ L ∥ 2 2 , we write u = ( | 1 | Ω | ∑ Ω ∇ L | γ L + ε ) p − 2 to facilitate the representation of subsequent optimization calculations, where the range of p is 0 < p < 1 , and this paper sets p = 0.4. Similarly, in the third term, we adopt the conventional L 2 norm and introduce the TV in exponential form to define the final reflectance regularization term, obtaining B ( ∇ R ) = ( | ∇ R | γ R + ε ) − 1 ∥ ∇ R ∥ 2 2 , again, for ease of representation in the later optimization, denoted v = ( | ∇ R | γ R + ε ) − 1 . FIGURE 3Open in figure viewerPowerPoint Comparison of images obtained by using only L 2 norm and by using a mixture of L 2 norm and L p norm regularization term: (a) original low-light image; (b)–(d) enhancement, illumination, and reflectance with L 2 norm constrained illumination and reflectance; (e)–(g) enhancement, illumination, and reflectance with L 2 norm and L p norm constrained 3.2 Optimization We organize the model and related definitions proposed in the previous section and simplify them to obtain: min R , L ∥ R · L − I ∥ 2 2 + λ l · u ∥ ∇ L ∥ 2 2 + λ r · v ∥ ∇ R ∥ 2 2 + λ b L − I B 2 2 . (9)Due to the existence of two unknowns, the traditional calculation method no longer works, and in order to better solve for the illumination and reflectance components of the model, we use an alternating minimization algorithm for optimization calculations, and decompose the objective function with two variables into two sub-objective functions, that is, iterative loops of two independent sub-problems, updating both the illumination and reflectance variables. Setting initial values L 0 = I , R 0 = I / L 1 , and having the following formula for the kth iteration: 1 ) : L k = min L R k − 1 · L − I 2 2 + λ l · u ∥ ∇ L ∥ 2 2 + λ b L − I B 2 2 , (10) 2 ) : R k = min R R · L k − I 2 2 + λ r · v ∥ ∇ R ∥ 2 2 . (11) We can find that the above two sub-problems have closed-form global optimal solutions. The specific algorithm is as follows: (1) We rewrite the objective function into a matrix form: ( R k − 1 · L − I ) T ( R k − 1 · L − I ) + λ l ( L T D x T U x D x L + L T D y T U y D y L ) + λ b ( L − I B ) T ( L − I B ) , where D x and D y denote the Toeplitz matrix of the discrete gradient operator, U x and U y denote the diagonal matrix containing the weights u x and u y . We take the derivative of this formula and make its derivative equal to zero, and the solution of L is: L k = R k − 1 T R k − 1 + λ l D x T U x D x + D y T U y D y k + λ b · E − 1 R k − 1 T I + λ b I B , (12)where E denotes the unit matrix. (2) After we obtain the solution of L, we can similarly obtain the solution of R as follows: R k = L k T L k + λ r D x T V x D x + D y T V y D y k − 1 · L k T I . (13) We repeat the above algorithm to update the variables L and R until ε l = ∥ L k − L k − 1 ∥ / ∥ L k − 1 ∥ ≤ ε , ε r = ∥ R k − R k − 1 ∥ / ∥ R k − 1 ∥ ≤ ε , or the maximum number of iterations K. The higher the number of iterations, the less texture detail is lost in the original low-light image and the better it captures the structure of the object, which stabilizes the resulting image after a certain number of iterations. We set the two parameter values to ε = 0.01 and K= 25. 4 EXPERIMENTS 4.1 Experimental settings We set the parameters in the experiment as follows: λ l = 0.001 , λ r = 0.0001 , λ b = 0.15 , γ L = 1.3 , γ R = 0.3 . The images used in the experiment come from image collections published in several papers [9, 29, 43-47]. We compare the method presented here with several methods in recent years, including those in [30, 43, 45, 48]. 4.2 Illumination adjustment Since the reflectance obtained by Retinex is prone to over-enhancement of low-light images, we generally adjust the illumination to improve the overall visual effect of the image. For this, this paper adopts the gamma correction [49] method to modify the previously estimated illumination, and its basic form is as follows: G ( L ) = W L W 1 γ , (14)where γ is the parameter of the gamma correction, we refer to [29] and set its value to 2.2, G(L) represents the value of the corrected pixel, W= 255 and L represents the greyscale value o