# Low-Light Image Enhancement Based on Guided Image Filtering in Gradient Domain

Authors: Xiankun Sun; Huijie Liu; Shiqian Wu; Zhijun Fang; Chengfan Li; Jingyuan Yin
Journal: International Journal of Digital Multimedia Broadcasting
 (2017)

---

## Abstract

We propose a novel approach for low-light image enhancement. Based on illumination-reflection model, the guided image filter is employed to extract the illumination component of the underlying image. Afterwards, we obtain the reflection component and enhance it by nonlinear functions, sigmoid and gamma, respectively. We use the first-order edge-aware constraint in the gradient domain to achieve good edge preserving features of enhanced images and to eliminate halo artefact effectively. Moreover, the resulting images have high contrast and ample details due to the enhanced illumination and reflection component. We evaluate our method by operating on a large amount of low-light images, with comparison with other popular methods. The experimental results show that our approach outperforms the others in terms of visual perception and objective evaluation.


---

## Body

## 1. Introduction

Video surveillance is now widely used in various fields, like public security, transportation, and so on. The surveillance systems are required to perform not only in day time but also in night time. However, the videos captured in some situations such as dark place or night are very poor so that the objects can hardly be perceived by humans. Thus, it is necessary to enhance the low-light images in image processing and video surveillance.Recently, the technique of low-light image enhancement has made remarkable progress. The commonly used methods include dark channel prior model [1, 2], neural network model [3, 4], histogram equalization (HE) [5, 6], image fusion [7, 8], wavelet domain algorithm [9, 10], and illumination-reflection model [11–14]. It is noted that the adaptability of dark channel prior model is poor in disposing the images with rich details and high brightness [2]. The design and use of neural network require domain knowledge so that the underlying neural system has good generalization. The idea of histogram equalization is to merge several bins of grayscales in order to increase contrast, but such process may yield detail loss. The image fusion method needs multiple frame images, which is not applicable for single image frame. Wavelet transform is alternative technique for low-light image enhancement as shown in [9]. The most popular method in enhancing low-light images is to decompose images with the illumination-reflection model.It was proposed by Land [15] that an image J(x,y) can be decomposed by the reflection component R(x,y) and the illumination component L(x,y):(1)Jx,y=Rx,y·Lx,y,where (x,y) represents the coordinates of image. Normally, the illumination component is determined by the dynamic range of the underlying image, while the reflection component is dependent on the intrinsic characteristics of the objects within the underlying image. Equation (1) is generally converted to the logarithmic domain so that the multiplication becomes addition.The illumination component can be obtained by various methods, such as multiscale Gaussian function in [11], an improved Gaussian function in [12], and the bilateral filtering-based method in [14]. The key idea behind the Gaussian function is to use a low-pass filter as shown in (2). In the equation, F(x,y) represents a center-around function, which is also a low-pass filter, and the symbol ∗ represents the convolution. Then, the reflection component can be obtained from (3) with the illumination component.(2)Lx,y=Fx,y∗Jx,y,(3)lg⁡Rx,y=lg⁡Jx,y-lg⁡Lx,y.While calculating the illumination component and the reflection component of an image, what is most important is edge preserving and making the flat region smooth. The Multiscale Retinex (MSR) algorithm in [11] processes each color channel individually and then removes the illumination component to keep reflection component. Such processing is liable to color distortion. In addition, the method cannot achieve edge preserving and easily yields halo artefact as shown in Figure 1(b). Bilateral filtering (BLF), proposed by Tomaci and Mabduchi [16], is a good method for edge preserving and eliminating halo artefact. It was pointed out in [17] that the BLF may result in gradient reversal as shown in Figure 2(b). Accordingly, guided image filter (GIF) [17] and its variants, that is, weighted guided image filter (WGIF) [18] and guided image filter in gradient domain (GDGIF) [19] were proposed to achieve good edge preserving.Figure 1
Halo artefact.
(a)
Input image
(b)
Image with halo artefactFigure 2
Gradient reversal.
(a)
Input image
(b)
Image with gradient reversalIn this paper, we present a new method for low-light image enhancement, which mainly contributes to the following three aspects:(1) In order to effectively cope with the halo artefact and gradient reversal, the proposed method uses the illumination-reflection model and selects the GIF in gradient domain characterized by smoothness and edge preserving to estimate the illumination component; (2) the proposed method works on HSI color space to eliminate color distortion; (3) the illumination component and the reflection component are enhanced by nonlinear sigmoid and gamma transforms, respectively, to improve image contrast and enhance image details.The remainder of the paper is organized as follows: Section2 depicts the proposed low-light image enhancement algorithm. Section 3 demonstrates experimental results, followed by conclusion drawn in Section 4.


## 2. The Proposed Approach

The proposed method for enhancing low-light images consists of the following steps:(1)
Converting the low-light image from RGB color space to HSI color space(2)
For intensity (illumination) layer, estimating the illumination component with guided image filter in gradient domain, followed by extracting the reflection component(3)
Enhancing the illumination component with a nonlinear sigmoid transform(4)
Enhancing the reflection component with a nonlinear gamma transform(5)
Taking the antilog of the sum of steps(3) and (4) as the enhanced intensity layer(6)
Converting the new HSI image back to RGB, which produces the final enhanced image


### 2.1. Guided Image Filter in Gradient Domain

GDGIF [19] and WGIF [18] are both the improvements of the guided image filter. Basically, the output image qi is a linear transform of guided image gi in a square window Mk centered at the pixel k:(4)qi=akgi+bk,∀i∈Mk.In (4), guided image gi can be input image pi itself. The pixel i is located in the window Mk of length (2r+1), in which r is filter radius. ak and bk are linear coefficients in pixel k. For GIF, WGIF, and GDGIF, the larger value of ak implies better edge preserving. On the contrary, if the value of ak is much closer to 0, the filters have good smoothing performance in flat regions.According to the gradient-domain optimization framework in [20], the task of filtering an image means converting one input image pi into the final image qi, which can be expressed as an energy function Eak,bk including the zero-order data cost function Ed(i) and first-order gradient cost function Eg(i) terms: (5)Eak,bk=∑i∈MkEdi+Egi,(6)Edi=akgi+bk-pi2,(7)Egi=φi∇qi2,where ∇qi is gradient magnitude; φ(i) is gradient weight constraint. In (4), the gradient magnitude of qi is ∇qi=ak.


#### 2.1.1. Guided Image Filter

In the guided image filter [17], φ1(i)=ε, the expressions of ak and bk can be obtained according to (4)–(7) shown as follows:(8)ak=1/w∑k:i∈Mkgipi-μkpk¯σk2+ε,bk=pk¯-akμk,where μk is the mean of image gi in the window Mk, and σk2 is the variance of image gi in the window Mk, w is the total pixel number in window Mk, and pk¯ is the mean of image pi in window Mk. ε is a regularization parameter, which controls the trade-off between edge preserving and smoothness. As ε is generally fixed ratherthan spatially varying in filtering process, halo artefact is unavoidable in edges.


#### 2.1.2. Weighted Guided Image Filter

Li et al. [18] proposed the WGIF, where a spatially varying gradient weight constraint φ2(i)=ε/Γg_k was added in (7).(9)Γg_k=1N∑i=1Nσg,12k+λσg,12i+λ,where Γg_k is a single-scale edge-aware weighting, which is defined by using local variances in 3×3 windows, σg,12(k) is the variance of guided image gi in the 3×3 window, N is the number of image pixels, λ is a small constant and its value is (0.001×L)2, andL is the dynamic range of the input image. The expressions of ak′ and bk′ in the WGIF can be obtained according to (4)–(7) as shown below:(10)ak′=1/w∑k:i∈Mkgipi-μkpk¯σk2+ε/Γg_k,bk′=pk¯-ak′μk.It is noted that the edge-aware weighting Γg_k is spatially varying in the WGIF; that is, Γg_k is larger than 1 when the pixel k locates in the edge area, and Γg_k is smaller than 1 when the pixel k locates in the flat area. As a result, ak′ is closer to 1 than ak in the GIF, which implies that WGIF has the better edge preservation than GIF.WGIF can reduce the halo artefact to some extent. However, both the GIF and the WGIF filters have no explicit constraints to cope with edges. Both cannot preserve edges well because image filtering is performed on edges, which definitely smoothed the edges [19, 21].


#### 2.1.3. Guided Image Filter in Gradient Domain

Kou et al. [19] proposed the guided image filter in gradient domain (GDGIF), by adding an explicit first-order (gradient domain) edge-aware constraint Ee(i) to gradient-domain equation (5) shown below:(11)Eak,bk=∑i∈MkEdi+Egi+ϕEei,Eei=α∇qi-∇pi2,where Ee(i) is edge-aware constraint. The aim is to perceive the changes in local neighbourhoods so that the similar filtering is performed in the similar regions. α is a weight value, ∇qi and ∇pi represent the gradient values of output image qi and input image pi.In the GDGIF, a multiscale edge-aware varying spatially gradient weight constraintφ3(i)=ε/Γ^g_k is added in (7). The energy function of GDGIF is shown in (12), in which the second item is the combination of the first-order gradient cost function Eg(i) and the edge-aware constraint Ee(i).(12)Eak′′,bk′′=∑ak′′gi+bk′′-pi2+εΓ^g_kak′′-γk2,(13)Γ^g_k=1N∑i=1Nχk+λχi+λ,(14)γk=1-11+eηχk-μχ,∞,where Γ^g_k is defined by local variances of (2r+1)×(2r+1) windows of all pixels; r is the filter radius. It is noted that Γ^g_k is a multiscale edge-aware weighting varying spatially, in which χk=σg,1k∗σg,rk. The edge-aware weighting detects the edge more accurately, and a pixel is detected as an edge pixel when the two scale variances are very large.The comparison ofΓ^g_k and Γg_k in WGIF of an image is shown in Figure 3. The edges of the image are detected accurately by using the multiscale edge-aware weighting Γ^g_k rather than using the single-scale edge-aware weighting Γg_k in WGIF.Figure 3
Comparison ofΓ^g_k and Γg_k.
(a)
Original image
(b)
Edge results byΓg_k
(c)
Edge results byΓ^g_kIn (14), γk is an edge-aware constraint to preserve edges. μχ,∞ is the mean of all χi, and the value of η is 4/(μχ,∞-min⁡(χi)). The value of γk is close to 1 when the pixel k locates in the edge area, and the value is close to 0 when the pixel k locates in the flat area.The expressions ofak′′ and bk′′ in the GDGIF can be obtained according to (12)–(14), shown as follows:(15)ak′′=1/w∑k:i∈Mkgipi-μkpk¯+ε/Γ^g_kγkσk2+ε/Γ^g_k,bk′′=pk¯-ak′′μk.When the input imagepi and guided image gi are the same, the GDGIF has better edge preserving and smoother features than the GIF and the WGIF due to the following two points.(1) When pixelk locates in the edge area, the expression of ak′′ can be computed as (16)ak′′=σk2+ε/Γ^g_kγkσk2+ε/Γ^g_k.It is seen that the value of γk is 1, and the value of ak′′ is 1, which is independent of parameter ε. In fact, the value of ak′′ in GDGIF is much closer to 1 than it is in GIF and WGIF. Hence, GDGIF has the best edge preserving feature.(2) When a pixelk locates in the flat areas, the value γk is close to 0, the parameter ak′′ is accordingly independent of the choice of ε. As a result, we can select larger ε in GDGIF than that in WGIF and GIF, so that better smoothing is achieved without affecting the edge preserving [19].For example, we use GIF, WGIF, and GDGIF for image filtering by choosing the same filter radiusr=16 and regularization parameter ε=0.8∧2. The result is shown in Figure 4. It is observed that the image filtered by GDGIF has the best edge preserving.Figure 4
The difference between GIF, WGIF, and GDGIF.
(a)
Original image
(b)
Image filtered by GIF
(c)
Image filtered by WGIF
(d)
Image filtered by GDGIFIn summary, the GDGIF has the best edge preserving and smoothing features. As a result, we choose the GDGIF to estimate the illumination component.


### 2.2. Enhancing the Intensity Layer

Generally, the processing of an RGB image is to operate the R, G, and B three channels separately, which is time-consuming. In this work, the low-light image is converted from RGB color space to HSI color space. The HSI color space stems from the human visual system, using three elements of color, hue (H), saturation (S), and intensity (I) to describe the color, which is more consistent with human visual feature than RGB color space. In HSI color space, the intensity component is enhanced while the hue and saturation components are extracted without further processing.


#### 2.2.1. Estimating Illumination Component

Based on illumination-reflection model, combined with (2), (4), and (15), we use the GDGIF to estimate the illumination component LI(x,y) of intensity layer image I(x,y) as follows:(17)LIx,y=qx,y∗Ix,y,where q(x,y) is represented by the GDGIF function, as shown in (4). Then the reflection component RI(x,y) of intensity layer image can be obtained according to (3) and (17) as follows:(18)rIx,y=lg⁡RIx,y=lg⁡Ix,y-lg⁡LIx,y.


#### 2.2.2. Enhancing Illumination Component

Normally, the methods based on illumination-reflection model are to extract and then enhance the reflection component without considering the illumination component. Such processing leads to a lack of coordination between gray levels and yields color distortion. To cope with this problem, Wu et al. [13] proposed enhancing the reflection component together with the illumination component. Inspired by this idea, we process the illumination component by the nonlinear tensile sigmoid transform as shown in Figure 5. It is indicated in [22] that the sigmoid transform has the ability to sharpen images, highlight the local details, and stretch the image contrast. The nonlinear tensile sigmoid transform is expressed in the following equations:(19)lSigmoidx,y=rmin⁡+rmax⁡-rmin⁡1+e-arx,y-b·rmax⁡-rmin⁡,lI_outx,y=lIx,y×lSigmoidx,y,where lSigmoid is self-defined sigmoid nonlinear function, whose range is [0,1], rmin⁡ and rmax⁡ represent the minimum and maximum values of reflection component, respectively, and lI(x,y) and lI_out(x,y) are the initial and the enhanced illumination component, respectively. It is noted that there are two important parameters a, b in which the parameter a controls how an image is enhanced and the parameter b controls the contrast enhancement. Generally, a large value of a enhances an image greatly. On the other hand, a small value of b enhances the contrast of the dark regions, and a large value of b enhances the contrast of the light regions. Figure 6 shows examples by taking different parameters. In this study, the parameters a, b are selected as 2 and 0.004, respectively.Figure 5
Sigmoid function curves.
(a)
Sigmoid curves in differenta values (b=0.004)
(b)
Sigmoid curves in differentb values (a=2)Figure 6
Illumination component enhanced by sigmoid curves.
(a)
Illumination enhancement in differenta values (b=0.004)
(b)
Illumination enhancement in differentb values (a=2)


#### 2.2.3. Enhancing the Reflection Component

It is known that human eyes are not sensible for high brightness difference but sensible for the small difference in low intensity. Thus, gamma transform [23] is normally employed to enhance the reflection component as follows:(20)rI_outx,y=crIγx,y,where c is a positive constant; γ is a parameter to control the image contrast.It is seen from Figure7 that when γ is less than 1, the contrast in low intensities is increased. On the contrary, the contrast in high intensities is enhanced in case γ>1. The effect of parameter c is shown in Figure 8. In this work, the parameters are experimentally selected as follows: γ=0.6; c=0.3.Figure 7
Gamma transforms(c=1).Figure 8
Reflection component enhanced by differentc values.


### 2.3. Final Enhanced Image

By taking antilog of the illumination component combined with the reflection component, we obtain the enhanced intensity image. The new HSI image is made up of enhanced intensity layer, original hue, and saturation layer. Then the enhanced HSI image is converted into RGB image to obtain the final enhanced image. The whole process is shown in Figure9.Figure 9
The proposed procedure of low-light images.
(a)
Original
(b)
Intensity layer image
(c)
Illumination component
(d)
Reflection component
(e)
Enhanced illumination component
(f)
Enhanced reflection component
(g)
Enhanced intensity image
(h)
Enhanced RGB image


## 2.1. Guided Image Filter in Gradient Domain

GDGIF [19] and WGIF [18] are both the improvements of the guided image filter. Basically, the output image qi is a linear transform of guided image gi in a square window Mk centered at the pixel k:(4)qi=akgi+bk,∀i∈Mk.In (4), guided image gi can be input image pi itself. The pixel i is located in the window Mk of length (2r+1), in which r is filter radius. ak and bk are linear coefficients in pixel k. For GIF, WGIF, and GDGIF, the larger value of ak implies better edge preserving. On the contrary, if the value of ak is much closer to 0, the filters have good smoothing performance in flat regions.According to the gradient-domain optimization framework in [20], the task of filtering an image means converting one input image pi into the final image qi, which can be expressed as an energy function Eak,bk including the zero-order data cost function Ed(i) and first-order gradient cost function Eg(i) terms: (5)Eak,bk=∑i∈MkEdi+Egi,(6)Edi=akgi+bk-pi2,(7)Egi=φi∇qi2,where ∇qi is gradient magnitude; φ(i) is gradient weight constraint. In (4), the gradient magnitude of qi is ∇qi=ak.


### 2.1.1. Guided Image Filter

In the guided image filter [17], φ1(i)=ε, the expressions of ak and bk can be obtained according to (4)–(7) shown as follows:(8)ak=1/w∑k:i∈Mkgipi-μkpk¯σk2+ε,bk=pk¯-akμk,where μk is the mean of image gi in the window Mk, and σk2 is the variance of image gi in the window Mk, w is the total pixel number in window Mk, and pk¯ is the mean of image pi in window Mk. ε is a regularization parameter, which controls the trade-off between edge preserving and smoothness. As ε is generally fixed ratherthan spatially varying in filtering process, halo artefact is unavoidable in edges.


### 2.1.2. Weighted Guided Image Filter

Li et al. [18] proposed the WGIF, where a spatially varying gradient weight constraint φ2(i)=ε/Γg_k was added in (7).(9)Γg_k=1N∑i=1Nσg,12k+λσg,12i+λ,where Γg_k is a single-scale edge-aware weighting, which is defined by using local variances in 3×3 windows, σg,12(k) is the variance of guided image gi in the 3×3 window, N is the number of image pixels, λ is a small constant and its value is (0.001×L)2, andL is the dynamic range of the input image. The expressions of ak′ and bk′ in the WGIF can be obtained according to (4)–(7) as shown below:(10)ak′=1/w∑k:i∈Mkgipi-μkpk¯σk2+ε/Γg_k,bk′=pk¯-ak′μk.It is noted that the edge-aware weighting Γg_k is spatially varying in the WGIF; that is, Γg_k is larger than 1 when the pixel k locates in the edge area, and Γg_k is smaller than 1 when the pixel k locates in the flat area. As a result, ak′ is closer to 1 than ak in the GIF, which implies that WGIF has the better edge preservation than GIF.WGIF can reduce the halo artefact to some extent. However, both the GIF and the WGIF filters have no explicit constraints to cope with edges. Both cannot preserve edges well because image filtering is performed on edges, which definitely smoothed the edges [19, 21].


### 2.1.3. Guided Image Filter in Gradient Domain

Kou et al. [19] proposed the guided image filter in gradient domain (GDGIF), by adding an explicit first-order (gradient domain) edge-aware constraint Ee(i) to gradient-domain equation (5) shown below:(11)Eak,bk=∑i∈MkEdi+Egi+ϕEei,Eei=α∇qi-∇pi2,where Ee(i) is edge-aware constraint. The aim is to perceive the changes in local neighbourhoods so that the similar filtering is performed in the similar regions. α is a weight value, ∇qi and ∇pi represent the gradient values of output image qi and input image pi.In the GDGIF, a multiscale edge-aware varying spatially gradient weight constraintφ3(i)=ε/Γ^g_k is added in (7). The energy function of GDGIF is shown in (12), in which the second item is the combination of the first-order gradient cost function Eg(i) and the edge-aware constraint Ee(i).(12)Eak′′,bk′′=∑ak′′gi+bk′′-pi2+εΓ^g_kak′′-γk2,(13)Γ^g_k=1N∑i=1Nχk+λχi+λ,(14)γk=1-11+eηχk-μχ,∞,where Γ^g_k is defined by local variances of (2r+1)×(2r+1) windows of all pixels; r is the filter radius. It is noted that Γ^g_k is a multiscale edge-aware weighting varying spatially, in which χk=σg,1k∗σg,rk. The edge-aware weighting detects the edge more accurately, and a pixel is detected as an edge pixel when the two scale variances are very large.The comparison ofΓ^g_k and Γg_k in WGIF of an image is shown in Figure 3. The edges of the image are detected accurately by using the multiscale edge-aware weighting Γ^g_k rather than using the single-scale edge-aware weighting Γg_k in WGIF.Figure 3
Comparison ofΓ^g_k and Γg_k.
(a)
Original image
(b)
Edge results byΓg_k
(c)
Edge results byΓ^g_kIn (14), γk is an edge-aware constraint to preserve edges. μχ,∞ is the mean of all χi, and the value of η is 4/(μχ,∞-min⁡(χi)). The value of γk is close to 1 when the pixel k locates in the edge area, and the value is close to 0 when the pixel k locates in the flat area.The expressions ofak′′ and bk′′ in the GDGIF can be obtained according to (12)–(14), shown as follows:(15)ak′′=1/w∑k:i∈Mkgipi-μkpk¯+ε/Γ^g_kγkσk2+ε/Γ^g_k,bk′′=pk¯-ak′′μk.When the input imagepi and guided image gi are the same, the GDGIF has better edge preserving and smoother features than the GIF and the WGIF due to the following two points.(1) When pixelk locates in the edge area, the expression of ak′′ can be computed as (16)ak′′=σk2+ε/Γ^g_kγkσk2+ε/Γ^g_k.It is seen that the value of γk is 1, and the value of ak′′ is 1, which is independent of parameter ε. In fact, the value of ak′′ in GDGIF is much closer to 1 than it is in GIF and WGIF. Hence, GDGIF has the best edge preserving feature.(2) When a pixelk locates in the flat areas, the value γk is close to 0, the parameter ak′′ is accordingly independent of the choice of ε. As a result, we can select larger ε in GDGIF than that in WGIF and GIF, so that better smoothing is achieved without affecting the edge preserving [19].For example, we use GIF, WGIF, and GDGIF for image filtering by choosing the same filter radiusr=16 and regularization parameter ε=0.8∧2. The result is shown in Figure 4. It is observed that the image filtered by GDGIF has the best edge preserving.Figure 4
The difference between GIF, WGIF, and GDGIF.
(a)
Original image
(b)
Image filtered by GIF
(c)
Image filtered by WGIF
(d)
Image filtered by GDGIFIn summary, the GDGIF has the best edge preserving and smoothing features. As a result, we choose the GDGIF to estimate the illumination component.


## 2.1.1. Guided Image Filter

In the guided image filter [17], φ1(i)=ε, the expressions of ak and bk can be obtained according to (4)–(7) shown as follows:(8)ak=1/w∑k:i∈Mkgipi-μkpk¯σk2+ε,bk=pk¯-akμk,where μk is the mean of image gi in the window Mk, and σk2 is the variance of image gi in the window Mk, w is the total pixel number in window Mk, and pk¯ is the mean of image pi in window Mk. ε is a regularization parameter, which controls the trade-off between edge preserving and smoothness. As ε is generally fixed ratherthan spatially varying in filtering process, halo artefact is unavoidable in edges.


## 2.1.2. Weighted Guided Image Filter

Li et al. [18] proposed the WGIF, where a spatially varying gradient weight constraint φ2(i)=ε/Γg_k was added in (7).(9)Γg_k=1N∑i=1Nσg,12k+λσg,12i+λ,where Γg_k is a single-scale edge-aware weighting, which is defined by using local variances in 3×3 windows, σg,12(k) is the variance of guided image gi in the 3×3 window, N is the number of image pixels, λ is a small constant and its value is (0.001×L)2, andL is the dynamic range of the input image. The expressions of ak′ and bk′ in the WGIF can be obtained according to (4)–(7) as shown below:(10)ak′=1/w∑k:i∈Mkgipi-μkpk¯σk2+ε/Γg_k,bk′=pk¯-ak′μk.It is noted that the edge-aware weighting Γg_k is spatially varying in the WGIF; that is, Γg_k is larger than 1 when the pixel k locates in the edge area, and Γg_k is smaller than 1 when the pixel k locates in the flat area. As a result, ak′ is closer to 1 than ak in the GIF, which implies that WGIF has the better edge preservation than GIF.WGIF can reduce the halo artefact to some extent. However, both the GIF and the WGIF filters have no explicit constraints to cope with edges. Both cannot preserve edges well because image filtering is performed on edges, which definitely smoothed the edges [19, 21].


## 2.1.3. Guided Image Filter in Gradient Domain

Kou et al. [19] proposed the guided image filter in gradient domain (GDGIF), by adding an explicit first-order (gradient domain) edge-aware constraint Ee(i) to gradient-domain equation (5) shown below:(11)Eak,bk=∑i∈MkEdi+Egi+ϕEei,Eei=α∇qi-∇pi2,where Ee(i) is edge-aware constraint. The aim is to perceive the changes in local neighbourhoods so that the similar filtering is performed in the similar regions. α is a weight value, ∇qi and ∇pi represent the gradient values of output image qi and input image pi.In the GDGIF, a multiscale edge-aware varying spatially gradient weight constraintφ3(i)=ε/Γ^g_k is added in (7). The energy function of GDGIF is shown in (12), in which the second item is the combination of the first-order gradient cost function Eg(i) and the edge-aware constraint Ee(i).(12)Eak′′,bk′′=∑ak′′gi+bk′′-pi2+εΓ^g_kak′′-γk2,(13)Γ^g_k=1N∑i=1Nχk+λχi+λ,(14)γk=1-11+eηχk-μχ,∞,where Γ^g_k is defined by local variances of (2r+1)×(2r+1) windows of all pixels; r is the filter radius. It is noted that Γ^g_k is a multiscale edge-aware weighting varying spatially, in which χk=σg,1k∗σg,rk. The edge-aware weighting detects the edge more accurately, and a pixel is detected as an edge pixel when the two scale variances are very large.The comparison ofΓ^g_k and Γg_k in WGIF of an image is shown in Figure 3. The edges of the image are detected accurately by using the multiscale edge-aware weighting Γ^g_k rather than using the single-scale edge-aware weighting Γg_k in WGIF.Figure 3
Comparison ofΓ^g_k and Γg_k.
(a)
Original image
(b)
Edge results byΓg_k
(c)
Edge results byΓ^g_kIn (14), γk is an edge-aware constraint to preserve edges. μχ,∞ is the mean of all χi, and the value of η is 4/(μχ,∞-min⁡(χi)). The value of γk is close to 1 when the pixel k locates in the edge area, and the value is close to 0 when the pixel k locates in the flat area.The expressions ofak′′ and bk′′ in the GDGIF can be obtained according to (12)–(14), shown as follows:(15)ak′′=1/w∑k:i∈Mkgipi-μkpk¯+ε/Γ^g_kγkσk2+ε/Γ^g_k,bk′′=pk¯-ak′′μk.When the input imagepi and guided image gi are the same, the GDGIF has better edge preserving and smoother features than the GIF and the WGIF due to the following two points.(1) When pixelk locates in the edge area, the expression of ak′′ can be computed as (16)ak′′=σk2+ε/Γ^g_kγkσk2+ε/Γ^g_k.It is seen that the value of γk is 1, and the value of ak′′ is 1, which is independent of parameter ε. In fact, the value of ak′′ in GDGIF is much closer to 1 than it is in GIF and WGIF. Hence, GDGIF has the best edge preserving feature.(2) When a pixelk locates in the flat areas, the value γk is close to 0, the parameter ak′′ is accordingly independent of the choice of ε. As a result, we can select larger ε in GDGIF than that in WGIF and GIF, so that better smoothing is achieved without affecting the edge preserving [19].For example, we use GIF, WGIF, and GDGIF for image filtering by choosing the same filter radiusr=16 and regularization parameter ε=0.8∧2. The result is shown in Figure 4. It is observed that the image filtered by GDGIF has the best edge preserving.Figure 4
The difference between GIF, WGIF, and GDGIF.
(a)
Original image
(b)
Image filtered by GIF
(c)
Image filtered by WGIF
(d)
Image filtered by GDGIFIn summary, the GDGIF has the best edge preserving and smoothing features. As a result, we choose the GDGIF to estimate the illumination component.


## 2.2. Enhancing the Intensity Layer

Generally, the processing of an RGB image is to operate the R, G, and B three channels separately, which is time-consuming. In this work, the low-light image is converted from RGB color space to HSI color space. The HSI color space stems from the human visual system, using three elements of color, hue (H), saturation (S), and intensity (I) to describe the color, which is more consistent with human visual feature than RGB color space. In HSI color space, the intensity component is enhanced while the hue and saturation components are extracted without further processing.


### 2.2.1. Estimating Illumination Component

Based on illumination-reflection model, combined with (2), (4), and (15), we use the GDGIF to estimate the illumination component LI(x,y) of intensity layer image I(x,y) as follows:(17)LIx,y=qx,y∗Ix,y,where q(x,y) is represented by the GDGIF function, as shown in (4). Then the reflection component RI(x,y) of intensity layer image can be obtained according to (3) and (17) as follows:(18)rIx,y=lg⁡RIx,y=lg⁡Ix,y-lg⁡LIx,y.


### 2.2.2. Enhancing Illumination Component

Normally, the methods based on illumination-reflection model are to extract and then enhance the reflection component without considering the illumination component. Such processing leads to a lack of coordination between gray levels and yields color distortion. To cope with this problem, Wu et al. [13] proposed enhancing the reflection component together with the illumination component. Inspired by this idea, we process the illumination component by the nonlinear tensile sigmoid transform as shown in Figure 5. It is indicated in [22] that the sigmoid transform has the ability to sharpen images, highlight the local details, and stretch the image contrast. The nonlinear tensile sigmoid transform is expressed in the following equations:(19)lSigmoidx,y=rmin⁡+rmax⁡-rmin⁡1+e-arx,y-b·rmax⁡-rmin⁡,lI_outx,y=lIx,y×lSigmoidx,y,where lSigmoid is self-defined sigmoid nonlinear function, whose range is [0,1], rmin⁡ and rmax⁡ represent the minimum and maximum values of reflection component, respectively, and lI(x,y) and lI_out(x,y) are the initial and the enhanced illumination component, respectively. It is noted that there are two important parameters a, b in which the parameter a controls how an image is enhanced and the parameter b controls the contrast enhancement. Generally, a large value of a enhances an image greatly. On the other hand, a small value of b enhances the contrast of the dark regions, and a large value of b enhances the contrast of the light regions. Figure 6 shows examples by taking different parameters. In this study, the parameters a, b are selected as 2 and 0.004, respectively.Figure 5
Sigmoid function curves.
(a)
Sigmoid curves in differenta values (b=0.004)
(b)
Sigmoid curves in differentb values (a=2)Figure 6
Illumination component enhanced by sigmoid curves.
(a)
Illumination enhancement in differenta values (b=0.004)
(b)
Illumination enhancement in differentb values (a=2)


### 2.2.3. Enhancing the Reflection Component

It is known that human eyes are not sensible for high brightness difference but sensible for the small difference in low intensity. Thus, gamma transform [23] is normally employed to enhance the reflection component as follows:(20)rI_outx,y=crIγx,y,where c is a positive constant; γ is a parameter to control the image contrast.It is seen from Figure7 that when γ is less than 1, the contrast in low intensities is increased. On the contrary, the contrast in high intensities is enhanced in case γ>1. The effect of parameter c is shown in Figure 8. In this work, the parameters are experimentally selected as follows: γ=0.6; c=0.3.Figure 7
Gamma transforms(c=1).Figure 8
Reflection component enhanced by differentc values.


## 2.2.1. Estimating Illumination Component

Based on illumination-reflection model, combined with (2), (4), and (15), we use the GDGIF to estimate the illumination component LI(x,y) of intensity layer image I(x,y) as follows:(17)LIx,y=qx,y∗Ix,y,where q(x,y) is represented by the GDGIF function, as shown in (4). Then the reflection component RI(x,y) of intensity layer image can be obtained according to (3) and (17) as follows:(18)rIx,y=lg⁡RIx,y=lg⁡Ix,y-lg⁡LIx,y.


## 2.2.2. Enhancing Illumination Component

Normally, the methods based on illumination-reflection model are to extract and then enhance the reflection component without considering the illumination component. Such processing leads to a lack of coordination between gray levels and yields color distortion. To cope with this problem, Wu et al. [13] proposed enhancing the reflection component together with the illumination component. Inspired by this idea, we process the illumination component by the nonlinear tensile sigmoid transform as shown in Figure 5. It is indicated in [22] that the sigmoid transform has the ability to sharpen images, highlight the local details, and stretch the image contrast. The nonlinear tensile sigmoid transform is expressed in the following equations:(19)lSigmoidx,y=rmin⁡+rmax⁡-rmin⁡1+e-arx,y-b·rmax⁡-rmin⁡,lI_outx,y=lIx,y×lSigmoidx,y,where lSigmoid is self-defined sigmoid nonlinear function, whose range is [0,1], rmin⁡ and rmax⁡ represent the minimum and maximum values of reflection component, respectively, and lI(x,y) and lI_out(x,y) are the initial and the enhanced illumination component, respectively. It is noted that there are two important parameters a, b in which the parameter a controls how an image is enhanced and the parameter b controls the contrast enhancement. Generally, a large value of a enhances an image greatly. On the other hand, a small value of b enhances the contrast of the dark regions, and a large value of b enhances the contrast of the light regions. Figure 6 shows examples by taking different parameters. In this study, the parameters a, b are selected as 2 and 0.004, respectively.Figure 5
Sigmoid function curves.
(a)
Sigmoid curves in differenta values (b=0.004)
(b)
Sigmoid curves in differentb values (a=2)Figure 6
Illumination component enhanced by sigmoid curves.
(a)
Illumination enhancement in differenta values (b=0.004)
(b)
Illumination enhancement in differentb values (a=2)


## 2.2.3. Enhancing the Reflection Component

It is known that human eyes are not sensible for high brightness difference but sensible for the small difference in low intensity. Thus, gamma transform [23] is normally employed to enhance the reflection component as follows:(20)rI_outx,y=crIγx,y,where c is a positive constant; γ is a parameter to control the image contrast.It is seen from Figure7 that when γ is less than 1, the contrast in low intensities is increased. On the contrary, the contrast in high intensities is enhanced in case γ>1. The effect of parameter c is shown in Figure 8. In this work, the parameters are experimentally selected as follows: γ=0.6; c=0.3.Figure 7
Gamma transforms(c=1).Figure 8
Reflection component enhanced by differentc values.


## 2.3. Final Enhanced Image

By taking antilog of the illumination component combined with the reflection component, we obtain the enhanced intensity image. The new HSI image is made up of enhanced intensity layer, original hue, and saturation layer. Then the enhanced HSI image is converted into RGB image to obtain the final enhanced image. The whole process is shown in Figure9.Figure 9
The proposed procedure of low-light images.
(a)
Original
(b)
Intensity layer image
(c)
Illumination component
(d)
Reflection component
(e)
Enhanced illumination component
(f)
Enhanced reflection component
(g)
Enhanced intensity image
(h)
Enhanced RGB image


## 3. Experimental Results and Discussions

As there is no public low-light image database, we collect 20 images from the Library of Congress and Internet as shown in Figure10. In the simulation, the parameters are predefined as follows: window radius r=16, regularization parameter ε=0.001, the sigmoid parameters a=2, b=0.004, and gamma parameters γ=0.6, c=0.3. All experiments are performed in Matlab code on Windows 7 operation system. The computer is Intel® core™ i5-4570 3.20 GHz, and the RAM is 4.00 GB. The popular algorithms such as traditional MSR [11], Hao’s algorithm [14], He’s algorithm [17], histogram equalization (HE), improved MSR [12], and Kim’s algorithm [9] are implemented for performance comparison.Figure 10
Datasets used in experiments.


### 3.1. Subjective Evaluation

It is observed from Figures11 and 12 that the images enhanced by traditional MSR algorithm have obvious “white” phenomenon, which indicates color distortion. The images enhanced by Hao’s method result in gradient reversal artefact as seen in red square from image 4 (white tower) in Figure 11, which indicates that the bilateral filtering has poor edge preserving property. Moreover, the resulting images by Hao’s method are blurred which can be seen from images 1 (church) and 5 (the study) in Figure 11. He’s algorithm also shows halo artefact which can be seen from image 1 (church), and this method cannot enhance the image contrast and brightness shown in Figure 12.Figure 11
Comparison among different algorithms.Figure 12
Results by different algorithms.Histogram equalization algorithm can enhance image contrast, but this method misses details and enlarges noise, which can be observed from Figures11 and 12. Improved MSR algorithm is better than the traditional MSR algorithm, but this method causes halo artefact which can be seen from image 1 (church) in Figure 11 (highlighted box). Moreover, “white” phenomenon exists as shown in image 13 (boy), where the tire is over-enhanced and loses detailed information. Our results show that improved MSR algorithm is not effective in dealing with night images, as observed in Figure 12.Kim’s algorithm also causes color distortion, which can be seen from image 1 (church) in Figure11, and the sky in the red square becomes gray. The enhanced images may be blurred as seen in image 1 (church), which implies that Kim’s algorithm has poor edge preserving.As seen from Figures11 and 12, the proposed algorithm has the best color fidelity and edge preserving. Also, the enhanced images have less noise and clearer details than the one enhanced by histogram equalization algorithm.


### 3.2. Objective Evaluation

Currently, no standard objective metrics are proposed for enhancement assessment of low-light images. In this study, we employ information entropy (IE) to evaluate image details and choose the average edge intensity (AEI) to evaluate the edge preserving feature of enhanced images. The IE is defined as follows:(21)IE=-∑s=0L-1Prslog2⁡Prs,where P(rs) represents the probability of gray level s; L is the total number of gray levels. Larger IE indicates more information, which implies that there are more details in the underlying image.The AEI is defined as follows:(22)AEI=Egi,j=ESxi,j2+Syi,j2,where Sx(i,j) and Sy(i,j) represent gradients in x and y directions on image edges. E represents expectation. In this study, Sobel operator is used for gradient computation. Larger AEI indicates good edge preserving. Also, the efficiency of the proposed method is compared with other methods.Table1 shows the average IE and AEI and operating time based on 20 test images. It is observed that the performance of He’s method is not good in terms of IE. The images enhanced by He’s method are usually dark and lose many details. Figure 13 shows the IE performances of the best three methods, that is, Hao’s algorithm, Kim’s algorithm, and the proposed method. It is seen that the median IE (marked by a red line) of the proposed method is the largest. The mean IE (marked by a green line) of the proposed method is 9.8% higher than Hao’s algorithm and 1.8% higher than Kim’s algorithm in the 20 test images.Table 1
The average IE, AEI, and operating time (sec) on twenty images.
Average IE
Average AEI
Average time
Original
11.057
50.756
N/R
Traditional MSR
13.439
67.485
1.77
Hao et al.’s [14]
14.091
57.838
16.289
He et al.’s [17]
11.867
61.952
0.377
HE
14.761
83.874
0.281
Improved MSR [12]
13.234
83.606
7.915
Kim et al.’s [9]
15.193
96.477
0.867
The proposed method
15.478
114.297
0.698Figure 13
IE results by Hao’s, Kim’s, and the proposed method.On the other hand, it is observed from Table1 that the improved MSR, histogram equalization (HE), Kim’s algorithm, and the proposed algorithm achieve top four performances in terms of AEI. It is noted that high AEI in the histogram equalization accounts for much noise. The AEI of the improved MSR, Kim’s method, and the proposed method on each individual image are shown in Figure 14. Results show that the median AEI (marked by a red line) of the proposed method is the largest, and the mean AEI (marked by a green line) of the proposed method is 26.8% higher than the improved MSR and 15.6% higher than Kim’s algorithm.Figure 14
AEI results by Kim’s, improved MSR, and the proposed method.In summary, the proposed method achieves the best performance in terms of both IE and AEI. Furthermore, the proposed method is very efficient and achieves the 3rd position among the 7 methods as shown in Table1. It is noted that He’s algorithm has the worst enhanced image quality. Figure 15 shows the median operating time (marked by a red line) and the mean operating time (marked by a green line) of HE, Kim’s, and the proposed method. Generally, the bilateral filtering is not as efficient as the GDGIF. The computation complexity of bilateral filtering is o(Nr2), where r refers to the window radius of bilateral filtering, and N is the number of image pixels while the GDGIF’s computation complexity is o(N). In other words, the GDGIF’s computation complexity is not related to the filter size. Results show that the average time by the proposed method is 59.7% higher than HE and 19.4% less than Kim’s algorithm.Figure 15
Operating time by HE, Kim’s, and the proposed method.


## 3.1. Subjective Evaluation

It is observed from Figures11 and 12 that the images enhanced by traditional MSR algorithm have obvious “white” phenomenon, which indicates color distortion. The images enhanced by Hao’s method result in gradient reversal artefact as seen in red square from image 4 (white tower) in Figure 11, which indicates that the bilateral filtering has poor edge preserving property. Moreover, the resulting images by Hao’s method are blurred which can be seen from images 1 (church) and 5 (the study) in Figure 11. He’s algorithm also shows halo artefact which can be seen from image 1 (church), and this method cannot enhance the image contrast and brightness shown in Figure 12.Figure 11
Comparison among different algorithms.Figure 12
Results by different algorithms.Histogram equalization algorithm can enhance image contrast, but this method misses details and enlarges noise, which can be observed from Figures11 and 12. Improved MSR algorithm is better than the traditional MSR algorithm, but this method causes halo artefact which can be seen from image 1 (church) in Figure 11 (highlighted box). Moreover, “white” phenomenon exists as shown in image 13 (boy), where the tire is over-enhanced and loses detailed information. Our results show that improved MSR algorithm is not effective in dealing with night images, as observed in Figure 12.Kim’s algorithm also causes color distortion, which can be seen from image 1 (church) in Figure11, and the sky in the red square becomes gray. The enhanced images may be blurred as seen in image 1 (church), which implies that Kim’s algorithm has poor edge preserving.As seen from Figures11 and 12, the proposed algorithm has the best color fidelity and edge preserving. Also, the enhanced images have less noise and clearer details than the one enhanced by histogram equalization algorithm.


## 3.2. Objective Evaluation

Currently, no standard objective metrics are proposed for enhancement assessment of low-light images. In this study, we employ information entropy (IE) to evaluate image details and choose the average edge intensity (AEI) to evaluate the edge preserving feature of enhanced images. The IE is defined as follows:(21)IE=-∑s=0L-1Prslog2⁡Prs,where P(rs) represents the probability of gray level s; L is the total number of gray levels. Larger IE indicates more information, which implies that there are more details in the underlying image.The AEI is defined as follows:(22)AEI=Egi,j=ESxi,j2+Syi,j2,where Sx(i,j) and Sy(i,j) represent gradients in x and y directions on image edges. E represents expectation. In this study, Sobel operator is used for gradient computation. Larger AEI indicates good edge preserving. Also, the efficiency of the proposed method is compared with other methods.Table1 shows the average IE and AEI and operating time based on 20 test images. It is observed that the performance of He’s method is not good in terms of IE. The images enhanced by He’s method are usually dark and lose many details. Figure 13 shows the IE performances of the best three methods, that is, Hao’s algorithm, Kim’s algorithm, and the proposed method. It is seen that the median IE (marked by a red line) of the proposed method is the largest. The mean IE (marked by a green line) of the proposed method is 9.8% higher than Hao’s algorithm and 1.8% higher than Kim’s algorithm in the 20 test images.Table 1
The average IE, AEI, and operating time (sec) on twenty images.
Average IE
Average AEI
Average time
Original
11.057
50.756
N/R
Traditional MSR
13.439
67.485
1.77
Hao et al.’s [14]
14.091
57.838
16.289
He et al.’s [17]
11.867
61.952
0.377
HE
14.761
83.874
0.281
Improved MSR [12]
13.234
83.606
7.915
Kim et al.’s [9]
15.193
96.477
0.867
The proposed method
15.478
114.297
0.698Figure 13
IE results by Hao’s, Kim’s, and the proposed method.On the other hand, it is observed from Table1 that the improved MSR, histogram equalization (HE), Kim’s algorithm, and the proposed algorithm achieve top four performances in terms of AEI. It is noted that high AEI in the histogram equalization accounts for much noise. The AEI of the improved MSR, Kim’s method, and the proposed method on each individual image are shown in Figure 14. Results show that the median AEI (marked by a red line) of the proposed method is the largest, and the mean AEI (marked by a green line) of the proposed method is 26.8% higher than the improved MSR and 15.6% higher than Kim’s algorithm.Figure 14
AEI results by Kim’s, improved MSR, and the proposed method.In summary, the proposed method achieves the best performance in terms of both IE and AEI. Furthermore, the proposed method is very efficient and achieves the 3rd position among the 7 methods as shown in Table1. It is noted that He’s algorithm has the worst enhanced image quality. Figure 15 shows the median operating time (marked by a red line) and the mean operating time (marked by a green line) of HE, Kim’s, and the proposed method. Generally, the bilateral filtering is not as efficient as the GDGIF. The computation complexity of bilateral filtering is o(Nr2), where r refers to the window radius of bilateral filtering, and N is the number of image pixels while the GDGIF’s computation complexity is o(N). In other words, the GDGIF’s computation complexity is not related to the filter size. Results show that the average time by the proposed method is 59.7% higher than HE and 19.4% less than Kim’s algorithm.Figure 15
Operating time by HE, Kim’s, and the proposed method.


## 4. Conclusion

A low-light image enhancement algorithm is presented in the paper. By decomposing a low-light image into the illumination component and the reflection component, it offers a solution to expand illumination and enhances image details separately. Specifically, the illumination component is processed using guided image filter in gradient domain, followed by nonlinear sigmoid transform. The reflection component is enhanced by the gamma transform. This solution enhances low-light images and effectively avoids distortions (for example color) and annoying artefacts (e.g., blurring, halo). Then, the final result is obtained by antilogging the sum of the enhanced two components. Experimental results demonstrate that the enhanced images by the proposed method are visually pleasing by subjective test and the performance of the proposed method outperforms the existing methods in terms of both IE and AEI assessments. Moreover, the proposed algorithm is efficient because the computation complexity is not related to filter size. The proposed method has great potential to implement in real-time low-light video processing.



---

*Source: 9029315-2017-09-28.xml*