Attention‐based multi‐channel feature fusion enhancement network to process low‐light images

IET Image ProcessingVolume 16, Issue 12 p. 3374-3393 ORIGINAL RESEARCHOpen Access Attention-based multi-channel feature fusion enhancement network to process low-light images Xintao Xu, Xintao Xu School of Information and electronic engineering, Shandong Technology and Business University, Yantai, China Institute of Network Technology(INT), Yantai, ChinaSearch for more papers by this authorJinjiang Li, Jinjiang Li orcid.org/0000-0002-2080-8678 School of Computer Science and Technology, Shandong Technology and Business University, Yantai, China Institute of Network Technology(INT), Yantai, ChinaSearch for more papers by this authorZhen Hua, Corresponding Author Zhen Hua huazhen66@foxmail.com orcid.org/0000-0003-1638-2974 School of Information and electronic engineering, Shandong Technology and Business University, Yantai, China Institute of Network Technology(INT), Yantai, China Correspondence Zhen HUA, School of Information and electronic engineering, Shandong Technology and Business University, Yantai 264005, China. Email: huazhen66@foxmail.comSearch for more papers by this authorLinwei Fan, Linwei Fan School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan, ChinaSearch for more papers by this author Xintao Xu, Xintao Xu School of Information and electronic engineering, Shandong Technology and Business University, Yantai, China Institute of Network Technology(INT), Yantai, ChinaSearch for more papers by this authorJinjiang Li, Jinjiang Li orcid.org/0000-0002-2080-8678 School of Computer Science and Technology, Shandong Technology and Business University, Yantai, China Institute of Network Technology(INT), Yantai, ChinaSearch for more papers by this authorZhen Hua, Corresponding Author Zhen Hua huazhen66@foxmail.com orcid.org/0000-0003-1638-2974 School of Information and electronic engineering, Shandong Technology and Business University, Yantai, China Institute of Network Technology(INT), Yantai, China Correspondence Zhen HUA, School of Information and electronic engineering, Shandong Technology and Business University, Yantai 264005, China. Email: huazhen66@foxmail.comSearch for more papers by this authorLinwei Fan, Linwei Fan School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan, ChinaSearch for more papers by this author First published: 25 June 2022 https://doi.org/10.1049/ipr2.12571AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract In realistic low-light environments, images captured by imaging devices often have problems such as low brightness and low contrast, serious loss of detail information, and a large amount of noise, posing major challenges to computer vision tasks. Low-light image enhancement can effectively improve the overall quality of the image, which has important significance and application value. In this study, an attention-based multi-channel feature fusion enhancement network (M-FFENet) is proposed to process low-light images. In this network, a feature extraction model is first used to obtain the deep features of the downsampled low-light images and fit them to an affine bilateral grid. Second, the addition of attention-based residual dense blocks (ARDB) allows the network to focus on more details and spatial information. Meanwhile, all color channels are considered. The channel features and bilateral meshes are then linearly interpolated using the feature reconfiguration model (FRM) to obtain high-quality features containing rich color and texture information. Next, the feature fusion module (FFM) is used to fuse features that contain different information. Enhancement model is used to further recover texture and detail in the image. Finally, the enhanced image is output. Numerous experimental results have shown that the method achieves better results in both quantitative and qualitative aspects compared to other methods. 1 INTRODUCTION Since deep learning is widely used in several fields, image processing techniques are constantly updated and developed, which has become one of the hotspots of research in recent years. High-quality image data is the basis of information transmission. However, in the actual image acquisition process, it is easily affected by bad weather, weak ambient light or the way of shooting. Generate a lot of noise and other problems that seriously affect the image quality. Such images not only fail to meet the demand for image quality, but also can greatly reduce the value of the image for use. The use of low-light enhancement techniques for such images can significantly improve brightness, enhance contrast, and extract detailed features of the image according to people's needs for easy analysis and processing by computer vision devices. Therefore, the study of low-light image enhancement techniques can be applied to a variety of fields such as target detection [1], behavioral analysis, and face recognition [2, 3], and plays a special role and value in these fields. Since image processing technology boomed, scholars have conducted many research on low-light images and proposed a range of effective image enhancement methods, among which the representative methods are: histogram equalization-based methods, Retinex theory-based methods and dehazing-based methods. Histogram equalization [4-6] is a class of methods by adjusting the grayscale histogram of the original image. As this type of method only enhances the overall area of the image without considering the local features of the image, over-enhancement and loss of detail information can easily occur. In the Retinex methods [7-9], the low-light image is decomposed into reflected and irradiated components, and the brightness of the image at different wavelengths is estimated by calculating the path between the target and the neighboring pixel points for image enhancement. For example, single-scale Retinex (SSR) [10] performs a filtering operation on the original image using a low-pass filter to obtain a reflection map, which is used as the output. Multi-sacle Retinex (MSR) [11] uses multiple low-pass filters based on SSR and augments each filter with appropriate weights. However, most Retinex-based methods are not accurate enough to estimate the light component, making the results prone to problems such as noise and color distortion. In addition, according to the dehazing method of foggy images, some researchers [12] invert low-light images to obtain similar foggy images, so that the low-light images and foggy images are correlated. Use image dehazing methods to solve low light problems. In recent years, the continuous improvement of deep learning [13, 14] has aroused the thinking of a large number of scholars, which have brought great development to image enhancement techniques. Lore et al. [15] proposed LLNet to solve the low-light image enhancement problem from a representation learning perspective using a deep autoencoder. Yang et al. [16] proposed a deep recurrent bandwidth network that extracts a series of coarse-to-fine bandwidth decomposition representations and reorganizations. Remove noise and restore details through end-to-end learning, using adversarial learning to improve perceptual quality. Wang et al. [17] considered low-light enhancement as a residual learning problem and proposed a new Lightening Back-Projection (LBP) block to generate high-quality enhanced images in an incremental manner. Although many existing methods can improve the brightness of low-light images, there are still space for improvement. For example, some methods cause color distortion after increasing the brightness, as shown in Figure 1b. The LIME method [18] considers both illumination mapping estimation and denoising to estimate the illumination image by solving a mathematical model. The brightness in the image is obviously improved, but the overall color of the image is distorted, and some areas are over-enhanced. There are also methods that do not suppress noise when enhancing brightness, as shown in Figure 1c. The BIMEF method [19] is a double exposure fusion algorithm that fuses the input image with the synthetic image according to the image fusion weight matrix. It can be seen that there is a lot of noise in Figure 1c, and there are more dark areas. In addition, some methods can cause the enhanced and denoised images to be too smooth in order to eliminate noise. In Figure 1d, the KinD method [20] decomposes the low-light image into an illuminance map and a reflection map, and uses different network structures to enhance the decomposed images, respectively. It can be seen that the edges are blurred and texture and detail information are lost in Figure 1d. Therefore, low-light image enhancement methods need to be further improved. FIGURE 1Open in figure viewerPowerPoint (a) Original low-light image; (b)–(e) enhanced images using LIME, BIMEF, KinD and our method, respectively In this work, we propose an attention-based multi-channel feature fusion enhancement network (M-FFENet) to process low-light images. Since the essence of the low-light phenomenon is the degradation of the brightness, contrast and color of the image, we first use a feature extraction model to obtain deep features and fit them to an affine bilateral grid. Then, an attention-based residual dense block is used to deepen the network, allowing the network to focus on more channel and spatial information. Next, we consider different color channels, using a feature reconstruction model to bilaterally interpolate each color channel feature with an affine grid. Finally, we fuse different feature maps and process the fused features in detail. As can be seen from Figure 1e, our method can effectively improve the image quality and generate clearer images. The contributions of our method are mainly reflected in the following aspects: (1) We propose a new end-to-end network for low-light enhancement. We decompose the low-light problem into three sub-tasks to gradually achieve image enhancement effects, which can not only better learn image features, but also give full play to the role of each model. Also, our method is compared with other methods on some challenging datasets. Experiments show that our method makes the enhanced images brighter and more natural, with better overall visual effects. (2) An attention-based residual dense block (ARDB) is proposed, which consists of a residual dense block (RDB) and a convolutional block attention module (CBAM). It can make full use of each of these layers, improving the flexibility of processing different feature information. In addition, the introduction of CBAM can help the network model to extract feature information of images from both the channel dimension and the spatial dimension, further improving the network's ability to obtain local information. (3) A feature extraction model is introduced to further extract detailed features in the image and improve the enhancement ability of the network. Meanwhile, the feature reconstruction model (FRM) is proposed to interpolate the features in the bilateral grid with different color channels, so that the network can obtain high-quality features. (4) We use the feature fusion module to fuse the feature maps with different channel information, so that the fused feature information has a higher degree of discrimination and improves the expressive ability of the feature map. 2 RELATED WORK Low-light image enhancement techniques have evolved in recent years, and researchers have proposed various methods to improve the overall quality of low-light images. According to the principle of enhancement method and the way of processing data, we briefly introduce several common low-light image enhancement algorithms, including low-light image enhancement algorithms based on histogram equalization, low-light image enhancement algorithms based on Retinex theory, and low-light image enhancement algorithms based on deep learning. In addition, we introduce the principles of the attention mechanism. 2.1 Low-light enhancement method based on histogram equalization The algorithm of histogram equalization (HE) makes the originally unevenly distributed image gray range redistributed by non-linear stretching of the input image using the transform function to obtain an enhanced image with the histogram uniformly distributed in the gray range. Low grayscale intervals in low-light images are stretched to high-grayscale intervals, thus adjusts the overall brightness of the image and enhances the contrast of the image. As HE is used to achieve enhancement by adjusting the global extent of the image, local areas of the image are easily ignored. Reza et al. [21] proposed the Constrained Contrast Adaptive Histogram Equalization algorithm to improve the block effect of images. For the enhancement of images with uneven illumination, Tan et al. [22] proposed a multi-histogram equalization method based on exposure regions. The method uses a sub-histogram based on exposure area thresholds and an entropy-based gray level assignment scheme to assign new output gray level ranges. Although these methods have been proven to be effective in enhancing the brightness and contrast of images, in the process of HE processing images, similar gray levels of images can merge with each other, resulting in the absence of some gray levels, causing the loss of image details, and problems such as poor enhancement or over-enhancement may occur on some areas of the image. 2.2 Low-light enhancement method based on retinex theory Retinex theory defines that an image is composed of physical reflection components and ambient lighting components. The algorithm based on Retinex theory believes that the final visual effect is not good due to the serious distortion of the illumination component in the process of image enhancement. Thus, the detail information of the image is extracted by reasonably estimating the illumination component and reducing its influence on the image enhancement process, while improving the reflection component. Then the optimized estimated illumination component and reflection component can be recombined to achieve the purpose of image enhancement. Jia et al. [23] proposed an extended variational image decomposition model that decomposed it into luminance, reflection, and color layers by solving the total variational minimization problem, and by adjusting these three layers. The model can apply high dynamic range tone mapping and contrast enhancement, among others. Hao et al. [24] used an efficient semi-decoupled method to decompose Retinex images for enhancement effects. In the Gaussian Total Variation filter, the illumination layer is estimated using the input image, and the reflection layer is estimated using the input and intermediate images. Liu et al. [25] proposed a new network framework based on Retinex theory. Combining principled optimization deployment techniques with cooperative prior architecture search strategies results in efficient and lightweight low-light enhanced networks. In conclusion, the enhancement algorithm based on Retinex theory can effectively improve the brightness and contrast of low-light images, and reduce the influence of noise to a certain extent. It makes the enhanced image effect more consistent with human visual characteristics and feels more realistic. However, the relevant parameters of the algorithm need to be manually set according to prior knowledge, and the adaptability to different types of images is poor. When testing in large-scale data, some of the enhanced result images tend to have low contrast, color distortion, and the appearance of light dizziness, etc. 2.3 Low-light enhancement method based on deep learning Since the rise of deep learning, it has been widely used in image classification [26-28], visual tracking [29, 30], and semantic segmentation [31-33], and has made breakthroughs in these fields. Convolutional neural network (CNN) [34] has become a representative model in deep learning because of its unique advantages and good robustness. Therefore, researchers have proposed many effective low-light image enhancement methods based on this network. In supervised learning-based augmentation methods, Wei et al. [35] constructed a paired dataset (LOL) containing both low and normal light conditions, captured in a real environment. And an enhancement network (RetinexNet) based on Retinex image decomposition is proposed by using this dataset according to Retinex theory. The network consists of Decom-Net, which performs image decomposition, and Enhance-Net, which enhances the light map, and is an end-to-end learning framework. Lv et al. [36] proposed the MBLLEN network, which includes a feature extraction module, an enhancement module, and a fusion module, and uses a combination of SSIM loss, VGG loss, and region loss to generate more robust results. Zhang et al. designed KinDNet [20], which adopts a staged model. First decompose the original low-light image, then restore the reflection map obtained after decomposition, and finally adjust the light map. In addition, the network is trained with gamma-corrected simulated data. Zamir S W et al [37] proposed MIRNet for low illumination enhancement tasks, which discards the downsampling operation between each submodule to obtain an accurate high-resolution representation of the location information, while multi-scale feature extraction and cross-scale fusion are used in each submodule to obtain rich contextual information. Although the existing network model makes the enhanced images visually better, it is necessary to prepare paired datasets and to adjust the loss function at any time during the training process to ensure that the expected enhancement effect is achieved. Among the enhancement methods based on unsupervised learning, Guo et al. [38] combined traditional methods with deep learning and proposed the Zero-DCE method. This method converts the image enhancement task to using a neural network to output the image as a curve that enhances the image by its own continuous iteration. Jiang et al. [39] proposed a novel method for processing low-light images inspired by the Retinex model. The network model consists of an image decomposition stage and a correction stage, which generates high-quality enhanced images using only low-light images and corresponding histogram equalized (HE) counterparts as input. In spite of using traditional convolutional neural networks (CNNs), many scholars have applied generative adversarial networks (GANs) to network models for the purpose of low-light enhancement. Chen et al. [40] adopted an improved U-Net as a generator, which could capture global features and facilitate adaptive enhancement of local regions. Jiang et al. [41] proposed an efficient unsupervised generative adversarial network (EnlightenGAN) inspired by unsupervised image-to-image transformation and incorporated self-regularized perceptual loss in the network. Although some unsupervised network models eliminate the need for paired datasets and can recover the color and brightness of images better, the enhanced results still suffer from problems such as missing detail information and noise. 2.4 Attention mechanism The main function of the attention mechanism is to let the system to focus on useful information and ignore some unimportant information. It can significantly improve the performance of systems processing data and plays an increasingly important role in artificial intelligence fields such as semantic segmentation, target recognition and image processing [42]. The realization principle of attention mechanism in the field of image processing is to use the method of quadratic weighting to highlight the key information in the image. Then make the deep model continuously learn the parts of the image that need attention during the training process, thus forming the attention. Thus, the basic principle of the attention mechanism is that a set of weight distributions acting on the original image can be obtained by learning. Based on the above principles, attention mechanisms can be divided into strong attention and soft attention. Among them, strong attention focuses on dynamic changes and is a random prediction process. It pays attention to each point in the image, and it randomly selects data when selecting a part of the image as input. Strong attention is not microscopic and usually requires the addition of augmented learning to complete the training. Unlike strong attention, soft attention can compute gradients, which are obtained by deep network training, and thus it is microscopic. The weight values after sotfmax processing are continuously distributed between [0,1], and the level of attention paid to each region is represented by a value between 0 and 1. Among them, soft attention mainly includes: spatial attention, channel attention and mixed attention. Spatial attention: Spatial attention mainly focuses on image features at the spatial level, and usually compresses the number of channels in the image, so that the model can focus on more spatial information. Channel attention: Channel attention mainly focuses on the features of the channel dimension, uses the feature factor to represent the correlation between the channel and the features extracted from the channel, and assigns more weights to the channels with greater correlation. 3 OUR APPROACH As shown in Figure 2, we design an attention-based multi-channel feature fusion enhancement network (M-FFENet) to process low-light images. It can be seen that we first downsample the original low-light image to generate a low-resolution image. The obtained image is then passed through a feature extraction model to better extract the deep feature information of the image, and the extracted features are fitted into a bilateral grid to obtain the predicted bilateral coefficients. Second, we pass the original low-light image through an attention-based residual dense block in order to obtain the spatial and detail features of the image. In addition, to obtain richer texture edge and color information, we consider all color channels(RGB) of the image. Immediately after, the features output in each channel are linearly interpolated with the bilateral grid by our proposed feature reconstruction module to generate feature maps containing rich information, and these high-resolution feature maps are concatenated by channel. Finally, to further enhance the detail information of the image, we pass the connected feature maps through the feature fusion module and the enhancement model to obtain the final enhanced image. FIGURE 2Open in figure viewerPowerPoint The overall framework of M-FFENet. Our proposed network structure consists of three parts. First, the original image is passed through a downsampling and feature extraction model to obtain the deep features of the image, and the obtained features are fitted into an affine bilateral grid. The original image is then passed through an attention-based residual dense block to obtain the detailed information of the image. Then use the feature reconstruction model to interpolate the generated features of different color channels and bilateral grids to obtain high-resolution features containing different information. Finally, the feature map is passed through the feature fusion module and the enhancement model to get the final output The algorithm flow chart of our proposed method is shown in Algorithm 1. ALGORITHM 1. Attention-based multi-channel feature fusion enhancement network Input: Low-light image L Output: Low-light enhanced image E 1: // extract advanced features by ARDB model 2: X = A R D B ( L ) $X = ARDB(L)$ 3: // consider all color channels 4: for i i n $in$ { 1 , 2 , 3 } ${\rm {\lbrace 1,2,3\rbrace }}$ do 5: X n e w i = g u i d e _ c o l o r ( X i ) ${X_{new}}^i = guide\_color({X^i})$ 6: T i = R B ( X n e w i ) ${T^i} = RB({X_{new}}^i)$ 7: end for 8: // downsample the original low-light image and use the feature extraction model to obtain rich features 9: L n e w = i n t e r p o l a t e ( L ) ${L_{new}} = interpolate(L)$ 10: F = F e a t u r e _ e x t r a c t ( L n e w ) $F = Feature\_extract({L_{new}})$ 11: // predict bilateral grid coefficients 12: c o e f f = A f f i n e _ g r i d ( F ) $coeff = Affine\_grid(F)$ 13: // use FRM to recover high quality features 14: for i i n $in$ { 1 , 2 , 3 } ${\rm {\lbrace 1,2,3\rbrace }}$ do 15: Z i = s l i c e ( c o e f f , T i ) ${Z^i} = slice(coeff,{T^i})$ 16: F n e w i = F R M ( Z i , L ) ${F_{new}}^i = FRM({Z^i},L)$ 17: end for 18: F n e w = C o n c a t ( F n e w 1 , F n e w 2 , F n e w 3 ) ${F_{new}} = Concat(F_{new}^1,F_{new}^2,F_{new}^3)$ 19: // use FFM to filter feature information 20: Y = F F M ( F n e w ) $Y = FFM({F_{new}})$ 21: Y n e w = m u l t i p l y ( Y , L ) ${Y_{new}} = multiply(Y,L)$ 22: // get the final enhanced image by enhancement model 23: E = E M ( Y n e w ) $E = EM({Y_{new}})$ 3.1 Attention-based residual dense block To enhance the depth and width of the network model, while allowing the network to focus on the key information of the feature map, we propose an attention-based residual dense block (ARDB). ARDB not only makes full use of the features output from each layer in the network, but also increases the flexibility of handling different features. The lower left area of Figure 2 shows the detailed structure of the ARDB. As seen in the figure, the ARDB contains the residual dense block (RDB) and the convolutional block attention module (CBAM). In which, the residual dense block (RDB) consists of a densely connected layer and a local feature fusion containing local residual learning. In the dense residual block, convolutional layers are used to activate and map deeper level features. The latter convolutional layer can be connected to all the previous layers by dense concatenation, so that all the features of these longitudinally concatenated convolutional layers can be fully utilized. Then all the feature information is fused together by local feature fusion. Finally, the original input is combined with the fused features to generate feature maps containing rich information. In addition, we consider that the features of different channels have completely different weighting information, and the illumination brightness distribution in low-light images is not uniform, we add the CBAM module after the residual dense block. It can make the network model highlight the key information of the image, which can not only enhance the underexposed areas of the image, but also avoid over-enhancement of the normal light area. The specific details are introduced in section 3.1.1. 3.1.1 CBAM module structure In enhancing low-light images, both channel information and spatial information of feature images are important, so we introduce the convolutional block attention module (CBAM) in the network. The CBAM module enables the network to process feature maps in both channel and spatial dimensions, allowing the network to focus on more effective information. Moreover, this module can precisely locate the dark area of the image for enhancement, restore the hidden details, suppress the generation of noise, and improve the enhancement effect. The detailed structure of the CBAM module is shown in Figure 3. It can be seen that in the channel attention model (CAM), the feature map F with original dimension H × W × C $H \times W \times C$ first goes through the average pooling layer and the maximum pooling layer to aggregate the spatial information of the feature map and generate two different spatial contextual descriptors: F a v g c $F_{avg}^c$ and F m a x c $F_{max}^c$ . These two feature descriptors are then forward propagated into a shared network, which consists of a multilayer perceptron ( M L P $MLP$ ) with hidden layers. Next, the two feature vectors output by M L P $MLP$ are subjected to a pixel-by-pixel summation operation, and then pass through the sigmoid activation function to generate the channel attention feature map M c ( F ) ${M_c}(F )$ . Finally, the input feature map F is multiplied by the channel attention feature map to generate a new feature map F1: F a v g c = A v g P o o l ( F ) , \begin{equation} \begin{split} F_{avg}^c = AvgPool(F), \end{split} \end{equation} (1) F max c = M a x P o o l ( F ) , \begin{equation} \begin{split} F_{\max }^c = MaxPool(F), \end{split} \end{equation} (2) M c F = σ M L P F a v g c + M L P F max c = σ W c ′ ′ W c ′ F a v g c + W c ′ ′ W c ′ F max c , \begin{equation} \begin{split} {M_c}{\left(F \right)} &= \sigma {\left({MLP{\left({F_{avg}^c} \right)} + MLP{\left({F_{\max }^c} \right)}} \right)}\\ &= \sigma {\left({{W^{\prime \prime }_c}{\left({{W_c}^\prime {\left({F_{avg}^c} \right)}} \right)} + {W^{\prime \prime }_c}{\left({{W_c}^\prime {\left({F_{\max }^c} \right)}} \right)}} \right)}, \end{split} \end{equation} (3) F 1 = M c F ⊗ F , \begin{equation} \begin{split} {F_1} = {M_c}{\left(F \right)} \otimes F, \end{split