SUGISAKAMASANORI

Journal of Robotics, Networking and Artificial Life
Vol. 9(4); March (2023), pp. 316–320
ISSN (Online):2352-6386; ISSN (Print):2405-9021
https://alife-robotics.org/jrnal.html

Xiwen Liang1, Xiaoning Yan2, Nenghua Xu2, Xiaoyan Chen1, Hao Feng1
1

Tianjin University of Science and Technology, No. 1038 Dagu Nanlu, Hexi District, Tianjin, China, 300222
Shenzhen softsz co. ltd, Shenzhen, China

2

Received 25 November 2021
Accepted 19 October 2022

Low-light image enhancement
Deep learning
Multi-branch fusion

Low illumination image enhancement is a difficult but scientific task. With the image brightness
increasing, the noises are amplified, and with the contrast and detail increasing, the false
information is generated. To solve this problem, a multi-branch attention network is proposed toprocess low-light images directly without additional operations. The proposed network iscomposed with enhancement module (EM) and Convolutional Block Attention Module (CBAM).
The attention module can make the CNN network structure gradually focus on the weak light areain the image, and the enhancement module can fully highlight the multi-branch feature graphunder the guidance of attention. In this way, the overall quality of the picture will be greatly
improved, including contrast, brightness, etc. Through a large number of experiments, our model
can produce better visual effects, and also achieve good results in quantitative indicators.

Convolutional neural network© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd.
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

can improve both visual perception and indicator
estimation. Our contributions are summarized as follows.
1) We introduce the convolution block attention module(CBAM) into the multi branch enhancement module,
which improve the feature extraction capability of themodel without significantly increasing the amount ofcomputation and parameters. 2) Our method is also
effective in suppressing image noise and artifacts in low
light region.

1. Introduction
Due to unavoidable environments or technical limitations,many photographs are often taken under less than ideal
lighting conditions. Poorly light photos are not only badfor aesthetic quality, but also bad for messaging. The
former affects the audience's experience, while the latter
leads to misinformation being communicated. To solvethese degradations and convert low-quality low lightlevel images into normal light high-quality images, it isnecessary to develop a good enhancement technology forlow light image. In this paper, a deep neural network
structure is proposed to improve the objective andsubjective image quality. Since many existing algorithms
basically brighten the whole picture, including traditionalalgorithms and deep learning algorithms, we consider
introducing CBAM into the neural network to make themodel have more self-attention. Extensive experiments
show that the introduction of this attention mechanism

2. Related Work
Low-light image enhancement techniques havecontinued to evolve in recent decades. Low light is a lowquality image, and enhancing low-light images can not
only improve the beauty of the image, but also help
subsequent high-level visual tasks. Existing neuralnetworks have some powerful tools such as end-to-end
networks and GAN[1] to process low-light images.

Corresponding author’s E-mail: cxywxr@tust.edu.cn, 1594838831@qq.com

316

LLNET[2] is the first real network to apply neuralnetwork to image enhancement. The paper uses deep
self-encoding to extract image features and improveimage brightness. Retinex-Net[3] splits the network into
two modules, and achieves good results by pairing theconstraints of the dataset and using the BM3D denoising
module. In order to integrate the advantages of CNN andGAN, Yang Etal[4] proposed a semi-supervised model toenhance the image in two stages. Lv F Etal[15] proposed
an attention-guided enhancement method andcorresponding multi-branch network structure forsimultaneous image enhancement and noise suppression.
Our model decomposes the complex image enhancement
problem into sub-problem levels related to different
features, which can be solved separately for multi-branchfusion. Channel and spatial attention are embedded in thenetwork, placed before each feature extraction module
EM, which can improve the capability of the model infeature extraction without significantly increasing theamount of computation and parameters. Through thisoperation, brightness/contrast enhancement and image
denoising are also better solved.

layers, retains a wealth of texture information, and uses
multi-scale contextual information to synthesize highquality images. I am using U-Net as the backbone of theEM network. The attention module can make the CNN
network structure gradually focus on the weak light areain the image, and the enhancement module can fully
highlight the multi-branch feature graph under theguidance of attention. The details of the entire framework
are shown in Fig 1.
3.1. Enhancement module (EM)EM contains multiple subnets, the number equals thenumber of branches, and the output color image is thesame size as the input image. Each subnet has asymmetric structure for applying convolution anddeconvolution. Their convolution kernel size is 3*3, thestride is 1, and the number of kernels is 8, 16, 16, 16, 16,8, 3, respectively. the RELU nonlinear activation
function is used to increase the model capacity. After
passing through 9 EM modules, finally concatenate them
together to get the fused features. It should be noted thatall subnets are trained at the same time, but areindependent and do not share any learning parameters.
The enhancement module is considered composing ofencoder and decoder. The first part is used to extract
features in the image, and then skip the connection withthe decoder to achieve information flow at the same scale.
Finally, the information on multiple scales is fused toobtain an enhanced image. The encoder and decoder areskipping connected for detail reconstruction.

3. Methodology
This article proposes a hybrid attention mechanism
framework A-MBLLEN based on multi-branch fusion.
The proposed network is composed with enhancementmodule (EM) and Convolutional Block Attention

3.2. CBAM
Convolutional Block Attention Module (CBAM)[5] has
two Attention submodules, CAM (Channel AttentionModule) and SAM (Spatial Attention Module). CBAM
details are shown below Fig 2.
Fig 1. This is the proposed multi-branch attention network
framework A-MBLLEN. The proposed network withenhancement module (EM) and Convolutional Block Attention
Module (CBAM). After feature fusion, adjust the channel to 3
and output the result

Module (CBAM). The EM input is the output of theCBAM layer. U-Net[16] has been very successful insemantic segmentation, image restoration, and expansion.
U-Net extracts multi-level features from different depth317

reduction, and finally a H*W*1 is output. Finally, thesigmoid activation function is performed to obtain the
final spatial attention map M_s (F). The expression of
M_s (F) is as Equation 2.
𝑀s (𝐹) = 𝜎 (𝑐𝑜𝑛𝑣(𝑐𝑜𝑛𝑐𝑎𝑡(𝐹 𝑠 avg , 𝐹 𝑠 max ))) (2)where 𝑐𝑜𝑛𝑐𝑎𝑡( ) denotes the concatenation operation;𝑐𝑜𝑛𝑣( ) denotes the convolution operation. Finally,𝑀s (𝐹) is multiplied by the input feature map of themodule to obtain the final generated feature map.
By adding two dimensions of attention, it not only
enhances the Structural similarity of the image, but also
suppresses the noise.

Fig 2. CAM(Channel Attention Module) and SAM(SpatialAttention Module). CAM is responsible for the attention
weight on Channel, SAM is responsible for the attention
weight on space (Height, Width).

The CAM framework is shown in Fig 3.

3.3. Loss Function
The function of loss function is to make the enhanced
image𝐸(𝑥,𝑦) of the input image 𝐼(𝑥,𝑦) after the trainableCNN with parameters W enhancement as close as
possible to the input reference image𝑅(𝑥,𝑦) . In order forthe enhancement network to improve the image quality,
we need a sophisticated loss function design to have agood effect in terms of quantity and quality. The MSEloss function to be minimized as:
MSE=‖𝐸(𝑥,𝑦) − 𝑅(𝑥,𝑦) ‖(3)
2
The structural loss function adopts DSSIM which iscomes from Structural Similarity SSIM[6] and can be
expressed as:
DSSIM= (1 − 𝑆𝑆𝐼𝑀(𝑅(𝑥,𝑦) − 𝐸(𝑥,𝑦) ))/2(4)We generally obtain the structural similarity of the pixel
p in the two images by the following simple expression.
2𝑢 𝑢 +𝐶
2𝜎 +𝐶
1
SSIM=− ∑𝑝∈𝑖𝑚𝑔 2 𝑥 𝑦 2 1 ∙ 2 𝑥𝑦 2 2(5)Fig 3. CAM struture

If the input image is H*W*C size, the input is subjected
to global average pooling and global maximum pooling,respectively, to obtain a 1*1*C feature map, and thenpass it into a two-layer perceptron. Channel attenuationis performed in the first layer to C/r, where c is theattenuation ratio. The second layer is still restored to theC channel, and its weights are shared. After two layers ofperceptrons, through an element addition operation, andfinally through the sigmoid activation function, thechannel attention feature𝑀C (𝐹) is obtained.
𝑀C (𝐹) = 𝜎 (𝑊1 (𝑊0 (𝐹 𝐶 avg )) + 𝑊2 (𝑊0 (𝐹 𝐶 𝑚𝑎𝑥 )))𝑁(1)

𝑢𝑥 +𝑢𝑦 +𝐶2

𝜎𝑥 +𝜎𝑦 +𝐶2

where 𝑢𝑥 and 𝑢𝑦 are pixel value averages, 𝜎𝑥 2 and+𝜎𝑦 2are variances, 𝜎𝑥𝑦 is covariance, and C1 and C2 areparameters to maintain the stability of SSIM.
The Context loss can improve the visual quality. Wechoose a VGG network[7] with stable structure to ensurethat the extracted features are more representative of theoriginal image. the context
loss is defined as follows:
𝐶𝐻𝑊
1𝑖，𝑗𝑖，𝑗𝑖，𝑗∑𝑥=1 ∑𝑦=1 ∑𝑧=1 ‖𝜑𝑖,𝑗 (𝐸)𝑥,𝑦,𝑧 − 𝜑𝑖,𝑗 (𝑅)𝑥,𝑦,𝑧 ‖ (6)
𝐿𝑉𝐺𝐺 =𝐶𝐻𝑊

where 𝐹 𝐶 avg is the result of global average pooling,and 𝐹 𝐶 𝑚𝑎𝑥 is the global maximum pooling result.
The SAM framework is shown in Fig 4.

𝑖，𝑗 𝑖，𝑗𝑖，𝑗where E and G represent the enhanced image and thereference image, respectively. 𝑤i,j , 𝐻i,j and 𝐶i,j is thefeature map under different dimensions in VGG[7].
Besides, 𝜑𝑖,𝑗 represents the result output from the firstconvolutions in several blocks in the VGG-19 Network.
The total loss can be expressed as:
𝐿𝑡𝑜𝑡𝑎𝑙 = 𝑀𝑆𝐸 + 𝐷𝑆𝑆𝐼𝑀 + 𝐿𝑉𝐺𝐺
(7)

Fig 4. SAM struture

The output result F of the channel attention module issent to the spatial attention module. After global
maximum pooling and global average pooling, the twofeature maps are connected in the channel direction toobtain the feature map of H*W*2. Finally, a 7*7
convolution kernel is used for channel dimension318

4. Experimental Evaluations

advantages compared with other algorithms, thus proving
the effectiveness of our proposed algorithm.

We are implementing on tensorflow and Keras. Using
pretrained weights we can fit faster. During training, we
use ADAM[8] optimizer and set parameters a =0.002, ε = 10−8 , β1 = 0.9 and β2 = 0.999.

5. Conclusions
As shown in Table 1, we basically have the best resultson all indicators, and one of our indicators, NIQE, isranked last. In general, our model has achieved goodresults from both qualitative and quantitative
perspectives. We applied the model to actual noisy
pictures, and found that A-MBLLEN can model noisy
pictures in real environments, resulting in enhanced
images with good visual perception. In the future, we willdevelop more useful algorithms to solve real-world lowlight image enhancement problems, such as Multi-scalefusion of low illumination image features and somesegmentation and noise image as prior information
enhance the enhancement and generalization ability ofthe model.

4.1. Dataset and Metrcis
It is extremely important for the model to have a training
set that is rich in variety. We pick 16,295 images fromthe VOC[17] dataset and use a scientific approach[9] tosynthesize low-light and noisy paired images. Withreference images in our dataset, we are able to perform
qualitative and quantitative enhancement assessments.
Among the evaluation methods with reference, wechoose SSIM[10] and PSNR, and the evaluation
indicators without reference include brightness average
AB[11] and natural image quality evaluation NIQE[12].
These metrics are good for an objective evaluation of theenhancement effect, and we quantitatively compare themodel(EnlightenGAN[13], MBLLEN[14]) proposed inthis paper with several existing models. The results areshown in Table 1.

References
1. A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B.
Sengupta and A. A. Bharath, "Generative Adversarial
Networks: An Overview," IEEE Signal Processing
Magazine, vol. 35, no. 1, 2018, pp. 53-65.
2. Kin Gwn Lore, Adedotun Akintayo, Soumik Sarkar,
LLNet: A deep autoencoder approach to natural low-lightimage enhancement, Pattern Recognition, 2017,61:pp.650-662.
3. Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu.
Deep
Retinex
Decomposition
forLow-Light
Enhancement. In BMVC, 2018. 2, 5, 6, 7, 8
4. W. Yang, S. Wang, Y. Fang, Y. Wang, and J. Liu, “Fromfidelity to perceptual quality: A semi-supervised approach
for low-light image enhancement,” CVPR, 2020, pp.
3063–3072.
5. Woo, Sanghyun, et al. "Cbam: Convolutional blockattention module." Proceedings of the European
conference on computer vision (ECCV). 2018.
6. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli,“Image quality assessment: from error visibility tostructural similarity,” IEEE transactions on image
processing, vol. 13, no. 4, , 2004, pp. 600–612.
7. Karen Simonyan and Andrew Zisserman. Very deep
convolutional networks for large-scale image recognition.
arXiv preprint arXiv:1409.1556, 2014.
8. Diederik P Kingma and Jimmy Ba. Adam: A method forstochastic optimization. arXiv preprint arXiv:1412.6980,
2014.
9. Lv F, Li Y, Lu F. Attention Guided Low-light Image
Enhancement with a Large Scale Low-light Simulation
Dataset[J]. 2019.
10. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P
Simoncelli. Image quality assessment: from error visibility

Table 1 Comparison of different models.
Models

PSNR↑

SSIM↑

AB↑

NIQE↑

RetinexNet3

23.66
24.07
25.95
26.57

0.747
0.827
0.885
0.894

-4.14
-3.13
0.008
0.010

30.57
27.76
29.47
27.62

EnlightenGAN

MBLLENOURS

Image Structural Similarity (SSIM) as a measure ofSimilarity between two images by calculating the SSIMof the enhanced results of the reference image and lowilluminance image, the performance of the algorithm canbe analyzed. Generally speaking, the larger SSIM valueis, the closer it is to the reference image, that is, theillumination enhanced the better the results. After thealgorithm proposed in this paper enhanced the lowillumination image, SSIM value was improved to 0.894,
which is superior to other methods, indicating that themodel proposed in this paper can improve the structuralsimilarity between the enhanced image and the realimage. Peak Signal-to-noise Ratio (PSNR) is acommonly used index to evaluate image quality. It
reflects the degree of distortion between the image
restored by low illumination image algorithm and thereference image. Generally speaking, The larger thePSNR, the better the picture fidelity, that is, the less theimage is distorted compared to the reference image. As
can be seen from the table, after using the image
enhancement algorithm proposed in this paper, The
PSNR is improved to 26.57, which has obvious
319

11.

12.

13.

14.
15.

16.

17.

to structural similarity. IEEE transactions on image
processing, 2004,13(4) pp:600–612,.
ZhiYu Chen, Besma R Abidi, David L Page, and Mongi AAbidi. Gray-level grouping (glg): an automatic method foroptimized image contrast enhancement-part i: the basicmethod. IEEE transactions on image processing, 2006,
15(8),pp :2290–2302,.
Anish Mittal, Rajiv Soundararajan, and Alan C Bovik.
Making a” completely blind” image quality analyzer.
IEEE Signal Process. 2013, Lett. 20(3) pp:209–212.
Y. Jiang, X. Gong, D. Liu, Y. Cheng, C. Fang, X. Shen, J.
Yang, P. Zhou,and Z. Wang, “EnlightenGAN: Deep lightenhancement without paired supervision,” 2019, arXiv
arXiv:1906.06972.
cF. Lv, F. Lu, J. Wu, and C. Lim, “MBLLEN: Low-lightimage/video enhancement using cnns,” in BMVC, 2018.
Lv F, Li Y, Lu F. Attention Guided Low-Light Image
Enhancement with a Large Scale Low-Light Simulation
Dataset[J]. International Journal of Computer Vision,
2021, 129(11).
Olaf Ronneberger, Philipp Fischer, and Thomas Brox.
Unet: Convolutional networks for biomedical image
segmentation. In International Conference on Medical
image computing and computer-assisted in1tervention,
pages 234–241. Springer, 2015. 4.
Mark Everingham, Luc V an Gool, Christopher KI
Williams, John Winn, and Andrew Zisserman. The pascal
visual object classes (voc) challenge. International journalof computer vision, 88(2):303–338, 2010

Prof. Ms. Xiaoyan Chen
She, professor of Tianjin University of
Science and Technology, graduatedfrom Tianjin University with PH.D(2009), worked as a Post-doctor atTianjin University (2009.5-2015.5). She
had been in RPI, USA with Dr.
Johnathon from Sep.2009 to Feb.2010
and in Kent, UK with Yong Yan fromSep-Dec.2012. She has researched electrical impedance
tomography technology in monitoring lung ventilation formany years. Recently, her research team is focus on thenovel methods through deep learning network models.
Mr. Nenghua Xu
He, general Manager of Shenzhen
Ansoft Technology Co. , Ltd.,
graduatedfrom
South
China
University of Technology, expert in
Huawei's range of engineeringcomputing acceleration and artificialintelligence. He has worked as a senior
engineer in Huawei and a high-level
innovative entrepreneur in Shenzhen.
Mr. Hao Feng
He received his bachelor's degree fromthe school of electronic information andautomation of Tianjin University ofscience and technology in 2020. He is
acquiring for his master's degree at
Tianjin University of science andtechnology.

Authors Introduction
Mr.Xiwen Liang
He received his bachelor's degree fromthe school of electronic information andautomation of Tianjin University ofscience and technology in 2021. He is
acquiring for his master's degree at
Tianjin University of science andtechnology.
Mr. Xiaoning Yan
He graduated from Tianhua South
University of Technology in 2012 witha bachelor of software engineering. In
2015, he received a master's degree incomputer science from the Universityof Texas at Dallas. Since 2017, he hasserved as the technical director ofShenzhen Ansoft Vision Technology
Co., Ltd.

320