SUGISAKAMASANORI

Journal of Advances in Artificial Life Robotics
Vol. 1(4); March (2021), pp. 163–167
ON LINE ISSN 2435-8061; ISSN-L 2435-8061
https://alife-robotics.org/jallr.html

Research Article

Image Super-Resolution Reconstruction Technology Based on
Deep Learning
Lingran An1, Fengzhi Dai2, Linghe An3

4th Department of Basic, North China Computing Technology Institute, Beijing, 100083, China
College of Electronic Information and Automation, Tianjin University of Science and Technology, China
3Changchun University of Science and Technology, China
12ARTICLE INFO
Article History
Received 25 October 2019
Accepted 17 August 2020
Keywords
Super-resolution
Deep learning
Neural network
Generative adversarial network

ABSTRACTThe traditional super-resolution method has limited ability of feature extraction and featureexpression, which cannot meet the requirements of high quality image in practical application.
This paper mainly applies the relevant theories of deep learning to image super-resolution
reconstruction technology. By comparing three classical network models used for image superresolution (SR), finally a generative adversarial network (GAN) is selected to implement image
super-resolution, which is called SRGAN. SRGAN consists of a generator and a discriminatorthat uses both perceived loss and counter loss to enhance the realism of the output image in detail.
Compared with other algorithms, although the improvement of PSNR and SSIM values of theSGRAN network obtained by the final training is not obvious, the output high-resolution images
are the best in the subjective feelings of human eyes, and the reconstruction effect in the image
details is far higher than that of other networks.

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd.
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

1. Introduction

Although the traditional super-resolution technology
can obtain high-resolution images to a certain extent, thereconstruction effect is not ideal, and there is still a greatroom for improvement. With the rise of deep learning,
more and more visual fields are trying to apply it. In 2014,
Dong et al. used convolutional neural network to achieve
image super-resolution reconstruction (SRCNN) [1].
Subsequently, Dong et al. proposed FSRCNN on the basisof SRCNN [2]. This model is the acceleration of SRCNN,
achieving a breakthrough in both the speed and quality ofreconstruction. Kim et al. combined the idea of residual
network and the structure of convolutional neural network
to propose the VDSR algorithm, which can deepen thenetwork and shorten the training time at the same time [3].
Later, Kim et al. proposed the DRCN algorithm to apply
the recursive neural network structure to the superresolution processing [4]. Shi et al. proposed a real-timesuper-resolution reconstruction method based on

Image resolution describes the number of pixels in animage, which is also a measure of the amount ofinformation in the image. It is an important indicator ofimage detail information presentation ability. In a large
number of electronic image applications, high-resolution
images are often expected. However, in practice,
constrained by many factors, we usually cannot directly
obtain the ideal high-resolution image with edge
sharpening and non-block blur.
The most direct measure to improve the image
resolution is to improve the optical hardware equipment in
the imaging system, but this method is expensive andtechnically difficult Therefore, it is very important toimprove the image resolution from the aspect of softwareand algorithm. In this context, super-resolution
reconstruction (SR) technology emerged.
Corresponding author’s E-mail: daifz@tust.edu.cn, URL: www.tust.edu.cn

163

projection (IBP), projection onto convex set (POCS) andmaximum a posteriori (MAP).
 Learning
The learning-based algorithm can train the images withhigh and low resolution and master the relationshipbetween them, so as to establish the mapping model. It
mainly includes neighborhood embedding (NE), sparse
representation (SR), anchored neighborhood regression(ANR), etc. All these methods belong to the field ofmachine learning.

convolutional neural network and named the model
ESPCN [5]. In 2016, Ledig et al. used the generative
adversarial network (GAN) for the super-resolution
reconstruction problem (SRGAN), and used the perceptual
loss and adversarial loss to improve the authenticity ofrecovered images [6].
2. Image super-resolution reconstruction
2.1. Principle
Image super-resolution technology is to transform lowresolution data into high-resolution data through a certain
method on the basis of unchanged image detection system,so as to obtain image observation of high definition images.
The formation of low resolution image is often caused by
the bad environment, which is often called image
degradation process. It can be expressed as:

L = DBMH + n

3. Super-resolution based on deep learning
3.1. Deep learning theory
In 2006, Hinton et al. first defined the concept of deeplearning [7]. It is clearly expressed in the concept that theartificial neural network model can extract and learn thefeatures of the original data through multi-layer nonlinear
transformation operation, so as to carry out reasonable andeffective prediction of the data information, and make thetraining results constantly approach the expected value ofthe target through layer by layer learning.
With the advancement of research, a variety of powerfuldeep learning models have emerged, among which thetypical ones are convolutional neural network, recurrent
neural network, deep belief network and generative
adversarial network. Among them, the convolutional
neural network was the first to be introduced into thesuper-resolution technology and has achieved greatsuccess. In recent years, the super-resolution method basedon the generative adversarial network structure has madethe reconstructed images improve in terms of details.
The convolutional neural network consists of multiple
layers stacked one on top of the other. It takes the raw data
that's coming in, and it extracts the high-level informationfrom it, and abstracts it, and that's the feedforward
operation of convolution. The error obtained bycomparison is fed back to the front layer continuously until
the error is reduced to the minimum, so that the model
converges to achieve the purpose of training. Its network
structure generally includes input layer, convolution layer,
pooling layer, full connection layer.
The structure of generative adversarial network is
composed of a generator and a discriminator. The
generator is used to synthesize the network data, and thediscriminator is used to judge whether the network data
generated by the experiment is feasible and effective toapproach the real value.

（1）

Where H and L represent high and low resolution
images, M is the matrix after displacement, B is the fuzzy
matrix after degradation, D is the matrix for downsampling, and n is the additional noise pollution.
The process of low-resolution image imaging is aforward process, while the reverse process is the processof image reconstruction. The lost information can berecovered according to the imaging principle to obtainhigh-quality images.
2.2. Classification
The traditional image super-resolution methods can bedivided into three categories: super-resolution technology
based on interpolation， reconstruction and learning.
 Interpolation
The algorithm based on interpolation uses the gray valueof adjacent pixels to generate the gray value of the pixels
to be interpolated so as to realize the super-resolution
reconstruction of the image. The classical interpolation
methods include neighborhood interpolation, bilinear
interpolation and bicubic interpolation.
 Reconstruction
The method based on reconstruction can realize theestimation and reconstruction of high-resolution images by
establishing the imaging model of low-resolution images
and constructing the prior constraint of high-resolution
images. The main research methods include iterative back-164

3.2. Three super-resolution models
In this section, three deep neural network models are
introduced for image super-resolution reconstruction,
namely SRCNN, ESPCN and SRGAN.
SRCNN uses only three layers of network to achieve
image super resolution. It uses mean square error (MSE)as the loss function, which is beneficial to obtain a higher
PSNR. The network structure of SRCNN is shown in Fig.1.
Fig.3. The network structure of SRGAN

3.3. Model selection
Among the above four models, the network model
structure of SRCNN is simple and the information
obtained is very limited. ESPCN can only deal with images
with a small magnification degree, and when the image isenlarged to a greater extent, the resulting image will beblurred. SRGAN can still generate the details in the image
in the case of 4 times or more magnification， which can
be close to the real effect visually. Therefore, we chose
SRGAN as the base network.

Fig.1. The network structure of SRCNNESPCN is an efficient method to extract features
directly from low-resolution image sizes and calculate
high-resolution images. The core concept of the networkis the sub-pixel convolutional layer, which greatly reduces
the computation, saves time and improves the speed of theexperimental process. The activation function is tanhfunction, and the loss function is MSE. The network
structure of ESPCN is shown in Fig.2.

4. Model training and testing4.1. Experimental environment
Experiments in this paper are carried out on Tensorflow,which is an open source deep learning framework, and allalgorithms are implemented by Python. The environment
configuration used in the network model training andtesting experiments is shown in Table 1.

Fig.2. The network structure of ESPCN

Table 1 Experimental environment

SRGAN applies the generative adversarial network tosolve the super-resolution problem. The network is
composed of a generator network and a discriminator
network, which can improve the sense of reality of thegenerated image by both the loss of perception and the lossof resistance. The network structure of SRGAN is shownin Fig.3.

Configuration

Attribute

System

16.04-Ubuntu

CPUIntel Xeon E5-2640 v4

Memory32GB DDR4GPU

NVIDIA TITAN Xp

4.2. The data set
3000 high-quality images are selected from ImageNet androtated by 0°, 90°, 180° and 270°, respectively, and thenflipped vertically. In this way, the training set is expanded
eight times to 24,000 images. The interpolation function isused to get the corresponding low-resolution image data
set by downsampling all the high-resolution images four165

times. Set5, Set14 and BSD100 data sets commonly usedin the field of super-resolution are adopted as test sets.
4.3 Evaluation index of experimental results
In evaluating the effect of image super-resolution, theevaluation criteria mainly lie in the gap in data between thereconstructed image and the expected real image.
Common indicators include Structural Similarity (SSIM)
and Peak Signal to Noise Ratio (PSNR). The value rangeof SSIM is [0,1], and the closer it is to 1, the better. PSNR
is in dB, and the bigger the value, the better.

Fig.4. Comparison of reconstruction results

4.4 Network training parameter setting

5 Conclusion

The adaptive moment estimation (Adam) method isselected as the optimization strategy for network training
to update the learning rate adaptively. During theexperiment, the learning rate is initialized to 0.001. At the
beginning of the training model, the epoch is preset as 300,
and then the optimal number of iterations is selectedaccording to the trend of loss and result curve during thetraining.

In this paper, an image super-resolution reconstruction
network based on GAN model is trained, and the trained
network is tested on the open data set and compared withthree classical super-resolution algorithms. It can be seenfrom the experimental results that although the PSNR andSSIM values of this network are not improved very much,the reconstructed image is closer to the real image.

4.5 Experimental results and analysis

Acknowledgements

We tested the trained SRGAN network model on the Set5,
Set14 and BSD100 test sets, and compared theperformance of SRGAN with Bicubic, SRCNN andESPCN, and summarized the results in Table 2, where thebold part is the maximum value.

Set5
Set14
BSD100

Table 2 The performance comparison
Bicubic
SRCNN
ESPCN
PSNR
29.26
30.07
30.76
SSIM
0.85940.86270.8784
PSNR
26.74
27.18
27.66
SSIM
0.76480.86270.8784
PSNR
26.13
26.68
27.02
SSIM
0.70450.72190.7442

The research is partly supported by the Project of Tianjin
Enterprise Science and Technology Commissioner toTianjin Tianke Intelligent and Manufacture Technology
Co., Ltd (19JCTPJC53700), and the Industry-University
Cooperation and Education Project (201802286009) from
Ministry of Education, China.

SRGAN
30.95
0.883627.78
0.879227.120.7449

As can be seen from the above table, the PSNR andSSIM values of SRGAN are slightly higher than those ofother algorithms. Although the improvement of objective
indicators is not obvious, but the high-resolution images
generated by SRGAN are the most realistic compared withother methods. A detailed comparison of a set of output
images is shown in Fig.4.

166

References
1. Dong C, Loy C C, He K M, et al, Learning a Deep
Convolutional Network for Image Super-Resolution,
European Conference on Computer Vision, 2014: pp 184199.
2. Dong C, Loy C C, Tang X O, et al, Accelerating the SuperResolution Convolutional Neural Network, EuropeanConference on Computer Vision, 2016: pp 391-407.
3. Kim J ， Lee J K ， Lee K M, Accurate Image SuperResolution Using Very Deep Convolutional Networks,IEEE Conference on Computer Vision and Pattern
Recognition, 2016: pp 1646-1654.
4. Kim J, Lee J K, Lee K M, Deeply-Recursive Convolutional
Network for Image Super-Resolution, IEEE Conference on
Computer Vision and Pattern Recognition, 2016: pp 16371645.
5. Shi W Z, Caballero J, Huszar F, et al, Real-Time Single
Image and Video Super-Resolution Using an Efficient SubPixel Convolutional Neural Network, IEEE Conference onComputer Vision and Pattern Recognition, 2016: pp 18741883.

6. Ledig C, Theis L, Huszar F, et al, Photo-Realistic Single
Image Super-Resolution Using a Generative Adversarial
Network, IEEE Conference on Computer Vision andPattern Recognition, 2017: pp:105-114.
7. Hinton G E, Osindero S, Teh Y W, A fast learning algorithm
for deep belief nets, Neural Computation, 2006, 18(7): pp1527-1554.

Mr. Linghe An

Authors Introduction
Ms. Lingran An

She received her Masterof Engineering degreefrom Tianjin UniversityofScienceandTechnology, China in2020. Her main researchareas are deep learning,imageprocessing,artificial intelligence androbotics. During her
study, she has publishedmany conference papers.
She started to work in North China Computing
Technology Institute, Beijing in July 2020, where she iscurrently an assistant software test engineer of the 4th
Department of Basic of the Institute.
Dr. Fengzhi Dai

He received an M.E. andDoctor of Engineering(PhD) from the Beijing
Institute of Technology,
China in 1998 and Oita
University, Japan in 2004
respectively. His main
research interests areartificialintelligence,pattern recognition androbotics. He worked inNational Institute of Technology, Matsue College,
Japan from 2003 to 2009. Since October 2009, he has
been the staff in Tianjin University of Science andTechnology, China, where he is currently an associate
Professor of the College of Electronic Information andAutomation.

167

He is a sophomore in theCollege of Science,
Changchun University ofScience and Technology,
China,majoring
ininformation
andcomputing science. His
major subject areas aremathematics
andcomputer science andtechnology.