The super‐resolution reconstruction of SAR image based on the improved FSRCNN

The Journal of EngineeringVolume 2019, Issue 19 p. 5975-5978 IET International Radar Conference (IRC 2018)Open Access The super-resolution reconstruction of SAR image based on the improved FSRCNN Zhenyu Luo, Corresponding Author Zhenyu Luo 578456788@qq.com Nanjing Institute of Electronic Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorJunpeng Yu, Junpeng Yu China Electronics Technology Group Corporation, Key Laboratory of Intelligent Sensing Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorZhenhua Liu, Zhenhua Liu Nanjing Institute of Electronic Technology, Nanjing, People's Republic of China China Electronics Technology Group Corporation, Key Laboratory of Intelligent Sensing Technology, Nanjing, People's Republic of ChinaSearch for more papers by this author Zhenyu Luo, Corresponding Author Zhenyu Luo 578456788@qq.com Nanjing Institute of Electronic Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorJunpeng Yu, Junpeng Yu China Electronics Technology Group Corporation, Key Laboratory of Intelligent Sensing Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorZhenhua Liu, Zhenhua Liu Nanjing Institute of Electronic Technology, Nanjing, People's Republic of China China Electronics Technology Group Corporation, Key Laboratory of Intelligent Sensing Technology, Nanjing, People's Republic of ChinaSearch for more papers by this author First published: 26 July 2019 https://doi.org/10.1049/joe.2019.0324Citations: 3AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Synthetic aperture radar (SAR) images have become an important way to obtain information in the military and civilian fields, because of its unique advantages. With the development of technology and the need of application, people subjectively put forward a higher demand for image quality. Image resolution is a key factor for evaluating digital image quality and is the basis for subsequent image processing. However, the image quality of SAR images is far worse than that of optical images because of the imaging mechanism and so on. Therefore, it is more difficult to realise super-resolution reconstruction on SAR images. In the image super-resolution reconstruction method, a reconstruction-based method is generally used, but the effect is poor. A method based on deep learning is used to realise the reconstruction of SAR images based on floating-point data by obtaining the mapping relationship between low-resolution images and high-resolution images. At the same time, the SSIM index is introduced into the loss function, so that the reconstructed SAR image is improved both in subjective visual and in objective evaluation indicators. It lays the foundation for subsequent SAR image recognition and other work. 1 Introduction SAR has imaging advantages such as all-weather, all-day, cloud-cloud, and vegetation capability. It has been widely used in military and civilian fields in recent years [1-3]. However, unlike optical images, there are many uncertainties in SAR images, and the target has strong uncertainty and variability. Due to its imaging mechanism, there are a lot of clutter and speckle noise in SAR images. [4] In order to analyse the SAR image effectively, we have put forward higher requirements on SAR image quality. Therefore, the super-resolution reconstruction of SAR images has become a focus and difficulty in the research field. In the early days, researchers tried to improve the resolution of the image by improving the hardware. The most intuitive method is to improve the image sensor and the photosensitive device. The image super-resolution technology was born in the 1960s and super-resolution reconstruction was realised by signal processing methods. In recent years, more and more attention has been paid to learning-based methods, and the trend of development is obvious. Its main idea is to calculate the corresponding relationship between corresponding low-resolution image blocks and high-resolution image blocks through a large number of training samples to obtain common prior knowledge and establish a mapping relationship model between the two. A large number of research results on deep-learning-based super-resolution reconstruction show that the method based on deep learning can achieve a more abstract description of image data and can achieve good reconstruction in terms of quality and speed [5]. However, at present, the research on super-resolution reconstruction is mainly based on optical images, and its application in the field of SAR images is still rare. Due to the characteristics of SAR images, the super-resolution reconstruction on SAR images needs to consider the following points: (i) Data format selection. Unlike optical images, SAR image data is generally stored in floating-point data format. Whether selecting floating point data for experiment can be adapted to the network structure and whether it can achieve better reconstruction effects than using jpg(8-bit) or other formats image. (ii) Effect evaluation. Due to the poor imaging quality of SAR images, how to select the index to evaluate the reconstruction effect of SAR images (Fig. 1). Fig. 1Open in figure viewerPowerPoint Optical image and SAR image of the military objective Aiming at the above factors, this paper proposes a super-resolution reconstruction method of SAR image based on improved fast super-resolution convolutional neural network (FSRCNN) network. First, we select floating point data as the experimental basis to verify its applicability in the network structure. Then improve the traditional FSRCNN network and introduce structural similarity index (SSIM) into its loss function. Finally, select appropriate indicators to evaluate the reconstruction result. Through experiments, the reconstruction results have achieved good results in both subjective visual and objective evaluation indicators, laying the foundation for subsequent work on SAR image analysis. 2 Deep learning network 2.1 Convolution neural network Convolution neural network is a typical feedforward artificial neural network; the neurons of the network are tiled to the overlapping area in the image scene. Convolutional neural networks mimic the biological processes and are another form of multi-layer perceptrons [6]. Convolutional neural network is a multi-layer neural network structure, which mainly consists of input layer, volume base layer, down-sampling layer, and full connection layer [7]. Fig. 2 shows the convolutional neural network. In a convolutional neural network, the input layer is used to receive the original image; the convolution layer processes the input image, and can learn features with higher robustness; the down-sampling layer performs the aggregation of spatial or feature types, and reduces the spatial dimension; and the output layer maps the extracted features to the resulting labels. Fig. 2Open in figure viewerPowerPoint Convolutional neural network 2.2 Super-resolution reconstruction network Super-resolution reconstruction network (SRCNN) is a pioneering work for deep learning used in super-resolution reconstruction. It is mainly divided into three steps: (i) extraction and special extraction of image blocks, using the nature of convolutional networks to extract features of image blocks, (ii) non-linear mapping, and (iii) reconstruction [8]. The main purpose of the FSRCNN is to accelerate the previous SRCNN model [9]. The SRCNN structure was redesigned mainly in three aspects: (i) A deconvolution layer was used in the end to map the low-resolution image without difference to the high-resolution image. (ii) Change the input feature dimension again. (iii) A smaller convolution kernel is used but more mapping layers are used [10]. The FSRCNN can be divided into the following five steps: (i) feature extraction, (ii) compression, (iii) mapping, (iv) expand, and (v) deconvolution [11]. The structure of SRCNN and FSRCNN is shown in Fig. 3. Fig. 3Open in figure viewerPowerPoint Structure of SRCNN and FSRCNN 2.3 Evaluation indicators In order to reduce the blindness of the super-resolution reconstruction effectively, we need to develop an evaluation mechanism that can reflect the super-resolution reconstruction quality and algorithm performance. 2.3.1 Peak signal-to-noise ratio Here, we assume that the SAR image has L grey scale levels. Peak signal-to-noise ratio PSNR (dB) is the most widely used indicator of image quality. X is the high-resolution original image, the size is M × N, and Y is the reconstructed high-resolution image with the same size. The higher PSNR means the better the reconstructed image [12]. Compared with the PSNR mentioned above, SSIM is more in line with the human eye's judgement of image quality. Since the human eye generally takes the image structure information as its priority, the basic concept of SSIM is to describe the structural features of the natural image [12]. This makes SSIM a key indicator for evaluating structural similarity. Here, we use the average structural similarity indicator mean structural similarity (MSSIM) to provide closer-aware indicators, at this point the image is divided into w small windows, and MSSIM is defined as Among them, and represent the image information in the w -th window. SSIM is defined as Among them, is the difference of brightness of pixel x and y, is the difference of contrast of pixel x, y, is the difference of structure of pixel x and y. The larger the MSSIM value means the higher the structural similarity between images and the better the reconstruction effect. 2.4 Improved FSRCNN In FSRCNN, the loss function is defined as In this paper, SSIM is introduced into loss function. Since the SSIM range is 0 to 1 and the higher the value means the better the reconstruction effect, we choose the new loss factor as 1 – SSIM. The final new loss function is defined as 3 Experimental data and parameters 3.1 Experimental data set The experimental data are based on the actual ground-based measured SAR data published by the defense advanced research projects agency (DARPA)-supported moving and stationary target acquisition and recognition (MSTAR) program. The research on target recognition for SAR images is basically based on this data set, both in China and internationally. The sensor that collects the data set is a high-resolution spotlight synthetic aperture radar with a resolution of 0.3 m × 0.3 m, working in the X-band, and the polarization used is HH polarization. 3.2 Data format Since digital images can be represented in the form of matrices, two-dimensional arrays are commonly used to store image data in computer digital image processing programs. In dealing with optical images, the image is generally saved as jpg(8-bit), bmp, or in other formats. When we read it, the grey scale range is usually 0–255. However, when we acquire a set of image data and save it in some image format for viewing, the compression process has brought the loss of information, and this loss is even more serious for SAR images. Therefore, we directly select the original floating-point data matrix of the SAR target to experiment (Fig. 4). Fig. 4Open in figure viewerPowerPoint Jpg(8-bit) format image data (left), original floating point data (right) Comparing Fig. 5 with Fig. 6 we can find that when MSTAR-T62 (tank) is read in the floating-point data, its details appear better: Enlarge part of it we can find that in Fig. 5, there is already information loss on the barrel of the tank, and in Fig. 6, the details of the tank show better. Therefore, this paper proposes a super-resolution reconstruction of SAR images method which uses the original floating-point data to train the reconstruction network to achieve a better reconstruction effect. Fig. 5Open in figure viewerPowerPoint Read the MSTAR-T62 target saved in jpg(8-bit) format and its partial enlarged image Fig. 6Open in figure viewerPowerPoint Read the MSTAR-T62 targets saved in floating point data and its magnified images 3.3 Experimental parameters The size of the low-resolution input image is 64 × 64 and the size of the high-resolution image is 128 × 128 (see Table 1). In the network, the learning rate is 0.001, and the batch size is selected as 50. Table 1. Size of the data size the original image 128×128 low-resolution image 64×64 high-resolution input image 128×128 4 Experimental results 4.1 Super-resolution reconstruction results based on different data format We select data samples in different formats and the reconstruction effect is shown in Fig. 7. Fig. 7Open in figure viewerPowerPoint Super-resolution reconstruction results based on different data format Due to the different grey levels of floating-point data and jpg(8-bit) data, it is impossible to use objective indicators to measure the reconstructive effect, but it can be clearly seen from the subjective visual observation that the results of reconstructing with the original floating-point data are better in details. Therefore, the experiments later in this paper are all based on original floating-point data. 4.2 Evaluation of super-resolution reconstruction based on FSRCNN Using SRCNN for super-resolution reconstruction, the results are compared with the reconstructed image using the traditional method in Fig. 8. Fig. 8Open in figure viewerPowerPoint Super-resolution reconstruction results based on traditional method and FSRCNN As can be seen from Fig. 8, the image reconstructed by the other methods is well restored on the edge, but there is a great loss of detail. The reconstruction method based on deep learning performs better in these two aspects. From Table 2, it can be seen that the method based on FSRCNN is also superior to the traditional method on the PSNR and MSSIM values. Table 2. PSNR and MSSIM of the reconstructed image PSNR MSSIM bicubic 58.4950 0.5879 sparse coding method 59.2392 0.6612 SRCNN 59.4280 0.6882 FSRCNN 60.5486 0.7093 the improved FSRCNN 62.4921 0.7209 After other experiments, we found that, overall, the reconstruction-based method has a simple structure and a general effect. The learning-based approach can already begin to learn features and improve on the details of recovery, but the edges are not preserved well. The method based on deep learning can be well reconstructed in both details and edges, making the reconstructed image more valuable. The average performance of these methods is shown in Fig. 9. Fig. 9Open in figure viewerPowerPoint Average performance of different methods 4.3 Evaluation of super-resolution reconstruction based on the improved FSRCNN Using the improved FSRCNN for super-resolution reconstruction, the results are shown in Fig. 9. From the point of the reconstructed image, compared to the image reconstructed based on FSRCNN, the image reconstructed based on the improved FSRCNN is more in line with the original image in detail, and the edge contour is more clear (Fig. 10). Fig. 10Open in figure viewerPowerPoint Super-resolution reconstruction results based on FSRCNN and the improved FSRCNN From the point of evaluation indicators, the improved network improves the PSNR and MSSIM values, and the reconstructed image has better structural similarity with the original image. From the convergence chart, it can be seen that the improved FSRCNN network convergence rate is also in line with expectations (Fig. 11). Fig. 11Open in figure viewerPowerPoint Speed of convergence of the two models 5 Conclusion SAR imaging plays an important role in both military and civil areas. With the development of technology and the increase of demand, SAR image super-resolution reconstruction technology has become the focus of research. However, the traditional method based on reconstruction has a simple structure and poor reconstruction effect. In recent years, learning-based methods have gradually emerged. This paper proposes a SAR image reconstruction method based on convolutional neural network. Taking into account the imaging mechanism and characteristics of the SAR image, the original floating point data and the FSRCNN network are used to reconstruct the processed MSTAR data. On this basis, the SSIM is introduced into the loss function of FSRCNN, so that the learned parameters can better reconstruct the details in high-resolution images. The experimental results show that deep learning combined with floating-point data of SAR image can better reconstruct the SAR image. The improved FSRCNN can also reconstruct the image detail more completely. This paper has only done basic work on the combination of SAR images and super-resolution reconstruction based on deep learning. In the future work study, there would still be some difficulties that need to be solved: (i) How to solve the problem of lack of training samples of SAR images [13]. Whether the concept of migration learning can be introduced into the super-resolution reconstruction of SAR images. (ii) How the evaluation criteria are more clearly defined. Since the floating point data is different from the normal image data format, how to modify the internal parameter values which are suitable for the evaluation criteria of the optical image to apply the floating point data. [14] 6 Acknowledgments This work was supported by China Postdoctoral Science Foundation (2016M591938). 7 References 1Shi W., Caballer O.J., Ledig C. et al.: 'Cardiac image super-resolution with global correspondence using multi-atlas patchmatch'. MICCAI 2013: Proceedings of the 2013 International Conference on Medical Image Computing and Computer-Assisted Intervention, Berlin, 2013, pp. 9 – 16 2Peled S., Yeshurun Y.: 'Super resolution in MRI: application to human white matter fiber tract visualization by diffusion tensor imaging', Magnetic Resonance in Medicine Official J. Soc. Magnetic Resonance in Medicine, 2001, 45, (1), p. 29 3Dong C., Chen C.L., He K. et al.: 'Learning a deep convolutional network for image super-resolution'. ECCV 2014: Proceedings of the 2014 European Conference on Computer Vision, Berlin, 2014, pp. 184 – 199 4Tho Rnton M.W., Atkinson P.M., Holland D.A.: 'Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super – resolution pixel-swapping', Int. J. Remote Sens., 2006, 27, (3), pp. 473 – 491 5Xiuxiu L.: ' Learning-based super-resolution reconstruction algorithm' ( South China University of Technology, Guangzhou, 2013) 6Guntu B.K., Batur A.U., Altunbasak Y. et al.: 'Eigenface-domain super-resolution for face recognition', IEEE Trans. Image Process., 2003, 12, (5), pp. 597 – 606 7Zhang L., Zhang H., Shen H. et al.: 'A super-resolution reconstruction algorithm for surveillance images', Signal Process., 2010, 90, (3), pp. 848 – 859 8Yuan Q., Jin S.X.: 'Improved sequence image super-resolution reconstruction method', J. Comput. Applic., 2009, 29, (12), pp. 3310 – 3313 9Timofte R., Smet V.D., Gool L.V.: 'A+: adjusted anchored neighborhood regression for fast super-resolution'. Proceedings of the 2014 Asian Conference on Computer Vision, Berlin, 2014, pp. 111 – 126 10Dong C., Chen C.L., Tang X.: 'Accelerating the super-resolution convolutional neural network'. ECCV 2016: Proceedings of the 2016 European Conference on Computer Vision, Berlin, 2016, pp. 391 – 407 11Sun J., Shum H.Y.: 'Image super-resolution using gradient profile prior: US9064476. 2015-06-23' 12Simonyan K., Zisserman A.: ' Very deep convolutional networks for large-scale image recognition [EB/OL]. [2017-03-21]'. Available at https://arxiv.org/abs/1409.1556 13Zhuangzhuang T.: 'Research on SAR image target recognition based on convolutional neural network', Chinese J. Radiol., 2016, 5, (3), pp. 320 – 325 14Juanping Z.: 'Classification of noisy labeled SAR images based on probabilistic transfer convolution neural networks', J. Radar, 2017, (6), pp. 1 – 10 Citing Literature Volume2019, Issue19October 2019Pages 5975-5978 FiguresReferencesRelatedInformation