Guided filter‐based multi‐scale super‐resolution reconstruction

CAAI Transactions on Intelligence TechnologyVolume 5, Issue 2 p. 128-140 Research ArticleOpen Access Guided filter-based multi-scale super-resolution reconstruction Xiaomei Feng, Xiaomei Feng School of Electronic and Communications Engineering, Shandong Technology and Business University, Yantai, 264005 People's Republic of China Co-innovation Center of Shandong Colleges and Universities: Future Intelligent Computing, Shandong Technology and Business University, Yantai, 264005 People's Republic of ChinaSearch for more papers by this authorJinjiang Li, Corresponding Author Jinjiang Li lijinjiang@gmail.com orcid.org/0000-0002-2080-8678 School of Electronic and Communications Engineering, Shandong Technology and Business University, Yantai, 264005 People's Republic of China Co-innovation Center of Shandong Colleges and Universities: Future Intelligent Computing, Shandong Technology and Business University, Yantai, 264005 People's Republic of ChinaSearch for more papers by this authorZhen Hua, Zhen Hua School of Electronic and Communications Engineering, Shandong Technology and Business University, Yantai, 264005 People's Republic of China Co-innovation Center of Shandong Colleges and Universities: Future Intelligent Computing, Shandong Technology and Business University, Yantai, 264005 People's Republic of ChinaSearch for more papers by this author Xiaomei Feng, Xiaomei Feng School of Electronic and Communications Engineering, Shandong Technology and Business University, Yantai, 264005 People's Republic of China Co-innovation Center of Shandong Colleges and Universities: Future Intelligent Computing, Shandong Technology and Business University, Yantai, 264005 People's Republic of ChinaSearch for more papers by this authorJinjiang Li, Corresponding Author Jinjiang Li lijinjiang@gmail.com orcid.org/0000-0002-2080-8678 School of Electronic and Communications Engineering, Shandong Technology and Business University, Yantai, 264005 People's Republic of China Co-innovation Center of Shandong Colleges and Universities: Future Intelligent Computing, Shandong Technology and Business University, Yantai, 264005 People's Republic of ChinaSearch for more papers by this authorZhen Hua, Zhen Hua School of Electronic and Communications Engineering, Shandong Technology and Business University, Yantai, 264005 People's Republic of China Co-innovation Center of Shandong Colleges and Universities: Future Intelligent Computing, Shandong Technology and Business University, Yantai, 264005 People's Republic of ChinaSearch for more papers by this author First published: 11 May 2020 https://doi.org/10.1049/trit.2019.0065Citations: 8AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The learning-based super-resolution reconstruction method inputs a low-resolution image into a network, and learns a non-linear mapping relationship between low-resolution and high-resolution through the network. In this study, the multi-scale super-resolution reconstruction network is used to fuse the effective features of different scale images, and the non-linear mapping between low resolution and high resolution is studied from coarse to fine to realise the end-to-end super-resolution reconstruction task. The loss of some features of the low-resolution image will negatively affect the quality of the reconstructed image. To solve the problem of incomplete image features in low-resolution, this study adopts the multi-scale super-resolution reconstruction method based on guided image filtering. The high-resolution image reconstructed by the multi-scale super-resolution network and the real high-resolution image are merged by the guide image filter to generate a new image, and the newly generated image is used for secondary training of the multi-scale super-resolution reconstruction network. The newly generated image effectively compensates for the details and texture information lost in the low-resolution image, thereby improving the effect of the super-resolution reconstructed image.Compared with the existing super-resolution reconstruction scheme, the accuracy and speed of super-resolution reconstruction are improved. 1 Introduction Single image super resolution (SISR) is a reconstruction technique that recovers high-resolution (HR) images from low-resolution (LR) images [[1]]. Recovery of HR from LR is an illness problem due to the loss of important information in LR [[2]]. However, since super-resolution reconstruction can be used as a monitoring facility [[3]], medical imaging [[4]] and other built-in modules [[5], [6]] for performing image restoration and recognition tasks. Therefore, super-resolution reconstruction has a wide range of applications and promotes the development of super-resolution reconstruction. The SISR method is mainly divided into three categories: based on learning [[7]]; based on interpolation [[8]]; and based on reconstruction. To improve the accuracy of super-resolution reconstruction, a reference-based super-resolution (RefSR) reconstruction method has been proposed in recent years. The RefSR method utilises HR detail in the reference image to achieve super-resolution reconstruction. The RefSR method is more competitive than the SISR method, but the key issue of the RefSR method is how to solve the problem of transmitting the high frequency detail of the reference image to the LR image. With the great success of deep learning in computer vision [[9], [10]], the learning-based super-resolution reconstruction method has also become a research hotspot. For example, such as Random Forest [[11]] and Convolutional Neural Network (CNN) [[12]] were first applied to super-resolution reconstruction, which opened a new era of super-resolution reconstruction. Subsequent SR methods included SRCNN [[13]], which used a three-layer CNN to perform super-resolution reconstruction of LR images. VDSR [[14]] used the residual network for the first time to solve the SR problem. ESPCN [[15]] made the up-sampling data more accurate and reduced noise interference. The generative adversarial network SRGAN [[16]] introduced the perceptual loss and improved the visual effect after super-resolution reconstruction. The dense residual block EDSR [[17]] with skip connections was used to fully extract the features of different layers. The wide residual WDSR [[18]], which considered that the effective features were mainly concentrated in low-level features, and the wide residual network was used for effective learning. Later DBPN [[19]] used up and down iterative sampling to learn features more effectively. The learning-based super-resolution reconstruction method is to input the LR image into the network and learn the non-linear mapping relationship between LR and HR through convolution. Because the input LR image part features are lost, the mapping relationship learned by the convolutional network are also different, super-resolution reconstruction effects are uneven. In recent years, with the emergence of perceptual loss [[20]] and adversarial loss [[21]], large-scale factor reconstruction has greatly improved visual effects [[16], [22]], but reconstructed images often exhibit artefacts. To further improve the accuracy of super-resolution reconstruction, the RefSR method introduced additional images to aid in super-resolution reconstruction tasks, but required careful selection of reference images. The reference image should have textures and content similar to the LR image, similar to the two initial images of image style migration [[23], [24]], which can be obtained from adjacent frames in the video [[25]], or from different visual images [[26]] or obtained from an external data set [[27]]. The reference images selected by these methods are all based on the reference of the LR image itself, and the LR image needs to be strictly aligned with the reference image [[28]], so the robustness is low. In this paper, we propose a multi-scale super-resolution reconstruction method based on a guided image filter. Like the traditional SISR, we use a multi-scale method to build a network and fully exploit the non-linear mapping between LR and HR. The difference with the traditional SISR is that we draw on the idea of RefSR. Real HR image features are applied to the super-resolution reconstruction network by using guided image filter. The algorithm is divided into two stages. In the first stage, the LR image is transmitted through the network to perform different scales of up-sampling for SR tasks. In the second stage, the HR image output from the first stage and the real HR image are subjected to a guide image filtering process to obtain a new image. The new image and real HR image are used to jointly train the network to achieve super-resolution reconstruction. In the first stage, in a multi-scale super-resolution reconstruction network, images of different scales at each level play different roles, and feature extraction is performed using a code–decoded structure with residual blocks [[29]], using the improved long short-term memory (LSTM) [[30]] processing inputs of different scales; Super-resolution reconstructed image distortion due to loss of LR features of the first stage input. In this paper, the method of guiding image filter is used to effectively supplement the details of LR image loss and texture information, and then train through the first-stage network to complete the super-resolution reconstruction task. 2 Related work 2.1 Super-resolution reconstruction 2.1.1 SISR reconstruction With the development of deep learning, the deep learning-based reconstruction method has shown superior performance in both qualitative and quantitative analysis of images [[31], [32]]. The SRNN proposed by Timofte et al. [[13]] introduced CNN into the SR field for the first time. SRNN only used a three-layer network to extract features and used mean square error (MSE) as the loss function. The experimental results are good, which proves the effectiveness of deep learning. As the depth of the network increases, the results of the training will become more and more accurate, but the deep network also brings difficulties to the training. The problem of gradient disappearance or gradient explosion hinders the design of deeper networks. In 2016, Kim et al. [[14]] proposed a network ResNet that can be connected by jumping, and its network depth can reach 152 layers. Effectively solve the problem of gradient disappearance or explosion, and make the network develop to a deeper level. The combination of deep networks and residual blocks has emerged, such as EDSR [[17]], WDSR [[18]], DBPN [[19]]. EDSR super-resolution reconstruction is better, but the number of network layers is deeper and the number of parameters is larger, the time used for super-resolution reconstruction is longer. Compared with EDSR, WDSR adopts weight standardisation and removes many redundant convolutional layers, which has improved structure and performance. DBPN is different from the previous method. DBPN uses the projection unit to perform up-and-down iterative sampling. The extraction features are more comprehensive, and the super-resolution reconstruction results are better, but the network complexity is higher. From the sampling method, SISR can be divided into four categories: predefined up-sampling, single up-sampling, progressive up-sampling, and iterative ups and downs. The predefined upsampling is to learn the non-linear mapping between LR and HR. Before inputting the network, the LR image is first interpolated to enlarge the image size to match the target image size, such as Bicubic. However, this method is easy to make noise and affects the quality of reconstruction. To solve this problem, a single up-sampling occurs, the predefined up-sampling interpolation operation is removed, and the LR deconvolution is performed on the last layer, such as FSRCNN [[33]] or ESPCN [[15]], but CNN has insufficient learning ability and poor reconstruction effect. Progressive upsampling uses the Laplacian pyramid network to progressively predict SR images, similar to a single upsampled stack. Enhanced ability to learn complex mappings, fewer parameters, shorter runtimes, and more effective for large-scale factors. The iterative up-and-down sampling DBPN [[19]] has two units of upper projection and lower projection, which implements iterative up-sampling and down-sampling. The network can deeply explore the direct interdependence between LR and HR. The network has higher complexity, but the super-resolution reconstruction is better. From the loss function, the model based on deep learning usually trains the parameters by minimising the MSE between the real image and the network output image, but this does not represent the true visual experience of human beings. However, the perceptual loss can lead to better visual effects, Johnson et al. [[21]] demonstrate the effectiveness of perceptual loss on network training. For example, the perceptual loss is used in generative adversarial networks (GANs) [[16]], and the adversarial loss is introduced, and the perceptual correlation distance between the real value and the network output value is minimised, but the GAN-based perceptual loss method is based on distortion. The cost is to improve the perceived image quality, so the perceptual loss function still has some limitations for the super-resolution reconstruction task. 2.1.2 RefSR reconstruction To increase the quality of super-resolution reconstruction, a method of assisting LR images for super-resolution reconstruction using additional reference images [[34], [35]] has emerged. These methods are called RefSR methods. The method Boominathan et al. [[34]] use DSLR to capture the HR features of the reference image as a reference, and apply a patch synthesis algorithm to reconstruct. Adding a patch registration in [[35]] to improve the algorithm before the nearest neighbour search, and then uses dictionary learning to reconstruct. Decompose the image in [[36]] into frequency subbands and apply patch matching for high-frequency subband reconstruction. Patch-based synthesis algorithms cannot handle the non-rigid deformation of irregularities, thus causing synthetic patches to have artefacts and blurring. Although the sliding window method proposed by Boominathan et al. [[34]] can effectively alleviate this problem, the calculation cost of the method is enormous. Compared with the existing RefSR method, we do not need to consider the corresponding problem between HR image and LR image. We use the guide graph filter to generate a new image as the training image, and use the L2 loss function to train, so that the reconstruction can be achieved real-time requirements. 2.2 Guide image filter The guide image filter is a filter with adaptive weights that can maintain the boundary while smoothing the image. Unlike Gaussian filtering and bilateral filtering, guided filtering is directional, selective for regions and edges, and is an anisotropic filter. Guided filtering uses the input image as a guide map to effectively distinguish the edges and regions of the image, and has edge sensing capability to better protect the edges and details of the image. Guide image filtering is simply defined as follows: (1) where I is the guide image, P is the input image to be filtered, q is the filtered image, W is the weight value determined according to the guide map I, and W can be calculated by the following equation: (2) where is the mean of the pixels in the window, are the values of two adjacent pixels, is the variance of the pixels within the window, and is the penalty value. For (2) analysis, when are located on both sides of the boundary, If the positive and negative signs of the and values are different, the weight value decreases. Instead, the weight value increases. Therefore, the weight value of the alien sign is much smaller than the weight value of the same number, and the pixels in the flat region have a larger weight, and the smoothing effect is better. The pixel weights on both sides of the boundary are less, the smoothing effect is weaker, and the boundary is better. The above theory proves the feasibility of guided filtering, but the running time is slow, and it is difficult to achieve real-time requirements. The fast-guided image filter proposed by He and Jian [[37]] is consistent with the original filtering effect. However, it is about 80% higher than the original running speed and reduces the time complexity from to . The fast guided image filtering is first driven by the local linear model as shown in (3), where I, p, and q are the guidance images, the filtered input image, and the filtered output image, respectively (3) where i is the pixel index, and k is the index of the local square window of radius r, where is (4) and is (5) (4) (5) where and are the mean and variance of I in window k, respectively, and is the regularisation parameter that controls the smoothness. The calculation method of the output image after filtering is as shown in the following equation: (6) where and are the average values of a and b of window centred on i, respectively. 2.3 U-Net network The U-Net network [[38]] is a variant of the fully convulted neural network. The network consists of two parts: the contracted path and the extended path. The image is first input into the network, convolved through two convolution kernels (no padding operation), and adjusted by the non-linear activation function ReLU, downsampled through the pooling layer of . The shrinking path is subjected to five such operations, and each time the pooling layer is subjected to a downsampling, the final downsampling result is obtained. Then, through the extended path, the convolution kernels is used for upsampling, the number of feature channels is halved, the number of channels corresponding to the contraction path portion is fused, and the features of the corresponding phase of the contraction path are spliced to the upsampling stage. The process also has five operations. At the last layer of the network, the convolution kernel is used for dimensionality reduction, and the result is mapped to the expected number. The network structure is shown in Fig. 1. Fig. 1Open in figure viewerPowerPoint U-Net network structure The shrink path is used to extract features and the extended path is used for accurate positioning. The U-Net network uses elastic deformation to enhance the data, thereby reducing the number of training samples, thus achieving end-to-end training for a very small number of samples, and the training results are superior to sliding window convolution [[38]]. Due to the small number of samples, the training speed of the network is also very fast. At first, the U-Net network was applied to medical images with a small number of samples, and later improved, and the processing of the SR problem [[29]] also achieved good results. 3 Proposed method This paper proposes a multi-scale super-resolution reconstruction algorithm based on guide pattern filtering. In the first stage, referring to Fig. 2, the LR image input network is up-sampled according to different proportions to obtain a series of images , , as sub-network input. First, the image is trained using an encoding–decoding network with a residual block to obtain training result , up-sampling and passing it to the next scale , trains again through the code–decoding network, obtains , up-samples and passing it to the next scale , training again through the encoding–decoding network, and finally outputting the super-resolution image ; In the second stage, the super-resolution image obtained in the first stage and the real super-resolution image are subjected to the guide image filtering process, and the obtained image is used as the training image of the first stage to retrain the network for the super-resolution reconstruction task. The method flow chart is shown in Fig. 2. Fig. 2Open in figure viewerPowerPoint Algorithm flowchart 3.1 Multi-scale super-resolution reconstruction network Encoding–decoding networks have a wide range of applications in the field of computer vision. We improved the U-Net network, introduced residual blocks to improve the encoding–decoding network [[39]], and added the number of convolution layers to better extract LR features on different scales. In addition, for the problem of slow network convergence caused by too many convolutional layers, a hidden state loop module [[40]] is used internally to make the network quickly converge. There are re-used modules inside the network. The strategy in [[29]] is adopted, and ConvLSTM [[41]] is included in the last layer of the contraction path, which is used to hide the state and can achieve different scale connections. In a multi-scale super-resolution reconstruction network, the LR image is upsampled into three scales, where the i th scale is half of the (i + 1)th scale. For a code–decoder network, the input image is first convolved and features are extracted, after passing two ResBlocks, followed by a ConvLSTM, and then through two ResBlocks. The network structure is shown in Fig. 3. First, the first convolutional layer maps the image to . After the first residual block processing, the changed image becomes . After the second residual block, the image changes to . Image size and channel number do not change with ConvLSTM. The network moves further forward, and after two transposed layers, the image size changes to . The convolution kernel has a size of , the convolution step of the residual block and the transposed layer is 2, and the other convolution step is 1. ReLU is used as an activation function between convolutional networks. The network structure is shown in Fig. 3. Fig. 3Open in figure viewerPowerPoint Multi-scale super-resolution reconstruction network structure 3.2 Extracting features When the LR image passes through the contraction path of the encoding–decoding network, the decoder adjusts the image size through the corresponding transposed convolution layer [[42]] to generate a large amount of data, and the network needs to capture the effective information in the data. LSTM can establish stable long-term dependence in various studies [[39], [42]]. However, LSTM only controls the time and does not process the space, considering the spatial correlation of the data. Xingjian et al. [[41]] improved FC-LSTM [[43]] and used convolution instead of full connectivity. ConvLSTM estimates the value of a pixel by the input of a pixel's adjacent area and the previous state. After the input LR image is downsampled, ConvLSTM is inserted into the intermediate stage as a natural selection of sequential input. The network structure includes the following: (7) (8) (9) where is the CNN of the encoder, is the parameter, and is the encoder network output. is the convolutional network of the decoder, is the parameter, and is the decoder network input. is the hidden state of the i th step ConvLSTM. is the sum of the intermediate feature maps of the encoder network and is used for jump connections. is the output image at the i th time. is the result of Bicubic sampling of . 3.3 Multi-scale super-resolution reconstruction based on guided filtering The image processing in computer vision and computer graphics mostly involves the concept of image filtering to suppress noise or extract useful image structures. Our commonly used linear translation invariant (LTI) filters, including Gaussian filters and Sobel filters, are widely used in image blurring, edge detection, and feature extraction. The filter kernel of the LTI filter is fixed, so the filtering does not consider the content of the image. In addition, although the LTI filter is effective, it takes a long time to calculate and cannot meet the real-time requirements. Bilateral filtering determines the weight by guiding the intensity of the image or colour similarity, and weights the average of the pixels, so the filter can filter based on the image content considerations. The bilateral filter can smooth the edges, but gradient inversion artefacts may appear near the edges and the calculation complexity is high. Unlike the above two types of filtering, the guide image filter is based on image content filtering. The guide image may be the input image itself or other different images. Guide image filters not only have the characteristics of bilateral filters to keep the edges smooth, but also have no gradient inversion artefacts. The guided image filtering uses a local linear structure, and the calculation is relatively small. For greyscale and colour images, the guided filter has an O (N) time (in pixels N) exact algorithm Fig. 4. Fig. 4Open in figure viewerPowerPoint Some examples showing the difference between guide image filter and ground truth images a As ground truth images, b As guide image filter images, c Is the result of (a) subtracts (b) In the first stage multi-scale super-resolution reconstruction network, we use real super-resolution images to train the network, and the loss function is L2. Such as formula (10). In the first stage, the large-scale super-resolution reconstructed image has blurred edges and artefacts, and the reconstruction effect of the details is not satisfactory. This shows that in the process of convolution, the mapping relationship between LR images and HR images has not been fully and effectively learned. To make the network learn the mapping relationship as fully as possible, we can increase the number of convolutions, or expand the receptive field. In this paper, to make the network learn a more effective mapping relationship, we use a guided image filter. The guided image filter can extract the structure information from the guide image and merge the structure information into the input image to achieve the integration of one information. First, a guided filtering operation is performed on the HR image output in the first stage, and the guided image is a real HR image. We use guided filtering to supplement the details that fail during the super-resolution reconstruction process and improve the performance of the super-resolution reconstruction image (10) where is the proportional weight, usually . is the number of elements. is a true HR image, and is a HR image of the network output. The image optimised by guided filtering cannot be used as the final super-resolution image. We combine the images after guided filtering with real HR images to jointly train the network. Such as formula (11). When the convolutional network learns the mapping relationship between LR and real HR images, the mapping relationship that may be learned is not comprehensive. However, we also have guided filtered images, using the same CNN, to learn the mapping relationship between LR and guided filtered images again, making the learned mapping relationship more comprehensive. Real HR images help super-resolution images to enhance detail, while guided filtered images ensure that the edges of super-resolution reconstruction are clear, achieve noise reduction, and reduce artefacts, making the network have higher performance (11) where represents the filtered image, is the real HR image, is the high- resolution image output by the network, and parameter is used to control the weight of the real image and the guided filtered image. The algorithm in this paper is shown in Algorithm 1. Algorithm 1.Multi-scale super-resolution reconstruction based on guided image filterRequire: LR image, Real high-resolution i Input: Low-resolution Image ii Output: High-resolution Image iii Initialise Downsample the Low-resolution image to get iv while do v Train the input low-resolution image. According to the (10): vi Network output to be improved high-resolution image vii end while viii Filter image obtained by (6): ix while do x Joint training with filtered images and real HR images by (11): xi Output high quality high-resolution image xii end while 4 Experimental analysis The computer used in the experiment was configured as Intel Core i7-6700K CPU@3.40 GHz, NVIDIA TITAN X GPU, 16 GB RAM, Win10 operating system. Use TensorFlow to build a platform framework [[44]]. The data set uses DIV2K [[45]]. Since the image size in the DIV2K is large, the image is first reduced. The DIV2K consists of 800 training images and 100 test images and 100 proof images. Since the 800 training images did not publish real images, our test was selected in 100 verified images. The DIV2K validation set has less data. It also uses four benchmark datasets Set5 [[46]], Set14 [[47]], B100 [[48]], and Urban100 [[49]] for performance ratios. In addition, the Manga109 dataset was used to perform super-resolution reconstruction of characters in the comics. The LR image is upsampled using Bicubic to obtain input images of different scales. The variables are initialised using the Xavier method [[50]]. iteration of (8). According to experience [[29]], in (8). 4.1 Model comparison To verify the effectiveness of the proposed method, it is compared with other traditional methods in both qualitative and quantitative aspects. The comparison methods are: bicubic interpolation, an