Deep residual refining based pseudo‐multi‐frame network for effective single image super‐resolution

IET Image ProcessingVolume 13, Issue 4 p. 591-599 Research ArticleFree Access Deep residual refining based pseudo-multi-frame network for effective single image super-resolution Kangfu Mei, Kangfu Mei School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, People's Republic of ChinaSearch for more papers by this authorAiwen Jiang, Corresponding Author Aiwen Jiang jiangaiwen@jxnu.edu.cn orcid.org/0000-0002-5979-7590 School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, People's Republic of ChinaSearch for more papers by this authorJuncheng Li, Juncheng Li orcid.org/0000-0001-7314-6754 School of Computer Science and Software Engineering, East China Normal University, Shanghai, People's Republic of ChinaSearch for more papers by this authorBo Liu, Bo Liu College of Computer Science and Software Engineering, Auburn University, Auburn, USASearch for more papers by this authorJihua Ye, Jihua Ye School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, People's Republic of ChinaSearch for more papers by this authorMingwen Wang, Mingwen Wang School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, People's Republic of ChinaSearch for more papers by this author Kangfu Mei, Kangfu Mei School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, People's Republic of ChinaSearch for more papers by this authorAiwen Jiang, Corresponding Author Aiwen Jiang jiangaiwen@jxnu.edu.cn orcid.org/0000-0002-5979-7590 School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, People's Republic of ChinaSearch for more papers by this authorJuncheng Li, Juncheng Li orcid.org/0000-0001-7314-6754 School of Computer Science and Software Engineering, East China Normal University, Shanghai, People's Republic of ChinaSearch for more papers by this authorBo Liu, Bo Liu College of Computer Science and Software Engineering, Auburn University, Auburn, USASearch for more papers by this authorJihua Ye, Jihua Ye School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, People's Republic of ChinaSearch for more papers by this authorMingwen Wang, Mingwen Wang School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, People's Republic of ChinaSearch for more papers by this author First published: 05 March 2019 https://doi.org/10.1049/iet-ipr.2018.6057Citations: 2AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Single image super-resolution (SISR) has gained great attraction and progress in recent years. Since the SISR is an ill-posed inverse problem, most researchers are concentrated on making efforts to learn effective and reasonable mapping functions from low-resolution observation to its potential high-resolution (HR) counterpart. In this study, the authors have proposed a deep residual refining based pseudo-multi-frame network for efficient SISR. A channel-wise attention mechanism is employed for residual refinement. It can ease residual learning process through explicitly modelling non-linear dependencies between channels by using global information embedding. Multiple potential HRs from different deconvolutional layers are further artificially learned, and then adaptively fused into final desired HR image. The authors call this strategy as pseudo-multi-frame SR. It could make full use of available redundant information possessed in hierarchical layers. They have evaluated the proposed network on several popular benchmark datasets. The experimental results have shown that the two highlights proposed can consistently boost final performance. The proposed network can outperform most of the state-of-the-art methods with acceptable less parameters. 1 Introduction Super-resolution (SR) is to estimate visual pleasing high-resolution (HR) image/video from its low resolution (LR) observations. It is widely used in many practical applications where high-frequency details are desired, such as video surveillance, medical imaging, remote imaging, HDTV and so on. Since a low-resolution image could be degraded from more than one high-resolution case, the SR is, therefore, a typical ill-posed inverse problem. Researchers on SR in these years were to make efforts on learning effective and reasonable mapping functions from LR to potential HR images. Due to the powerful learning ability of deep neural networks, deep learning (DL) based methods have recently gained much attractions and achieved great progress compared with conventional none-DL methods [1, 2] on SR. In this paper, we considered the construction of SR networks from three main aspects: the computational efficiencies of network, the strategies of residual learning and the information available for fusion. The computational efficiencies of network: SRCNN [3] was the first DL-based work on image SR. It employed a lightweight structure with three convolution layers to learn an end-to-end mapping between LR and HR. FSRCNN [4] was an accelerating version of SRCNN. One of its highlights was that it directly applied network on original LR image, instead of bicubic interpolating original LR before network input. At the end of network, it introduced a deconvolution layer for HR mapping. SRCNN and FSRCNN represented two general strategies on how to deal with input LR images before applying them into network. The pre-processing step of bicubic interpolation inevitably increased the computation burden, over-smoothed and blurred original LR image, which might result important details lost. Similarly, ESPCN [5] introduced an efficient sub-pixel convolution layer to effectively replace the handcrafted bicubic filter. Feature mappings are directly extracted from original LR image. As a result, the computational complexity was reduced. Therefore, from the experience of previous work, applying network directly on original LR before sub-pixel convolutional layer to desired HR is more computational efficient. The strategies of residual learning: VDSR [6] was one of the pioneers that proposed residual learning-based strategies for the single image super-resolution (SISR) problem. Its core idea was based on the observation that LR and HR images shared similar low-frequency information. Therefore, it was reasonable to explicitly modelling the differences, which were also referred to high-frequency details. Residual-based strategies were proved to be more suitable for solving SR problems and have now become the commonly accepted network's configurations. Explicitly modelling dynamic, non-linear dependencies between channels by using global information could ease learning process [7]. Therefore, we considered introducing a channel-wise attention mechanism into local residual learning block. The introduced mechanism could simplify the designation of residual-based learning strategy while improving final performance. We would demonstrate the effectiveness in our ablation experiments. The information available for fusion: Most of methods on SR can be categorised into two main streams: SISR and multiple image SR (MISR). The SISR concerns on estimating HR from a single LR image of the same scene, under the assumption that the original imaging setup is not available. In MISR, the input usually consists of more than one LR image, under the assumption that each one is a degraded version of an underlying HR scene. In terms of available redundant information possessed, recovering a HR image from multiple LR images is to some extent easier than from only single LR image. It is not difficult to find that most of SISR methods are limited that HR could only be restored from only one input, either in the case of function learning from the interpolated LR to HR, or in the case of deconvolutoinal learning from original LR to final HR at the end part of network. They were concentrated on an deterministic learning in one-to-one mode. However, SR is an ill-posed problem and the relationship between LR and HR is in many-to-many mode. That means, one LR could be degraded from different HRs, and one HR could also generate different LRs depending on different degradations. Motivated by the fact, we proposed to artificially create multiple potential HRs from different deconvolutional layers according to different level LR information in network, and then adaptively fuse these potential HRs into final desired HR output. As these intermediate HRs are not restored from different real LRs, respectively, we take this idea as a kind of pseudo-multiple-frame SR strategy. To verify our considerations on how to construct an effective SISR network, we have proposed a compositional network that contains our concerns in this paper. We have performed two ablation experiments and comparisons with several state-of-the-art methods on public benchmark datasets. As we are not completely committed to performance excellence, in our experiments, we avoid using highly tricky training strategies to enhance performance. Nonetheless, the experiment results still demonstrate that our proposed strategies are effective and the proposed compositional network could achieve top performance with reasonable parameters scale and depth. The contributions of this paper are two-fold: A residual refine block via channel-wise attention mechanism is proposed for SR. It explicitly models the dynamic, non-linear dependencies between channels by using global information. As a result, the residual learning is eased and the SR performance is boosted as well. Moreover, the residual refine based strategy is general, which can easily embedded into any residual learning based SISR model. A pseudo-multiple-frame strategy is proposed to augment redundant information for effectively solving SISR problem. It is motivated by the many-to-many relationship between LRs and HRs. Different from most of current SISR methods in which residual blocks were stacked in a chain way, the pseudo-multiple-frame SR strategy can adaptively make full use of hierarchical information and has been demonstrated that it can outperform most of state-of-the-art methods with reasonable parameters size under the same experiment conditions. 2 Related work Our proposed network falls into the category of deep learning based SISR methods. Therefore, in this paper, we mainly discuss the most related deep learning based work recently proposed on SISR. 2.1 Residual-based SISR Due to the proximity from original image to ill-posed recovered image, such as LR to HR, most values of residual images are likely to be small even zero. The learning process from input to output can be speed up through residuals. Moreover, skip connections between input and output have also been demonstrated to be able to avoid gradient vanishing/exploding. Therefore, residual-based learning has been playing great importance on ill-posed image restoring problems. SRGAN(SRResNet) [8] had proposed a generative adversarial network for SR. The proposed perceptual loss consisted of an adversarial loss and a content related loss. It benefited from restoring more photo-realistic HR image. However, the original architecture of ResNet [9] was directly employed without much modification in SRResNet. EDSR [10] was an improved version of SRResNet, and won the NTIRE2017 Challenge on Single Image Super-Resolution [11]. As the authors claimed, the significant performance improvement of their model was due to optimisation by removing unnecessary modules in conventional residual networks and their performance was further improved by expanding the model size. Therefore, it could be said that the high performance of EDSR was in cost of training an extremely large scale and very wide network. DRCN [12] was the first to employ recursive neural network on SR solutions. Global residual learning and recursive supervision was included in the training architecture of network. Different from the chains structure of recursive layers in DRCN, the recursive layers in DRRN [13] were in multi-path mode, in which both global and local residuals were learned. Similar recursive idea was also supported in MemNet [14]. It introduced a memory block consisted of a recursive unit and a gate unit. Dense skip connections between the current recursive unit and outputs from previous memory blocks were employed. The concatenations were input into gate unit to maintain persistent 'memory'. LapSRN [15] progressively reconstructed sub-band residuals of HR images at multiple pyramid levels. Instead of the commonly used L2 loss, Charbonnier loss was adopted to better handle outliers and improve the performance. SRDenseNet [16] employed densely connected convolutional networks [17] as its basic block, and utilised dense skip connections to combine features from different level blocks to provide rich information for final SR reconstruction. 2.2 Squeeze-and-excitation networks SENet [7] was the winner of image classification task in ILSVRC 2017. A 'squeeze-and-excitation' (SE) block was introduced to improve the representational power of a network by explicitly modelling the interdependencies between the channels of its convolutional features. Feature recalibrate was applied to selectively emphasise informative features and suppress less useful ones. The basic structure of SE building block is illustrated in Fig. 1. Fig. 1Open in figure viewerPowerPoint Basic structure of SE building block The SE block embeds global information into each descriptor through squeeze operation. A gating mechanism is parameterised in excitation stage to learn non-linear interaction between each channels. It contains two fully connected (FC) layers. One FC is for dimensionality reduction with ratio r and capturing channel-wise dependencies of features. The other performs sample-specific activations to govern the excitation of each channel based on the learned channel dependence. The advantage of SE block is that it introduces an effective channel-wise attention mechanism and can be stacked at any place needed in network architecture. 2.3 Dense connection block Different from ResNets [9], the feature maps in DenseNet [17] were concatenated rather than directly summed. DenseNet adopted a simple connectivity pattern that each layer in the network took in additional inputs from all preceding layers and passed on its output feature maps to all subsequent layers. The authors claimed that this connectivity pattern could ensure maximum information flow between layers in feed-forward nature. The basic structure of dense connection block consists of a composite sub-block followed by a transition layer, as shown in Fig. 2. The composite sub-block consists of three consecutive operations: batch normalisation (BN), a rectified linear unit (ReLU) and a convolution (Conv). The transition layer consists of a convolutional layer followed by a average pooling layer. Due to the compelling experimental advantages of DenseNet, SRDenseNet [16] had introduced the dense block for SR, and achieved state-of-the-art performance. Fig. 2Open in figure viewerPowerPoint Basic structure of dense block in DenseNet 3 Residual refine based pseudo-multiple-frame network for effective SISR In this section, we will describe our proposed residual refine based pseudo-multi-frame network in detail. The architecture of our proposed network is as shown in Fig. 3. It mainly consists of two parts: residual refine based dense blocks (RRDs) and pseudo-multi-frame information fusion networks. and are the input and the output of the network. Fig. 3Open in figure viewerPowerPoint Architecture of our proposed residual refine based pseudo-multi-frame network As suggested in VDSR [6], when dealing with SR problem, surrounding pixels were useful to correctly infer central pixel. With larger receptive field, a SR model could utilise more contextual information to better learn correspondences from LR to HR. Therefore, in our proposed network, a convolutional layer is first applied on original LR image for aggregating informative features. The size of the convolution filters is , and the step size is 1. As a result, a feature map is extracted with the same spatial dimensions as the input . For the convenience of description, we denote the mapping function of the residual refine based dense block (RRD) as . Supposing we have N residual refine dense blocks, then the output of each RRD is formulated in recursive as follows: (1)where represents the output of refined residuals from the RRD. is the pre-defined initial residual. Since the residuals at each level are hierarchically learned from its previous residuals, we call them as refined residuals. We take the signals as a kind of intermediate 'degraded' LRs. Then we learn respective deconvolution layer for it to HR image. As a result, we artificially create multiple recovered potential 'HR' images by . These intermediate 'HR' images possess redundant information from hierarchical level features. In the final, we adaptively fuse them to obtain the desired HR output, by using a learned fusion function as formulated in the following equation: (2) 3.1 Residual refine based dense block In this section, we present the details about our proposed residual refine block. It consists of a dense block followed by a SE block. The structure is as shown in Fig. 4. Fig. 4Open in figure viewerPowerPoint Structure details of residual refine block As BN layers would increase computational complexity and pooling operations potentially discard pixel-level information, in this paper, following the experiences of previous SR work [16], we remove both the BN layer and the pooling layer out of the dense sub-block in our residual refine block. Dense skip connections are implemented among Conv + ReLU layers. All outputs of Conv + ReLU layers together with the block's input are concatenated on channel dimension, and then transferred into a convolutional transition layer. We formulate the process as , where the is the convolutional weights for transition layer. As we know, increasing the depth of dense block could potentially achieve better performance, such as the dense block used in SRDenseNet has eight Conv + ReLU layers. However, by considering the efficiency and memory usage, in this paper, we set the number of Conv + ReLU layers in our dense sub-block part as , and the growth rate G of the dense block as . That means the convolutional kernel at level is in size of . Our residual refine blocks take role of learning different level residuals in the proposed network. In order to further boost the discriminative ability of learned residuals, a SE block is applied after dense block U. The SE block intrinsically introduces dynamical channel-wise attentions on its input. The resulted recalibrate mappings are hierarchically learned as refined residuals for level k. Specifically, at squeeze stage, a statistic is generated by shrinking feature map on spatial dimensions . The element of z is calculated by (3). The z could be interpreted as a collection of local descriptors, whose statistics are expressive for the whole image (3)Then, at excitation stage, two succeed FC layers are employed to learn the channel-wise attention coefficients , just as (4) does (4)The final output of the SE block is obtained by scaling feature map U with the activations s, as shown in the following equation: (5) 3.2 Pseudo-multi-frame information fusion network In this section, we present the details of pseudo-multiple-frame fusion strategy for restoring HR. The information fusion network consists of several parallel deconvolution layers followed by one concatenation layer and one convolutional bottle layer, as shown in Fig. 3. Sub-pixel convolution network is employed as the deconvolution layer . The sub-pixel convolutional networkconsists of convolutional layer followed by a periodicshuffling layer. We formulate the process as , where represents the deconvolutional weights forthe potential HR image at the level. is a periodic shuffling operator thatrearranges the elements of a input tensor to an output tensor of size. r is the scalefactor of HR compared to the size of LR. The architecture of the deconvolutionnetwork is illustrated in Fig. 5. Fig. 5Open in figure viewerPowerPoint Sub-pixel network for learning HR mapping After N intermediate HR mappings are artificially generated in parallel, they are adaptively fused into the final HR. The process is formulated as , where represents network weights. Specifically, N pseudo-HRs are first concatenated on channel dimension, followed by a bottle layer . The bottle layer is a convolution layer, transforming mapping from to . As the these intermediate HRs are learned from pseudo-LRs of different level degradations, they enrich much redundant information for final HR. Through bottle layer, each pixel information on the resulted HR becomes learnable, which can be adaptively learned from pixel information from multiple intermediate frames. Therefore, we name this strategy as pseudo-multiple-frame based SISR. Increasing the number of pseudo-frames would enhance final performance, while increase depth of network. Therefore, in this paper, to balance the effectiveness and efficiency, we set the number of pseudo-frames N empirically to be . The experiments show that network with these parameter settings can sufficiently achieve satisfactory performance. 4 Experiments In this section, we evaluate the effectiveness of our proposed network on solving SISR problem. The related source code and pre-trained models have been distributed on public at source-code https://github.com/MKFMIKU/RPMNet. 4.1 Datasets On SR issue, in the past, different learning-based methods were trained on different training datasets. Typically, the 91 images from Wright et al. [18] were first used in classic methods, such as A+ [2]. The 291 images consisting of 91 images from Wright et al. [18] and 200 images from Berkeley Segmentation Dataset [19] were widely used in some popular SR methods, such as DRCN [12], DRRN [13], VDSR [6] and MemNet [14]. The large-scale ImageNet dataset was also often used for training deep SR models. For example, in SRResNet [8], 350 thousand images were randomly selected for training. Similarly, in ESPCN [5] and SRDenseNet [16], 5000 images were randomly selected from ImageNet for training. DIV2K [11] dataset is a newly distributed image set for SR challenge. It consists of 800 high-definition high-resolution images for training model. The concurrent state-of-the-art methods like EDSR [10] and our RPMNet were all trained on the DIV2K training set. Data augmentation were applied on all available training images, no matter whatever dataset the above-mentioned methods used. The main differences among them were the different augmentation tricks they used. From our experience, the final performance is in some extent affected by the quality of data augmentation. During test, five popular benchmark datasets: Set5 [20], Set14 [21], BSDS100 [19], Urban100 [22], Manga109 [23] are commonly used. The Set5 and Set14, respectively, contain 5 and 14 images with rich textures. The BSDS100 consists of 100 natural images from test set of the Berkeley segmentation dataset. The Urban100 contains 100 challenge urban images with details in different frequency bands. The Manga109 contains 109 images of Japanese manga. For the wide varieties in visual content, these dataset are widely used for SR evaluation. 4.2 Experiment details Data augmentation: We convert images into YCbCr colour space. Then all operations in our experiment are performed on the luminance component Y. In order to set up training pairs and testing pairs, following [15], we augment the training data in three ways: (i) scaling: downscale images on three scales {1, 0.7, 0.5}; (ii) rotation: randomly rotate image by ; (iii) flipping: flip images in three mirror cases . As a result, 36 variants are generated from original image. After data argumentation, we use sliding window to extract image patches in size of . The stride step is set pixels. r is the scaling factor. We apply bicubic downsampling to simulate LR patches by using the Matlab function imresize with the option 'bicubic' on original patches. In our experiment, for each scaling case, the LR training patches is constantly set in size of pixels. The corresponding original patches are taken as HR ground-truths during training. Training details: The network in our proposed RPMNet model is very straightforward. Four structure parameters were used to measure our network. They are the channels C of the initial convolutional layer, the number of convolution layers D in the dense part of the RRD, the growth rate G of the RRD and the number of pseudo-frames N for fusion. In this experiment, as previously described, these parameters are empirically set to be . We denote the resulted network structure as . The proposed model is trained through minimising the loss between the reconstructed images and the corresponding HR ground-truths. Given a set of restored HR images and their corresponding ground truth , we use mean squared error as the loss function, as shown in the following equation: (6)where M is the number of training samples. The loss is minimised by using stochastic gradient descent with the standard back-propagation. ADAM [24] with learning rate is employed as the optimiser. The batch size is . To avoid different tricks for the learning rate decay, the learning rate in our experiment is kept constant during training process. We implement the network in PyTorch without special weight initialisation method or other training tricks. Three scaling factors are experimented, respectively. We train a specific model for each scaling case. It roughly took one day to train a model to converge by using a NVIDIA Titan Xp GPU. Evaluation: Two commonly-used image quality metrics: PSNR (peak-signal-to-noise ratio) and SSIM (structural similarity index measurement) [25] are employed to evaluate the performance, as defined in the following equation: (7)where is the mean-squared error. x and y are two images in size of , is mean of image's pixel values and is the standard variance. and are commonly set as and . 4.3 Results and analysis 4.3.1 Three ablation experiments One of the ablation experiments is to analyse the impact of the residual refine based strategy on SR. We remove the SE block out of RRD, and the resulted network is denoted as 'RPMNet/SE'. As shown in Fig. 6, the introduced dynamical channel-wise attentions in residual refine block can consistently improve image restoration performance. Under the same conditions, without applying SE block onto the dense block U, the resulted performance decreased averagely about 0.2 dB when tested at scale rate on benchmark dataset Set5 [20] and at scale rate on benchmark dataset Set14 [21]. On the other tested datasets, the conclusion is similar. Fig. 6Open in figure viewerPowerPoint Ablation experiment 1: testing curves of RPMNet on PSNR performance with/without SE in RRD. 'RPMNet/SE' represents network without SE block in RRD We further design an ablation experiment to verify the effectiveness of pseudo-multi-frame based strategy. We construct an ablation network as shown in Fig. 7. It shares similar structure with most of popular published SISR models, which only employ one deconvolution layer at the end part. We take this strategy as single frame case. Fig. 7Open in figure viewerPowerPoint Ablation experiment 2: network in single frame case Compared with our full model RPMNet, the network in Fig. 7 has differences on information fusion part. It employs skip connections and direct concatenations for deconvolution layer. The deconvolution layer is also a sub-pixel convolution network. The convolutional kernel is , where and r is the scaling factors. We argue that single frame case cannot take full advantage of hierarchical information that intermediate levels possess. The performance comparisons between pseudo-multi-frame case and the single frame case are shown in Fig. 8. They are both trained and tested under the same experiment conditions. Fig. 8Open in figure viewerPowerPoint Ablation experiment 2: testing curves of RPMNet on PSNR performance with pseudo-multiple frame or not. The 'RPMNet-Single' represents the network as shown in Fig. 7 From the experimental comparisons in Fig. 8, the pseudo-multi-frame strategy consistently improves the SISR performance over the single frame case on public benchmark datasets. It should be emphasised that the improvements are obtained under the same experiment settings, which can consolidate the advantage of the pseudo-multi-frame strategy. More important is that the pseudo-multi-frame strategy can be easily borrowed for reference by any SISR network with hierarchical information learning based blocks. In addition to verify RPMNet can suit different components, we further perform an ablationexperiment, in which we compare models in the same RMPNet architecture butwith different block types. We name our default RPMNet with Denseblock as . If replace Denseblock in Fig. 3 by Resblock from ResNet [9], we call this varied