# Multilevel Feature Exploration Network for Image Superresolution

Authors: Xinbo Liu; Ling Wang; Xinyu Chen; Yuqing Liu
Journal: Scientific Programming
 (2022)

---

## Abstract

Image superresolution (SR) is a classical issue in computer vision area. Recently, there are elaborated convolutional neural networks (CNNs) demonstrating remarkable effectiveness on image SR. However, most of the previous works lack effective exploration on the structural information, which plays a critical role for image quality. In this paper, we find that the hierarchical design can effectively restore the structural information and devise a multilevel feature exploration network for image SR (MFSR). Specially, we design an encoder-decoder architecture to concentrate on structural information from different levels and devise a spatial attention mechanism to address the inherent correlation among features for effective restoration. Experimental results show the proposed MFSR can restore more correct edges and lines and achieves both better objective and subjective performances than the state-of-the-art methods with higher PSNR/SSIM results, indicating the effectiveness on structural information restoration.


---

## Body

## 1. Introduction

Image superresolution (SR), as a traditional image processing issue, is widely considered in advanced vision applications, such as medical image enhancement, slope collapse detection, and art image classification [1]. Given a low-resolution (LR) image, the task of image SR is to find a corresponding high-resolution (HR) instance with satisfactory visual experience and more correct textures [2].Recently, convolutional neural networks (CNNs) show the amazing performance on image restoration. As far as we know, superresolution convolutional neural network (SRCNN) [3] proposed by Dong et al. is the first CNN-based method for image SR. After SRCNN, there are elaborate network designs for effective image SR, such as enhanced deep residual superresolution network (EDSR) [4], residual dense network for image superresolution (RDN) [5, 6], deep residual channel attention image superresolution network (RCAN) [7], deep Laplacian pyramid superresolution network (LapSRN) [8, 9], and cascading residual network (CARN) [10]. These works concentrate on elaborate designs and utilize different blocks for effective performance. However, very few of them specially concentrate on the structural information restoration.There are also works considering the structural information as a prior. Fang et al. design a soft-edge assisted network for single image SR, which is termed as SeaNet [11]. Cheng et al. consider gradient guidance into structure-preserving superresolution (SPSR) [12] and utilize generative adversarial network for satisfactory performance. Sparse-mask superresolution network (SMSR) is also investigated by Zhang et al. for efficient inference. Yang et al. devise a deep edge guided recurrent residual network for image SR (DEGREE) [13]. However, the structural information is usually applied as a prior term, and they almost neglect to focus on designing effective architecture to explore the edges and textures.Multiscale structure proves to be an effective design pattern for structural information exploration. Multiscale residual network for image superresolution (MSRN) [14] designs a multiscale residual block for effective feature exploration. Recently, multiscale dense cross network (MDCN) [15] is also proposed for hierarchical exploration. However, these works lack finding the inherent correlations among features with restricted performance.Attention mechanism has become a popular design in recent CNNs for different tasks. Recently, channel-wise attention [16, 17] has also been widely considered in image SR networks, such as residual-in-residual channel attention network for image SR (RCAN) [7] and channel-wise and spatial feature modulation network (CSFM) for single image SR [18]. However, the channel-wise attention usually utilizes a global pooling method to estimate the information of different channels, which misses the spatial diversity of feature maps.In this paper, we design a multilevel feature exploration network for image superresolution, which is termed as MFSR. To effectively explore the structural information, we consider a UNet-style [19] to hierarchically process the image features and concatenate the multiscale features for joint restoration. Besides the UNet-like backbone, we also devise a spatial attention mechanism to consider the inherent correlation among spatial information and remove the global pooling operation in traditional channel-wise attention, which effectively avoids the information loss. Based on the spatial attention mechanism, we devise a spatial attention residual block (SARB) for effective structural information exploration. Experimental results show our MFSR can restore the HR images with more satisfactory textures and closer to the ground truth. Compared with the state-of-the-art methods, specially multiscale structure-based networks, our network achieves much higher peak signal-to-noise rate (PSNR) [20] and structural similarity (SSIM) [21] results with better visual quality.For this paper, the main contributions are as follows:(i)
We design a multilevel feature exploration network for image superresolution, which can effectively process the hierarchical structural information.(ii)
We devise a spatial attention mechanism to explore the inherent correlation among spatial information without global pooling operation, which effectively avoids the information loss. Based on the spatial attention mechanism, we devise a spatial attention residual block (SARB) for effective structural information exploration.(iii)
Experimental results show our MFSR can restore the HR images with more satisfactory textures and closer to the ground truth with higher PSNR/SSIM results.In the following, we introduce the related work in Section2. Then, the detail of MFSR is described in Section 3. Finally, the experiment results are shown in Section 4.


## 2. Related Work

### 2.1. Image Superresolution

Image SR is a traditional topic in computer vision area, which is widely investigated in recent years. Classical SR methods can be generally separated into five types: interpolation-based methods, filter-based methods, patch-based methods [22, 23], optimization-based methods [24, 25], and sparse-coding-based methods [26, 27]. Interpolation aims to build a function that uses the neighbor pixel values to estimate the missing pixel value. Cao et al. bring low-rank matrix completion and recovery into image SR and investigate a novel representation of interpolation function [28]. Similar to interpolation methods, filter-based methods try to find a feasible filter to change the pixel value with the help of neighbor’s information and generate pleasant visual results. Patch-based methods separate the image into several adjacent patches and optimize the performance of each patch separately to improve the whole image quality. Zhu et al. use deformable patches for single image SR [22]. Yang et al. convert the patches into sparse representation and perform the image restoration for raw data [23]. Optimization-based methods consider single image SR as minimizing the energy function and utilize iterative optimization to approximate the solution. Huang and Xia use matrix-variable optimization for fast blind image superresolution [24], and Shi et al. optimize the image quality with low-rank and total variation regularization [25]. Different from previous works, sparse-coding-based methods enable the learning step into image SR and derive a codebook for adaptive image restoration. Shi and Qi consider low-rank sparse representation with self-similarity for image SR [26]. Lu et al. also perform geometry constrained sparse coding to produce an acceptable result.Convolutional neural networks (CNNs) demonstrate amazing performance on image SR task. To the best of our knowledge, superresolution convolutional neural network (SRCNN) [3] is the first CNN-based method for image SR, which brings a greater performance improvement than traditional methods. SRCNN follows a sparse-coding-like structure and uses three convolutional layers to denote the feature extraction, nonlinear mapping, and restoration operations separately. After SRCNN, there are numerous CNN-based works with elaborate designs for superior capacity. A fast version of SRCNN termed as FSRCNN [29] is proposed by Dong et al. to accelerate the inference phase and achieves better performance than SRCNN with a deeper but faster network. A very deep network with residual connection termed as VDSR [30] also increases the network depth for boosting the capacity. Kim et al. design a deep recursive network for image SR, which is termed as DRCN [31]. These works use deconvolution to upscale the feature map and generate the HR image or preupsample the image before the network processing. Shi et al. provide a new perspective to upscale the feature map by subpixel convolution and integrate the upsampling operation into the efficient subpixel convolutional neural network (ESPCN) [32]. After ESPCN, most of the image SR networks consider subpixel convolution to upscale the feature map instead of using deconvolution. Recent end-to-end networks can be generally separated into single-scale methods and multiscale methods. Single-scale networks consider a deeper and wider design to seek higher quantitative performance. Enhanced deep residual network for image SR (EDSR) is a representative method to achieve the state-of-the-art performance, which stacks numerous residual blocks [33] and removes the batch normalization [34] layers. After EDSR, there are well-designed blocks for image SR with state-of-the-art performance. Dense net for image SR (SRDenseNet) [35] introduces the densely connection into image SR. Zhang et al. combine the superiority of dense connection and residual connection and propose a residual dense connection network (RDN) for image SR [5, 6]. Recently, residual-in-residual network with channel attention for image SR (RCAN) [7], residual feature aggregation deep network (RFDN) [36], and other works also achieve state-of-the-art performance. Deep network with component learning (DNCL) is recently presented for fast and accurate image SR. Inspired by traditional filter-based methods, Li et al. propose an adaptive filter learning network termed as FilterNet [37].Besides the single-scale networks, there are also multiscale methods that concentrate on the hierarchical structural information exploration. Laplacian pyramid image SR network (LapSRN) [8, 9] designs a Laplacian pyramid inspired network to progressively restore the residual image and increases the image resolution. Multiscale residual network (MSRN) [14] utilizes a multiscale residual block for accurate image restoration. Multiscale dense cross network (MDCN) upgrades the MSRN and achieves state-of-the-art image SR performance. Resolution-aware network (RAN) [38] was recently considered for simultaneous SR of multiple factors. He et al. consider a multiscale design to accelerate the network speed, which is named as MRFN [39]. Recently, a deep inception-residual Laplacian pyramid network (IRLP) is also proposed for accurate single image SR [40].


### 2.2. Attention Mechanism

As far as we know, channel-wise attention [16, 17] is the first tiny but effective attention block for CNN design. The channel-wise attention omits the spatial information and only concentrates on the different importance of channels. Residual-in-residual channel attention network (RCAN) [7], information multiple distillation network (IMDN) [41], and other recent works consider the channel-wise attention for better restoration performance. Besides the channel-wise attention, there are also effective attention mechanisms for image SR. Residual feature information distillation network (RFDN) [42] proposes an enhanced spatial attention to overcome the shortcoming of channel-wise attention and considers the spatial information for better attention calculation.


## 2.1. Image Superresolution

Image SR is a traditional topic in computer vision area, which is widely investigated in recent years. Classical SR methods can be generally separated into five types: interpolation-based methods, filter-based methods, patch-based methods [22, 23], optimization-based methods [24, 25], and sparse-coding-based methods [26, 27]. Interpolation aims to build a function that uses the neighbor pixel values to estimate the missing pixel value. Cao et al. bring low-rank matrix completion and recovery into image SR and investigate a novel representation of interpolation function [28]. Similar to interpolation methods, filter-based methods try to find a feasible filter to change the pixel value with the help of neighbor’s information and generate pleasant visual results. Patch-based methods separate the image into several adjacent patches and optimize the performance of each patch separately to improve the whole image quality. Zhu et al. use deformable patches for single image SR [22]. Yang et al. convert the patches into sparse representation and perform the image restoration for raw data [23]. Optimization-based methods consider single image SR as minimizing the energy function and utilize iterative optimization to approximate the solution. Huang and Xia use matrix-variable optimization for fast blind image superresolution [24], and Shi et al. optimize the image quality with low-rank and total variation regularization [25]. Different from previous works, sparse-coding-based methods enable the learning step into image SR and derive a codebook for adaptive image restoration. Shi and Qi consider low-rank sparse representation with self-similarity for image SR [26]. Lu et al. also perform geometry constrained sparse coding to produce an acceptable result.Convolutional neural networks (CNNs) demonstrate amazing performance on image SR task. To the best of our knowledge, superresolution convolutional neural network (SRCNN) [3] is the first CNN-based method for image SR, which brings a greater performance improvement than traditional methods. SRCNN follows a sparse-coding-like structure and uses three convolutional layers to denote the feature extraction, nonlinear mapping, and restoration operations separately. After SRCNN, there are numerous CNN-based works with elaborate designs for superior capacity. A fast version of SRCNN termed as FSRCNN [29] is proposed by Dong et al. to accelerate the inference phase and achieves better performance than SRCNN with a deeper but faster network. A very deep network with residual connection termed as VDSR [30] also increases the network depth for boosting the capacity. Kim et al. design a deep recursive network for image SR, which is termed as DRCN [31]. These works use deconvolution to upscale the feature map and generate the HR image or preupsample the image before the network processing. Shi et al. provide a new perspective to upscale the feature map by subpixel convolution and integrate the upsampling operation into the efficient subpixel convolutional neural network (ESPCN) [32]. After ESPCN, most of the image SR networks consider subpixel convolution to upscale the feature map instead of using deconvolution. Recent end-to-end networks can be generally separated into single-scale methods and multiscale methods. Single-scale networks consider a deeper and wider design to seek higher quantitative performance. Enhanced deep residual network for image SR (EDSR) is a representative method to achieve the state-of-the-art performance, which stacks numerous residual blocks [33] and removes the batch normalization [34] layers. After EDSR, there are well-designed blocks for image SR with state-of-the-art performance. Dense net for image SR (SRDenseNet) [35] introduces the densely connection into image SR. Zhang et al. combine the superiority of dense connection and residual connection and propose a residual dense connection network (RDN) for image SR [5, 6]. Recently, residual-in-residual network with channel attention for image SR (RCAN) [7], residual feature aggregation deep network (RFDN) [36], and other works also achieve state-of-the-art performance. Deep network with component learning (DNCL) is recently presented for fast and accurate image SR. Inspired by traditional filter-based methods, Li et al. propose an adaptive filter learning network termed as FilterNet [37].Besides the single-scale networks, there are also multiscale methods that concentrate on the hierarchical structural information exploration. Laplacian pyramid image SR network (LapSRN) [8, 9] designs a Laplacian pyramid inspired network to progressively restore the residual image and increases the image resolution. Multiscale residual network (MSRN) [14] utilizes a multiscale residual block for accurate image restoration. Multiscale dense cross network (MDCN) upgrades the MSRN and achieves state-of-the-art image SR performance. Resolution-aware network (RAN) [38] was recently considered for simultaneous SR of multiple factors. He et al. consider a multiscale design to accelerate the network speed, which is named as MRFN [39]. Recently, a deep inception-residual Laplacian pyramid network (IRLP) is also proposed for accurate single image SR [40].


## 2.2. Attention Mechanism

As far as we know, channel-wise attention [16, 17] is the first tiny but effective attention block for CNN design. The channel-wise attention omits the spatial information and only concentrates on the different importance of channels. Residual-in-residual channel attention network (RCAN) [7], information multiple distillation network (IMDN) [41], and other recent works consider the channel-wise attention for better restoration performance. Besides the channel-wise attention, there are also effective attention mechanisms for image SR. Residual feature information distillation network (RFDN) [42] proposes an enhanced spatial attention to overcome the shortcoming of channel-wise attention and considers the spatial information for better attention calculation.


## 3. Methodology

In this section, we introduce the MFSR in the following manner. Firstly, we describe the overall network design for restoring HR image from LR instance. Then, we specially discuss the blocks design of SARB. Finally, we show the hyperparameter settings for MFSR.


### 3.1. Network Design

We aim to restore the HR imageIH from the given LR image IL. Let fMFSR⋅ be the proposed MFSR; then, the restoration is(1)IH=fMFSRIL.Figure1 shows the network structure of proposed MFSR. There are three main modules in the MFSR: feature extraction module, multilevel feature exploration module, and restoration module. Before the restoration, one convolutional layer converts the image into feature space for further exploration as(2)Fcvt=ConvIL,where Fcvt is the converted feature maps from IL.Figure 1
The network structure of the proposed multilevel feature exploration image superresolution network (MFSR).The feature extraction module extracts the feature maps fromFcvt. Let fFE⋅ denote the feature extraction module; then, there is(3)FL=fFEFcvt,where FL represents the extracted features from LR image.After feature extraction, the multilevel feature exploration module processesFL in a hierarchical manner for effectively restoring the structural information. Firstly, we build the multilevel features form FL for further processing. We use max pooling to downscale FL and generate feature maps with different resolutions as(4)F1×1L=FL↓1×1,F2×2L=FL↓2×2,F4×4L=FL↓4×4,where ⋅↓s×s denotes the max pooling with scaling factor s×s. The downscaled features with different resolutions are explored by several spatial attention residual blocks (SARB) for generating the feature maps with HR information. There are ms SARBs for processing the feature maps with scaling factor s×s. After SARBs, a padding structure with residual connection is devised to keep the original information and improve the gradient transmission. The padding structure consists of two convolutional layers and one ReLU activation. Thus, the multilevel exploration is(5)F1×1H=f1×1Padfm1SARBF1×1L+F1×1L,F2×2H=f2×2Padfm2SARBF2×2L+F2×2L,F4×4H=f4×4Padfm4SARBF4×4L+F4×4L,where fs×sPad⋅ is the padding structure for scaling factor s×s and fmsSARB⋅ means the processing procedure with ms SARBs. Fs×sH is the HR feature with scaling factor s×s.After exploration, the multilevel features should be concatenated jointly for hierarchical consideration. In this paper, we use bilinear interpolation to upscale the resolution. Bilinear interpolation considers the image interpolation from two directions and finds a suitable value for missing pixels [43]. The concatenation is demonstrated as(6)FMLH=ConcatF1×1H,F2×2H↑2×2F4×4H↑4×4,where Concat⋅ is the concatenation operation, ⋅↑s×s is the bilinear upsampling operation, and the concatenated multilevel HR feature is FMLH.After multilevel exploration, the restoration module generates the HR image from the feature maps. There are two steps in the restoration module. Firstly, we explore the multilevel HR feature for further integrating the multilevel information. There aren SARBs for the information integration as(7)FH=fnSARBFMLH,where FH is the explored hierarchical HR feature. Then, the upscale block generates the image from the feature maps, which is composed of one convolutional layer and one subpixel convolution. Skip connection is designed between the LR and HR features for better gradient transmission. Finally, there is(8)IH=UpscaleFH+Fcvt,where Upscale⋅ is the upscale block.The overall proposed MFRN can be concluded as in Algorithm 1.(1)
MFSRILInput: The input LR imageILOutput: The restored HR imageIH(2)
Convert LR image into feature space by convolution asFcvt=ConvIL.(3)
Extract LR features by the feature extraction module asFL=fFEFcvt.(4)
Generate the hierarchical HR feature according equation (4)–(7).(5)
Produce the HR image from the feature asIH = Upscale (FH+Fcvt)(6)
ReturnIH.Algorithm 1: The proposed MFSR for generating HR image from the LR instance


### 3.2. Spatial Attention Residual Block

Spatial attention residual block (SARB) is the main component for the MFSR, which is generally utilized in the multilevel feature exploration module and the restoration module. The left-middle box in Figure1 shows the block design of SARB. SARB enjoys a residual design with two convolutional layers, one Leaky ReLU activation, and one spatial attention (SA) layer. The skip connection in SARB helps effective gradient transmission and information preservation. Specially, we mainly introduce the spatial attention in the SARB that effectively explore the spatial correlation among features.Figure2 shows the architecture of SA layer, which is composed of four convolutional layers, three Leaky ReLU activation, and a Sigmoid activation. Let Finsa be the input feature of SA, there is one convolution with ReLU activation to process the feature as(9)F1sa=LReLUConvFinsa.Figure 2
The structure of spatial attention (SA) mechanism.After convolution, the feature maps are downsampled by max pooling with scaling factor3×3 to increase the receptive field. Then, one convolutional layer and one ReLU activation explore the correlation among features as(10)F2sa=LReLUConvF1sa↓3×3.The processed feature will be upscaled by bilinear upsampling to the original size for calculating the attention. After upscaling, two convolutional layers with a ReLU activation explore the feature to find the attention map. Sigmoid activation is used to maintain the nonnegativity. Finally, the output feature of SA layer is(11)Foutsa=σConvLReLUConvF2sa↑3×3∗Finsa,where σ⋅ denotes the Sigmoid activation and ↑3×3 denotes the bilinear interpolation with scaling factor 3×3.It is worth noting that SA considers the spatial correlation of features. Different from channel-wise attention which utilizes global pooling to embed the channel information [7, 16, 18], SA considers the spatial correlation by exploring the attention with the same size of the input feature maps. By utilizing SA, every pixel of feature map will get a specific attention value.


### 3.3. Settings

In this paper, all layers are set with filter number as 64 except for the upscale block. The filter number of the upscale block follows the same design as IMDN [41]. There are m1=16 SARBs in the 1×1 scale of multilevel feature exploration, m2=8 SARBs in the 2×2 scale, and m4=4 SARBs in the 4×4 scale. It is worth noting that the block numbers are set to maintain the same receptive field for different scaling factors. There are four convolutions in feature extraction module, and five SARB blocks in restoration module. The filter sizes of all convolutional layers are set as 3×3.


## 3.1. Network Design

We aim to restore the HR imageIH from the given LR image IL. Let fMFSR⋅ be the proposed MFSR; then, the restoration is(1)IH=fMFSRIL.Figure1 shows the network structure of proposed MFSR. There are three main modules in the MFSR: feature extraction module, multilevel feature exploration module, and restoration module. Before the restoration, one convolutional layer converts the image into feature space for further exploration as(2)Fcvt=ConvIL,where Fcvt is the converted feature maps from IL.Figure 1
The network structure of the proposed multilevel feature exploration image superresolution network (MFSR).The feature extraction module extracts the feature maps fromFcvt. Let fFE⋅ denote the feature extraction module; then, there is(3)FL=fFEFcvt,where FL represents the extracted features from LR image.After feature extraction, the multilevel feature exploration module processesFL in a hierarchical manner for effectively restoring the structural information. Firstly, we build the multilevel features form FL for further processing. We use max pooling to downscale FL and generate feature maps with different resolutions as(4)F1×1L=FL↓1×1,F2×2L=FL↓2×2,F4×4L=FL↓4×4,where ⋅↓s×s denotes the max pooling with scaling factor s×s. The downscaled features with different resolutions are explored by several spatial attention residual blocks (SARB) for generating the feature maps with HR information. There are ms SARBs for processing the feature maps with scaling factor s×s. After SARBs, a padding structure with residual connection is devised to keep the original information and improve the gradient transmission. The padding structure consists of two convolutional layers and one ReLU activation. Thus, the multilevel exploration is(5)F1×1H=f1×1Padfm1SARBF1×1L+F1×1L,F2×2H=f2×2Padfm2SARBF2×2L+F2×2L,F4×4H=f4×4Padfm4SARBF4×4L+F4×4L,where fs×sPad⋅ is the padding structure for scaling factor s×s and fmsSARB⋅ means the processing procedure with ms SARBs. Fs×sH is the HR feature with scaling factor s×s.After exploration, the multilevel features should be concatenated jointly for hierarchical consideration. In this paper, we use bilinear interpolation to upscale the resolution. Bilinear interpolation considers the image interpolation from two directions and finds a suitable value for missing pixels [43]. The concatenation is demonstrated as(6)FMLH=ConcatF1×1H,F2×2H↑2×2F4×4H↑4×4,where Concat⋅ is the concatenation operation, ⋅↑s×s is the bilinear upsampling operation, and the concatenated multilevel HR feature is FMLH.After multilevel exploration, the restoration module generates the HR image from the feature maps. There are two steps in the restoration module. Firstly, we explore the multilevel HR feature for further integrating the multilevel information. There aren SARBs for the information integration as(7)FH=fnSARBFMLH,where FH is the explored hierarchical HR feature. Then, the upscale block generates the image from the feature maps, which is composed of one convolutional layer and one subpixel convolution. Skip connection is designed between the LR and HR features for better gradient transmission. Finally, there is(8)IH=UpscaleFH+Fcvt,where Upscale⋅ is the upscale block.The overall proposed MFRN can be concluded as in Algorithm 1.(1)
MFSRILInput: The input LR imageILOutput: The restored HR imageIH(2)
Convert LR image into feature space by convolution asFcvt=ConvIL.(3)
Extract LR features by the feature extraction module asFL=fFEFcvt.(4)
Generate the hierarchical HR feature according equation (4)–(7).(5)
Produce the HR image from the feature asIH = Upscale (FH+Fcvt)(6)
ReturnIH.Algorithm 1: The proposed MFSR for generating HR image from the LR instance


## 3.2. Spatial Attention Residual Block

Spatial attention residual block (SARB) is the main component for the MFSR, which is generally utilized in the multilevel feature exploration module and the restoration module. The left-middle box in Figure1 shows the block design of SARB. SARB enjoys a residual design with two convolutional layers, one Leaky ReLU activation, and one spatial attention (SA) layer. The skip connection in SARB helps effective gradient transmission and information preservation. Specially, we mainly introduce the spatial attention in the SARB that effectively explore the spatial correlation among features.Figure2 shows the architecture of SA layer, which is composed of four convolutional layers, three Leaky ReLU activation, and a Sigmoid activation. Let Finsa be the input feature of SA, there is one convolution with ReLU activation to process the feature as(9)F1sa=LReLUConvFinsa.Figure 2
The structure of spatial attention (SA) mechanism.After convolution, the feature maps are downsampled by max pooling with scaling factor3×3 to increase the receptive field. Then, one convolutional layer and one ReLU activation explore the correlation among features as(10)F2sa=LReLUConvF1sa↓3×3.The processed feature will be upscaled by bilinear upsampling to the original size for calculating the attention. After upscaling, two convolutional layers with a ReLU activation explore the feature to find the attention map. Sigmoid activation is used to maintain the nonnegativity. Finally, the output feature of SA layer is(11)Foutsa=σConvLReLUConvF2sa↑3×3∗Finsa,where σ⋅ denotes the Sigmoid activation and ↑3×3 denotes the bilinear interpolation with scaling factor 3×3.It is worth noting that SA considers the spatial correlation of features. Different from channel-wise attention which utilizes global pooling to embed the channel information [7, 16, 18], SA considers the spatial correlation by exploring the attention with the same size of the input feature maps. By utilizing SA, every pixel of feature map will get a specific attention value.


## 3.3. Settings

In this paper, all layers are set with filter number as 64 except for the upscale block. The filter number of the upscale block follows the same design as IMDN [41]. There are m1=16 SARBs in the 1×1 scale of multilevel feature exploration, m2=8 SARBs in the 2×2 scale, and m4=4 SARBs in the 4×4 scale. It is worth noting that the block numbers are set to maintain the same receptive field for different scaling factors. There are four convolutions in feature extraction module, and five SARB blocks in restoration module. The filter sizes of all convolutional layers are set as 3×3.


## 4. Experiment

In this paper, we train our network with DIV2K [44] dataset. DIV2K is firstly proposed in NTIRE 2017 challenge [44] and contains numerous high-resolution instances for image SR. There are numerous SR works trained on this dataset [4–7]. We choose 800 images from DIV2K for training, and 5 images for validation. For testing, we use five widely used benchmarks to evaluate the performance: Set5 [45], Set14 [46], B100 [47], Urban100 [48], and Manga109 [49]. Peak signal-to-noise ratio (PSNR) [20] and structural similarity [21] are selected to estimate the objective performance of restored images. We update our model by Adam [50] optimizer with learning rate as lr=10−4 and halve the learning rate for every 200 epochs. The loss function of this paper is set as L1-loss. We use PyTorch 0.4.1 to train the network. The CUDA version is 10.0. The batch size is set as 32. The experimental environment is Ubuntu 18.04, and we train the model on one NVIDIA GTX-1080Ti GPU.


### 4.1. Ablation Study

#### 4.1.1. Investigation on the Multilevel Exploration

In this paper, we set different number of SARBs in the three scales of multilevel exploration module to keep the receptive field. To investigate the effectiveness of the number settings, we consider different combinations for comparison. Table1 shows the performances of different block settings. Herein, (m1, m2, m4) denotes the different number of SARBs for the scaling factor as 1×1, 2×2, and 4×4. In the table, we find that, with the increase of SARBs, the PSNR/SSIM results are higher. According to the third and forth rows, the PSNR of m1=16 on Set5 is 32.23 dB while the PSNR of m1=8 is 32.17 dB. The decrease of m1 leads to near 0.06 dB PSNR drop on Set5. However, the PSNR results of the second, third, and last rows on Set5 are 32.26 dB, 32.23 dB, and 32.26 dB separately, which shows the decrease of m2 and m4 leads to less than 0.03 dB drop on PSNR. In this point of view, m1 influences the performance more than blocks from other levels. We believe this is because the resolution of m1 is larger than other levels, and the computation complexity is much higher.Table 1
Investigation on PSNR/SSIM performances of different multilevel settings. Herein, (m1, m2, m4) denotes the different number of SARB on the three pathways.
SettingSet5Set14B100(16, 0, 0)32.24/0.895428.64/0.783427.62/0.7380(16, 8, 0)32.26/0.895828.63/0.783427.62/0.7378(16, 4, 2)32.23/0.895228.63/0.783527.62/0.7379(8, 4, 2)32.17/0.894828.60/0.782227.59/0.7368(16, 8, 4)32.26/0.896128.65/0.783827.63/0.7381


#### 4.1.2. Investigation on Spatial Attention

We also investigate the effectiveness of spatial attention. Table2 shows the performance comparisons on spatial attention. The model with spatial attention enjoys the SARB design proposed in the paper. The model without spatial attention holds the same block design as EDSR [4]. In the table, we can find that the spatial attention mechanism brings 0.04 dB, 0.02 dB, and 0.04 dB PSNR gain on Set5, Set14, and B100 datasets separately. Similarly, the model with spatial attention achieves 0.8961, 0.7838, and 0.7381 SSIM results on different benchmarks, which are all higher than the model without attention mechanism. In this point of view, attention mechanism is an effective component for better performance.Table 2
Investigation on PSNR/SSIM performances of spatial attention (SA) mechanism.
<!—Col Count:4F0E0AttentionSet5Set14B100w/o32.22/0.895428.63/0.783127.59/0.7373w32.26/0.896128.65/0.783827.63/0.7381


### 4.2. Comparison with State-of-the-Art Methods

We mainly compare our model with several classical and recent works, including SRCNN [3], FSRCNN [29], VDSR [30], DRCN [31], LapSRN [8, 9], RAN [38], DNCL [51], FilterNet [37], MRFN [39], SeaNet [11], DEGREE [13], IRLP [40], FSN [52], and DSRLN [53]. Table 3 shows the PSNR/SSIM performance on five benchmarks with scaling factor ×2, ×3, and ×4.Table 3
Average PSNR/SSIM with degradation modelBI×2, ×3, and ×4 on five benchmarks. The best performances are shown in bold. The dash line means the paper does not report their performance.
ScaleModelSet5 [45]Set14 [46]B100 [47]Urban100 [48]Manga109 [49]PSNR/SSIMPSNR/SSIMPSNR/SSIMPSNR/SSIMPSNR/SSIM×2SRCNN [3]36.66/0.954232.42/0.906331.36/0.887929.50/0.894635.74/0.9661FSRCNN [29]37.00/0.955832.63/0.908831.53/0.892029.88/0.902036.67/0.9694VDSR [30]37.53/0.958733.03/0.912431.90/0.896030.76/0.914037.22/0.9729DRCN [31]37.63/0.958833.04/0.911831.85/0.894230.75/0.913337.63/0.9723LapSRN [9]37.52/0.959033.08/0.913031.80/0.895030.41/0.910037.27/0.9740RAN [38]37.58/0.959233.10/0.913331.92/0.8963——DNCL [51]37.65/0.959933.18/0.914131.97/0.897130.89/0.9158—FilterNet [37]37.86/0.961033.34/0.915032.09/0.899031.24/0.9200—MRFN [39]37.98/0.961133.41/0.915932.14/0.899731.45/0.922138.29/0.9759SeaNet-baseline [11]37.99/0.960733.60/0.917432.18/0.899532.08/0.927638.48/0.9768DEGREE [13]37.58/0.958733.06/0.912331.80/0.8974——IRLP [40]37.83/0.969233.18/0.924732.01/0.902130.86/0.919237.71/0.9861FSN [52]37.68/0.906533.51/0.918032.09/0.901531.68/0.9248—DSRLN [53]38.05/—33.56/—32.38/—31.79/——MFSR38.07/0.960833.69/0.919132.22/0.900232.35/0.930738.75/0.9768×3SRCNN [3]32.75/0.909029.28/0.820928.41/0.786326.24/0.798930.59/0.9107FSRCNN [29]33.16/0.914029.43/0.824228.53/0.791026.43/0.808030.98/0.9212VDSR [30]33.66/0.921329.77/0.831428.82/0.797627.14/0.827932.01/0.9310DRCN [31]33.82/0.922629.76/0.831128.80/0.796327.15/0.827632.31/0.9328DRRN [31]34.03/0.924429.96/0.834928.95/0.800427.53/0.837832.74/0.9390RAN [38]33.71/0.922329.84/0.832628.84/0.7981——DNCL [51]33.95/0.923229.93/0.834028.91/0.799527.27/0.8326—FilterNet [37]34.08/0.925030.03/0.837028.95/0.803027.55/0.8380—MRFN [39]34.21/0.926730.03/0.836328.99/0.802927.53/0.838932.82/0.9396SeaNet-baseline [11]34.36/0.928030.34/0.842829.09/0.805328.17/0.852733.40/0.9444DEGREE [13]33.76/0.921129.82/0.832628.74/0.7950——IRLP [40]34.22/0.943730.16/0.847229.21/0.816327.58/0.832732.61/0.9492DSRLN [53]34.56/—30.36/—29.29/—27.88/——MFSR34.49/0.928030.42/0.844229.16/0.806828.39/0.857733.72/0.9457×4SRCNN [3]30.48/0.862827.49/0.750326.90/0.710124.52/0.722127.66/0.8505FSRCNN [29]30.71/0.865727.59/0.753526.98/0.715024.62/0.728027.90/0.8517VDSR [30]31.35/0.883828.01/0.767427.29/0.725125.18/0.752428.83/0.8809DRCN [31]31.53/0.885428.02/0.767027.23/0.723325.14/0.751028.98/0.8816LapSRN [9]31.54/0.885028.19/0.772027.32/0.728025.21/0.756029.09/0.8845RAN [38]31.43/0.884728.09/0.769127.31/0.7260——DNCL [51]31.66/0.887128.23/0.771727.39/0.728225.36/0.7606—FilterNet [37]31.74/0.890028.27/0.773027.39/0.729025.53/0.7680—MRFN [39]31.90/0.891628.31/0.774627.43/0.730925.46/0.765429.57/0.8962SeaNet-baseline [11]32.18/0.894828.61/0.782227.57/0.735926.05/0.789630.44/0.9088DEGREE [13]31.47/0.883728.10/0.766927.20/0.7216——IRLP [40]31.94/0.902128.49/0.791027.70/0.747925.61/0.776429.49/0.9025FSN [52]32.10/0.895928.57/0.787427.53/0.743825.76/0.7817—MFSR32.26/0.896128.65/0.783827.63/0.738126.25/0.791930.62/0.9103In the table, we can find the proposed MFSR achieves the best performance on all testing benchmarks. It is worth noting that our model is mainly designed for structural information restoration. In this point of view, we achieve significant PSNR/SSIM improvement on the benchmark with plentiful lines and edges, such as Urban100 and Manga109. Particularly, our method achieves near 0.3 dB higher on Manga109 and Urban100 than other works with scaling factor×2 and ×3. When scaling factor is ×4, we achieve near 0.2 dB higher than other works. Urban100 is composed of high-resolution building images, and Manga109 is composed of comic cover pages. Thus, the high PSNR/SSIM performance demonstrates our effectiveness on structural information restoration.We also achieve superior performance than other edge preserving and multiscale SR works. FilterNet, DEGREE, and SeaNet are aiming at structure information preserving, and MRFN is a modern effective multiscale network for image SR. In Table3, our model gains more than 0.2 dB on PSNR, which is a significant improvement.Besides quantitative comparisons, we also compare the visualization performance with different methods. We mainly compare our method with LapSRN [9], MSRN [14], and CARN [10]. LapSRN is a representative progressive structure inspired by Laplacian pyramid, and MSRN is a state-of-the-art multiscale design. Thus, we compare with them to show the effectiveness of our multilevel design. Figure 3 shows the visualization comparison. In the first and second rows of the figure, our method recovers the lines on the building more accurately than MSRN and LapSRN. In the last row, our method can restore more correct small lines, while both LapSRN and MSRN fail to recover the straight lines.Figure 3
Visualization comparison on Urban100 dataset.


## 4.1. Ablation Study

### 4.1.1. Investigation on the Multilevel Exploration

In this paper, we set different number of SARBs in the three scales of multilevel exploration module to keep the receptive field. To investigate the effectiveness of the number settings, we consider different combinations for comparison. Table1 shows the performances of different block settings. Herein, (m1, m2, m4) denotes the different number of SARBs for the scaling factor as 1×1, 2×2, and 4×4. In the table, we find that, with the increase of SARBs, the PSNR/SSIM results are higher. According to the third and forth rows, the PSNR of m1=16 on Set5 is 32.23 dB while the PSNR of m1=8 is 32.17 dB. The decrease of m1 leads to near 0.06 dB PSNR drop on Set5. However, the PSNR results of the second, third, and last rows on Set5 are 32.26 dB, 32.23 dB, and 32.26 dB separately, which shows the decrease of m2 and m4 leads to less than 0.03 dB drop on PSNR. In this point of view, m1 influences the performance more than blocks from other levels. We believe this is because the resolution of m1 is larger than other levels, and the computation complexity is much higher.Table 1
Investigation on PSNR/SSIM performances of different multilevel settings. Herein, (m1, m2, m4) denotes the different number of SARB on the three pathways.
SettingSet5Set14B100(16, 0, 0)32.24/0.895428.64/0.783427.62/0.7380(16, 8, 0)32.26/0.895828.63/0.783427.62/0.7378(16, 4, 2)32.23/0.895228.63/0.783527.62/0.7379(8, 4, 2)32.17/0.894828.60/0.782227.59/0.7368(16, 8, 4)32.26/0.896128.65/0.783827.63/0.7381


### 4.1.2. Investigation on Spatial Attention

We also investigate the effectiveness of spatial attention. Table2 shows the performance comparisons on spatial attention. The model with spatial attention enjoys the SARB design proposed in the paper. The model without spatial attention holds the same block design as EDSR [4]. In the table, we can find that the spatial attention mechanism brings 0.04 dB, 0.02 dB, and 0.04 dB PSNR gain on Set5, Set14, and B100 datasets separately. Similarly, the model with spatial attention achieves 0.8961, 0.7838, and 0.7381 SSIM results on different benchmarks, which are all higher than the model without attention mechanism. In this point of view, attention mechanism is an effective component for better performance.Table 2
Investigation on PSNR/SSIM performances of spatial attention (SA) mechanism.
<!—Col Count:4F0E0AttentionSet5Set14B100w/o32.22/0.895428.63/0.783127.59/0.7373w32.26/0.896128.65/0.783827.63/0.7381


## 4.1.1. Investigation on the Multilevel Exploration

In this paper, we set different number of SARBs in the three scales of multilevel exploration module to keep the receptive field. To investigate the effectiveness of the number settings, we consider different combinations for comparison. Table1 shows the performances of different block settings. Herein, (m1, m2, m4) denotes the different number of SARBs for the scaling factor as 1×1, 2×2, and 4×4. In the table, we find that, with the increase of SARBs, the PSNR/SSIM results are higher. According to the third and forth rows, the PSNR of m1=16 on Set5 is 32.23 dB while the PSNR of m1=8 is 32.17 dB. The decrease of m1 leads to near 0.06 dB PSNR drop on Set5. However, the PSNR results of the second, third, and last rows on Set5 are 32.26 dB, 32.23 dB, and 32.26 dB separately, which shows the decrease of m2 and m4 leads to less than 0.03 dB drop on PSNR. In this point of view, m1 influences the performance more than blocks from other levels. We believe this is because the resolution of m1 is larger than other levels, and the computation complexity is much higher.Table 1
Investigation on PSNR/SSIM performances of different multilevel settings. Herein, (m1, m2, m4) denotes the different number of SARB on the three pathways.
SettingSet5Set14B100(16, 0, 0)32.24/0.895428.64/0.783427.62/0.7380(16, 8, 0)32.26/0.895828.63/0.783427.62/0.7378(16, 4, 2)32.23/0.895228.63/0.783527.62/0.7379(8, 4, 2)32.17/0.894828.60/0.782227.59/0.7368(16, 8, 4)32.26/0.896128.65/0.783827.63/0.7381


## 4.1.2. Investigation on Spatial Attention

We also investigate the effectiveness of spatial attention. Table2 shows the performance comparisons on spatial attention. The model with spatial attention enjoys the SARB design proposed in the paper. The model without spatial attention holds the same block design as EDSR [4]. In the table, we can find that the spatial attention mechanism brings 0.04 dB, 0.02 dB, and 0.04 dB PSNR gain on Set5, Set14, and B100 datasets separately. Similarly, the model with spatial attention achieves 0.8961, 0.7838, and 0.7381 SSIM results on different benchmarks, which are all higher than the model without attention mechanism. In this point of view, attention mechanism is an effective component for better performance.Table 2
Investigation on PSNR/SSIM performances of spatial attention (SA) mechanism.
<!—Col Count:4F0E0AttentionSet5Set14B100w/o32.22/0.895428.63/0.783127.59/0.7373w32.26/0.896128.65/0.783827.63/0.7381


## 4.2. Comparison with State-of-the-Art Methods

We mainly compare our model with several classical and recent works, including SRCNN [3], FSRCNN [29], VDSR [30], DRCN [31], LapSRN [8, 9], RAN [38], DNCL [51], FilterNet [37], MRFN [39], SeaNet [11], DEGREE [13], IRLP [40], FSN [52], and DSRLN [53]. Table 3 shows the PSNR/SSIM performance on five benchmarks with scaling factor ×2, ×3, and ×4.Table 3
Average PSNR/SSIM with degradation modelBI×2, ×3, and ×4 on five benchmarks. The best performances are shown in bold. The dash line means the paper does not report their performance.
ScaleModelSet5 [45]Set14 [46]B100 [47]Urban100 [48]Manga109 [49]PSNR/SSIMPSNR/SSIMPSNR/SSIMPSNR/SSIMPSNR/SSIM×2SRCNN [3]36.66/0.954232.42/0.906331.36/0.887929.50/0.894635.74/0.9661FSRCNN [29]37.00/0.955832.63/0.908831.53/0.892029.88/0.902036.67/0.9694VDSR [30]37.53/0.958733.03/0.912431.90/0.896030.76/0.914037.22/0.9729DRCN [31]37.63/0.958833.04/0.911831.85/0.894230.75/0.913337.63/0.9723LapSRN [9]37.52/0.959033.08/0.913031.80/0.895030.41/0.910037.27/0.9740RAN [38]37.58/0.959233.10/0.913331.92/0.8963——DNCL [51]37.65/0.959933.18/0.914131.97/0.897130.89/0.9158—FilterNet [37]37.86/0.961033.34/0.915032.09/0.899031.24/0.9200—MRFN [39]37.98/0.961133.41/0.915932.14/0.899731.45/0.922138.29/0.9759SeaNet-baseline [11]37.99/0.960733.60/0.917432.18/0.899532.08/0.927638.48/0.9768DEGREE [13]37.58/0.958733.06/0.912331.80/0.8974——IRLP [40]37.83/0.969233.18/0.924732.01/0.902130.86/0.919237.71/0.9861FSN [52]37.68/0.906533.51/0.918032.09/0.901531.68/0.9248—DSRLN [53]38.05/—33.56/—32.38/—31.79/——MFSR38.07/0.960833.69/0.919132.22/0.900232.35/0.930738.75/0.9768×3SRCNN [3]32.75/0.909029.28/0.820928.41/0.786326.24/0.798930.59/0.9107FSRCNN [29]33.16/0.914029.43/0.824228.53/0.791026.43/0.808030.98/0.9212VDSR [30]33.66/0.921329.77/0.831428.82/0.797627.14/0.827932.01/0.9310DRCN [31]33.82/0.922629.76/0.831128.80/0.796327.15/0.827632.31/0.9328DRRN [31]34.03/0.924429.96/0.834928.95/0.800427.53/0.837832.74/0.9390RAN [38]33.71/0.922329.84/0.832628.84/0.7981——DNCL [51]33.95/0.923229.93/0.834028.91/0.799527.27/0.8326—FilterNet [37]34.08/0.925030.03/0.837028.95/0.803027.55/0.8380—MRFN [39]34.21/0.926730.03/0.836328.99/0.802927.53/0.838932.82/0.9396SeaNet-baseline [11]34.36/0.928030.34/0.842829.09/0.805328.17/0.852733.40/0.9444DEGREE [13]33.76/0.921129.82/0.832628.74/0.7950——IRLP [40]34.22/0.943730.16/0.847229.21/0.816327.58/0.832732.61/0.9492DSRLN [53]34.56/—30.36/—29.29/—27.88/——MFSR34.49/0.928030.42/0.844229.16/0.806828.39/0.857733.72/0.9457×4SRCNN [3]30.48/0.862827.49/0.750326.90/0.710124.52/0.722127.66/0.8505FSRCNN [29]30.71/0.865727.59/0.753526.98/0.715024.62/0.728027.90/0.8517VDSR [30]31.35/0.883828.01/0.767427.29/0.725125.18/0.752428.83/0.8809DRCN [31]31.53/0.885428.02/0.767027.23/0.723325.14/0.751028.98/0.8816LapSRN [9]31.54/0.885028.19/0.772027.32/0.728025.21/0.756029.09/0.8845RAN [38]31.43/0.884728.09/0.769127.31/0.7260——DNCL [51]31.66/0.887128.23/0.771727.39/0.728225.36/0.7606—FilterNet [37]31.74/0.890028.27/0.773027.39/0.729025.53/0.7680—MRFN [39]31.90/0.891628.31/0.774627.43/0.730925.46/0.765429.57/0.8962SeaNet-baseline [11]32.18/0.894828.61/0.782227.57/0.735926.05/0.789630.44/0.9088DEGREE [13]31.47/0.883728.10/0.766927.20/0.7216——IRLP [40]31.94/0.902128.49/0.791027.70/0.747925.61/0.776429.49/0.9025FSN [52]32.10/0.895928.57/0.787427.53/0.743825.76/0.7817—MFSR32.26/0.896128.65/0.783827.63/0.738126.25/0.791930.62/0.9103In the table, we can find the proposed MFSR achieves the best performance on all testing benchmarks. It is worth noting that our model is mainly designed for structural information restoration. In this point of view, we achieve significant PSNR/SSIM improvement on the benchmark with plentiful lines and edges, such as Urban100 and Manga109. Particularly, our method achieves near 0.3 dB higher on Manga109 and Urban100 than other works with scaling factor×2 and ×3. When scaling factor is ×4, we achieve near 0.2 dB higher than other works. Urban100 is composed of high-resolution building images, and Manga109 is composed of comic cover pages. Thus, the high PSNR/SSIM performance demonstrates our effectiveness on structural information restoration.We also achieve superior performance than other edge preserving and multiscale SR works. FilterNet, DEGREE, and SeaNet are aiming at structure information preserving, and MRFN is a modern effective multiscale network for image SR. In Table3, our model gains more than 0.2 dB on PSNR, which is a significant improvement.Besides quantitative comparisons, we also compare the visualization performance with different methods. We mainly compare our method with LapSRN [9], MSRN [14], and CARN [10]. LapSRN is a representative progressive structure inspired by Laplacian pyramid, and MSRN is a state-of-the-art multiscale design. Thus, we compare with them to show the effectiveness of our multilevel design. Figure 3 shows the visualization comparison. In the first and second rows of the figure, our method recovers the lines on the building more accurately than MSRN and LapSRN. In the last row, our method can restore more correct small lines, while both LapSRN and MSRN fail to recover the straight lines.Figure 3
Visualization comparison on Urban100 dataset.


## 5. Conclusion

In this paper, we proposed a multilevel feature exploration network (MFSR) to restore the structural information for better visualization experience, which is designed with a UNet-like architecture for hierarchical exploration. To furthermore explore the structural information, spatial attention is considered in the MFSR for better performance. Spatial attention residual blocks (SARBs) are specially designed based on the proposed spatial attention mechanism. Experiment results show MFSR achieves better PSNR/SSIM performance than other multiscale and structural-preserving image SR works with scaling factor×2, ×3, and ×4. When the scaling factor is larger, there is near 0.2 dB PSNR improvement on the benchmarks with plentiful edges and lines, indicating the effectiveness on structural information restoration. The visual comparison result also demonstrates the superiority of MFSR on producing more accurate lines and edges.



---

*Source: 2014627-2022-02-07.xml*