# Lightweight Image Super-Resolution Based on Re-Parameterization and Self-Calibrated Convolution

Authors: Sufan Zhang; Xi Chen; Xingwei Huang
Journal: Computational Intelligence and Neuroscience
 (2022)

---

## Abstract

Image super-resolution technique can improve image quality by increasing image clarity, bringing a better user experience in real production scenarios. However, existing convolutional neural network methods usually have very deep network layers and a large number of parameters, which causes feature information to be lost as the network deepens, and models with a large numbers of parameters are not suitable for deploying on resource-constrained mobile devices. To address the above problems, we propose a novel lightweight image super-resolution network (RepSCN) based on re-parameterization and self-calibration convolution. Specifically, to reduce the computational cost while capturing more high-frequency details, we designed a re-parameterization distillation block (RepDB) and a self-calibrated distillation block (SCDB). They can improve the reconstruction results by aggregating the local distilled feature information under different receptive fields without introducing extra parameters. On the other hand, the positional information of the image is also crucial for super-resolution reconstruction. Nevertheless, existing lightweight SR methods mainly adopt the channel attention mechanism, which ignores the importance of positional information. Therefore, we introduce a lightweight coordinate attention mechanism (CAM) at the end of RepDB and SCDB to enhance the feature representation at both spatial and channel levels. Numerous experiments have shown that our network has better reconstruction performance with reduced parameters than other classical lightweight super-resolution models.


---

## Body

## 1. Introduction

Image super-resolution (SR) is an important technique used in computer vision tasks for image processing. It is a process that enables image reconstruction by learning a nonlinear mapping between a high-resolution (HR) image and its low-resolution (LR) counterpart. Since SR can increase the resolution while preserving as much as possible the image texture details, it plays a significant role in medical imaging [1], security and surveillance imaging [2], remote sensing images, and preprocessing of some advanced computer vision tasks [3,4]. In general, SR is an ill-posed problem because one LR image may correspond to multiple HR images [5].Recently, deep learning methods [6–11] have demonstrated great value in computer vision tasks. With the development of convolutional neural networks (CNNs), various CNN-based methods for image super-resolution have emerged and have achieved significant performance breakthroughs. Dong et al. [12] first proposed a network (SRCNN) consisting of three convolutional layers to learn the mapping of LR images to HR images. SRCNN upsamples the input image to a larger size before mapping it, increasing computational difficulty and slower model training. To address this issue, Dong et al. [13] introduced FSRCNN by upsampling the feature maps after the network output, which reduces a significant amount of computation and running time. After this, considering that a limited number of CNN layers cannot fully utilize the feature information of LR images, Kim et al. [14] presented the core idea that deepening and widening the network structure can lead to better performance and proposed a 20-layer network (VDSR). VDSR employs residual learning for the first time in the SR task, effectively speeding up the convergence of the network and avoiding vanishing or exploding gradient problems. EDSR [15] removes the batch normalization (BN) layer, which is disadvantageous to the SR task and improves the performance of the deep network. DRCN [16] and DRRN [17] use a recursive approach and parameter sharing strategy to reduce parameters further but increase the depth and width of the network. Zhang et al. [18] proposed a residual dense network (RDN) with dense skip connections and residual structure that can capture more contextual information to reconstruct images.Most of the above approaches are focused on improving performance by designing wider and deeper network. Still, as the network depth and width increase, the computational requirements and memory consumption also increase, making deep networks unsuitable for applications on mobile devices. Moreover, the deepening of the network causes the low-dimensional information of the image to fade away in the continuous nonlinear mapping, which is not conducive to the reconstruction of high-quality images. Therefore, Ahn et al. [19] proposed the lightweight cascading residual network CARN-M, which replaces the vanilla convolution with group convolution to speed up the network inference while reducing the number of parameters. Hui et al. [20] proposed an information distillation network (IDN), which improves the expressiveness of the network by combining the output relevant information with the redundant feature information obtained from distillation. However, these approaches reduce parameters at the cost of substantial performance and do not achieve a favorable trade-off. Then, Hui et al. improved the IDN by proposing the information multidistillation network (IMDN) [22]. IMDN designed an information multi-distillation block (IMDB) to progressively extract helpful information and aggregate two different features, achieving a good trade-off between network complexity and reconstruction quality. Nevertheless, although IMDN reduces the number of network parameters, it sacrifices part of the performance and cannot fully utilize the representational power of convolutional neural networks.To further improve the performance of the lightweight super-resolution model, we elaborately design a lightweight image super-resolution network based on re-parameterization and self-calibrated convolution, named RepSCN. Due to the excessive amount of deep neural network parameters and the computational resources required, we follow the shallow network structure of IMDN. Considering that information multidistillation block (IMDB) in IMDN employs channel splitting operation to extract features, making features inconsistent in the number of channels before and after convolution, inconvenient to benefit from residual learning using identity connection, and brings inflexibility to the network structure design. To improve the inferential and representational power of the network, we designed a re-parameterization distillation block (RepDB) and a self-calibrated distillation block (SCDB) to replace the IMDB. In the shallow feature extraction stage, the re-parameterization convolution (RepConv) in RepDB can collect more useful information than the standard convolution without introducing extra cost in the inference phase. In the deep feature extraction stage, the self-calibrated block (SCB) with a more extensive convolutional receptive field works as the feature extraction module of SCDB, which can help the network to generate feature maps containing rich high-frequency details. It is worth mentioning that RepDB and SCDB do not introduce additional parameters. On the other hand, lightweight models with a small number of parameters limit the SR performance improvement. The contrast-aware attention (CCA) layer in IMDN only learns feature mappings from the channel dimension, which is inefficient. Therefore, we choose to introduce a lightweight coordinate attention mechanism (CAM) [22] at the end of RepDB and SCDB, which captures not only cross-channel but also direction-aware and position-sensitive information. It can be demonstrated through comparative and ablation experiments that the proposed network structure achieves high SR quality while maintaining a modest model size.For this paper, the main contributions are as follows:(1)
We, for the first time, introduce the idea of structural re-parameterization in the distillation network and propose a re-parameterization distillation block (RepDB) to speed up the inference of the model while further improving the performance of SR reconstruction.(2)
We propose a self-calibrated distillation block (SCDB) with a self-calibrated block (SCB) to increase the receptive field of convolutional layers, which can generate features containing more contextual information without introducing additional parameters.(3)
Based on RepDB and SCDB, we design a lightweight and efficient SR network (RepSCN), which can fuse multiscale features under different receptive fields to enhance feature representation. A coordinate attention mechanism (CAM) is also introduced to further improve performance. Numerous experiments have demonstrated that RepSCN achieves comparable SR performance with state-of-the-art models while using a modest number of parameters.


## 2. Related Work

### 2.1. Singe-Image Super-Resolution

Recently, deep learning models have dramatically advanced the development of single-image super-resolution (SISR) tasks. Dong et al. [12] first combined deep learning techniques with SISR and proposed a network (SRCNN) consisting of three convolutional layers to learn the mapping between HR images and LR images in an end-to-end manner. Compared with SRCNN, ESPCN [23] designed a subpixel convolution layer to upsample the feature maps to the target size only at the end part of the network, greatly reducing the computational and storage complexity of the model. VDSR [14] introduced residual learning to tackle the problem of difficulty in training deep networks, and SR performance has improved with the deepening of the network. Later, to reduce the number of network parameters, Kim et al. [16] proposed a novel recursive network DRCN that achieves good performance using a parameter sharing strategy. LapSRN [24] used a pyramid framework to gradually enlarge the size of the input image patch. Tai et al. [25] proposed a persistent memory network (MemNet), which merges previous feature information using skip connections and solves the long-term dependency problem of deep models. EDSR [15] removes the BN layer in residual blocks, based on which RDN [18] introduces the dense connections, which reduces the parameters and improves the performance compared to EDSR. RCAN [26] introduced the channel attention mechanism to form a new residual structure. SRFBN [27] proposes a feedback mechanism to improve the representational power of the network.To better deploy the model on mobile devices, Ahn et al. [19] proposed CARN, which uses a recursive cascading mechanism to learn multilevel feature representations. IDN [20] effectively combines local long and short path features using group convolution. IMDN [27] proposes an information multiple-distillation block (IMDB) that extracts hierarchical features using channel split operation. Later, RFDN [28] improved IMDB with two parallel convolutional operations to separate the feature channels. FDIWN [29] proposed the Wide-residual Distillation Interaction Block (WDIB) to interact features with different scales. LBNet [30] integrates CNN and Transformer for building a more efficient model.In perceptual-driven methods, Ledig et al. [31] proposed a generative adversarial network (GAN)-based model SRGAN with a new perceptual loss function defined. Then, ESRGAN [32] introduced a Residual-in-Residual Dense Block (RRDB) on the top of SRGAN to enhance the visual quality further. However, although GAN-based models can generate finer texture details, there is always the problem of geometric structure distortion in the recovered images.


### 2.2. Attention Mechanism

The attention mechanism is a mechanism for resource allocation, and in computer vision tasks, it is used to find correlations between data and then focus on certain important features. Hu et al. [33] proposed a squeeze-and-excitation network (SENet) that learns the degree of dependency of each channel and redistributes channel feature information according to the interdependence between channels. After that, considering that channel attention can effectively help the network to learn the high-frequency information of images in SR tasks, Zhang et al. [26] introduced the channel attention mechanism into the residual learning-based block and proposed the RCAN model. The Efficient Channel Attention (ECA) module [34] improves SENet by generating channel attention via fast 1D convolution and is suitable for application in lightweight networks. Moreover, combining channel and spatial attention is an important development in attention mechanisms. CBAM [35] inferred attention weights along both spatial and channel dimensions. SRRAM [36], on the other hand, proposed a residual attention module (RAM) based on CBAM that is more suitable for SR tasks. Recently, Wang et al. [37] proposed a balanced attention mechanism (BAM) for SISR, combining the structure of MaxPool for spatial attention and AvgPool for channel attention, which is lightweight and efficient. Due to the effectiveness of the attention model, our network introduces a coordinate attention mechanism [22] in the local feature aggregation part to further enhance the network performance.


## 2.1. Singe-Image Super-Resolution

Recently, deep learning models have dramatically advanced the development of single-image super-resolution (SISR) tasks. Dong et al. [12] first combined deep learning techniques with SISR and proposed a network (SRCNN) consisting of three convolutional layers to learn the mapping between HR images and LR images in an end-to-end manner. Compared with SRCNN, ESPCN [23] designed a subpixel convolution layer to upsample the feature maps to the target size only at the end part of the network, greatly reducing the computational and storage complexity of the model. VDSR [14] introduced residual learning to tackle the problem of difficulty in training deep networks, and SR performance has improved with the deepening of the network. Later, to reduce the number of network parameters, Kim et al. [16] proposed a novel recursive network DRCN that achieves good performance using a parameter sharing strategy. LapSRN [24] used a pyramid framework to gradually enlarge the size of the input image patch. Tai et al. [25] proposed a persistent memory network (MemNet), which merges previous feature information using skip connections and solves the long-term dependency problem of deep models. EDSR [15] removes the BN layer in residual blocks, based on which RDN [18] introduces the dense connections, which reduces the parameters and improves the performance compared to EDSR. RCAN [26] introduced the channel attention mechanism to form a new residual structure. SRFBN [27] proposes a feedback mechanism to improve the representational power of the network.To better deploy the model on mobile devices, Ahn et al. [19] proposed CARN, which uses a recursive cascading mechanism to learn multilevel feature representations. IDN [20] effectively combines local long and short path features using group convolution. IMDN [27] proposes an information multiple-distillation block (IMDB) that extracts hierarchical features using channel split operation. Later, RFDN [28] improved IMDB with two parallel convolutional operations to separate the feature channels. FDIWN [29] proposed the Wide-residual Distillation Interaction Block (WDIB) to interact features with different scales. LBNet [30] integrates CNN and Transformer for building a more efficient model.In perceptual-driven methods, Ledig et al. [31] proposed a generative adversarial network (GAN)-based model SRGAN with a new perceptual loss function defined. Then, ESRGAN [32] introduced a Residual-in-Residual Dense Block (RRDB) on the top of SRGAN to enhance the visual quality further. However, although GAN-based models can generate finer texture details, there is always the problem of geometric structure distortion in the recovered images.


## 2.2. Attention Mechanism

The attention mechanism is a mechanism for resource allocation, and in computer vision tasks, it is used to find correlations between data and then focus on certain important features. Hu et al. [33] proposed a squeeze-and-excitation network (SENet) that learns the degree of dependency of each channel and redistributes channel feature information according to the interdependence between channels. After that, considering that channel attention can effectively help the network to learn the high-frequency information of images in SR tasks, Zhang et al. [26] introduced the channel attention mechanism into the residual learning-based block and proposed the RCAN model. The Efficient Channel Attention (ECA) module [34] improves SENet by generating channel attention via fast 1D convolution and is suitable for application in lightweight networks. Moreover, combining channel and spatial attention is an important development in attention mechanisms. CBAM [35] inferred attention weights along both spatial and channel dimensions. SRRAM [36], on the other hand, proposed a residual attention module (RAM) based on CBAM that is more suitable for SR tasks. Recently, Wang et al. [37] proposed a balanced attention mechanism (BAM) for SISR, combining the structure of MaxPool for spatial attention and AvgPool for channel attention, which is lightweight and efficient. Due to the effectiveness of the attention model, our network introduces a coordinate attention mechanism [22] in the local feature aggregation part to further enhance the network performance.


## 3. Proposed Methods

### 3.1. Network Framework

The overall framework of the proposed network is illustrated in Figure1, and our RepSCN consists of four parts: feature extraction module, nonlinear feature mapping module, feature fusion part, and reconstruction module. Algorithm 1 formulates the network forward step. Taking ILR and ISR as the input and output of the network, respectively, the feature extraction module is composed of a convolutional layer with 56 channels and 3 × 3 kernel size, which can be expressed as follows:(1)F0=fsILR,where fs· denotes the shallow feature extraction function. After that, the shallow feature F0 is fed to the nonlinear feature mapping module for extracting useful feature information, which consists of three re-parameterization distillation block (RepDB) and three self-calibrated distillation block (SCDB) stacked in a chain-like manner, as shown in the following equation:(2)Fi=fRepDBFi−1,i=1,…,n,Fj=fSCDBFj−1,j=n+1,…,n+m,where fRepDB· and fSCDB· denote the RepDB function and SCDB function, respectively. Fi−1 and Fi represent the input and output features of the i-th RepDB, and similarly, Fj−1 and Fj represent the input and output features of the j−n-th SCDB, respectively. All these intermediate features are aggregated together by a concatenation operation, and then a 1 × 1 and 3 × 3 convolutional layer is used to compress the feature channels and further refine the features. The feature fusion part can be formulated as follows:(3)Ff=ffcatF1,…,Fn,…,Fm,where cat· stands for the concatenation operation of channel dimension andff· is the 1 × 1 convolution and 3 × 3 convolution. Finally, the aggregated featureFfand shallow feature F0 are element-wise summed and upsampled to the target size by the upsampling reconstruction module as follows:Figure 1
Architecture of the proposed framework.Algorithm 1: RepSCN function.
Input: the low-resolution image: IL,R, the shallow feature extraction function: fs(.), the number of RepDB: n = 3, the number of SCDB: m=3, the RepDB function: fRepDB( ), the SCDB function: fSCDB( ), the feature fusion part: ff(cat( )), the reconstruction function: frec(.)Output: the super resolved image: ISR1:
the shallow feature F0 = fs(ILR)2:
//Calculate intermediate featuresFi and Fj3:
fori = 1,...,ndo4:
Fi = fRepDB(Fi−1)5:
end for6:
forj = n + 1,...,n + m do7:
Fj = fSCDB(Fj−1)8:
end for9:
the aggregated featureFf = ff (cat(F1,...,Fn,...,Fn + m))10:
//Reconstruct the super-resolved imageISR11:
ISR = frec (Ff + F0)12:
return ISR(4)ISR=frecFf+F0,where frec· denotes the reconstruction module, which upsamples the image using subpixel convolution to obtain the reconstructed SR image.We denote the training set asIHRt,ILRtt=1T, which contains T iterations of all LR-HR image pairs. The loss function of our RepSCN can be described as(5)ℒθ=1T∑t=1TIHRt−fRepSCNILRt1,where θ represents all learnable parameters in the network and fRepSCN· is our RepSCN function. ∥·∥1 means the L1 norm.


### 3.2. Re-Parameterization Distillation Block

As shown in Figure2, our re-parameterization distillation block (RepDB) consists of cascaded re-parameterization convolutions (RepConv), 1 × 1 convolution, and 3 × 3 convolution for reducing the number of feature channels and coordinate attention mechanism (CAM). The whole module utilizes residual connections to extract valuable features progressively.Figure 2
The left figure shows the architecture of our proposed reparameterization distillation block (RepDB). The right figure presents the reparameterization convolution (RepConv). Here, the number of input channels and the number of output channels both are set to 2, so a convolution layer has four matrices.In Figure2, inspired by RFDN [28], the feature distillation operation is implemented by a 1 × 1 convolution that compresses the feature channels at a rate of 0.5, and the re-parameterization convolution is utilized to refine the features little by little. Given an input feature Fin, the process of the i-th RepDB can be formulated as follows:(6)Fdistill_1i,Frefine_1i=D1iFini,R1iFini,Fdistill_2i,Frefine_2i=D2iFrefine_1i,R2iFrefine_1i,Fdistill_3i,Frefine_3i=D3iFrefine_2i,R3iFrefine_2i,Frefine_4i=R4iFrefine_3i,where Fdistill_ji and Frefine_ji represent the i-th distilled feature and refined feature of the j-th RepDB, respectively. R1i,R2i,R3i represent the three re-parameterization convolutional layers of the i-th RepDB, and R4i is a 3 × 3 convolutional layer to decrease the number of channels of the refined features. Rji represents the j-th 1 × 1 convolutional layer of the i-th RepDB. Then, the distilled features and the final refined feature are concatenated together and added to the input feature Fin to obtain the output feature Fout of the module. It can be expressed as follows:(7)Fout=fCAMfCcatFdistill1i,Fdistill2i,Fdistill3i,Frefine4i+Fin,where cat· is a concatenation operation in channel dimensions and fC· denotes a 1 × 1 convolution used to compress feature channels to the same size as the input features. fCAM· is the coordinate attention mechanism.Since the multibranch structure can lead to high-performance benefits for the network, inspired by RepVGG [38], we propose a re-parameterization convolution (RepConv) based on the RepVGG block but more applicable for SR tasks. Unlike the RepVGG block, as shown in Figure 2, RepConv removes the BN layer that is ineffective for the SR task. In the training phase, RepConv can be represented as y=gx+fx+x, where gx and fx correspond to 1 × 1 convolution and 3 × 3 convolution, respectively. Although we can obtain various receptive fields by employing different convolution kernels on different branches, the multibranch structure is slower during inference and has an increased memory occupation.To address the above problem, we transform the trained multipath model into a single-path model in the inference phase. The specific procedure is shown in Figure2. Given a re-parameterization convolution with the number of input channels and output channels both set to 2, the 1 × 1 convolution can be transformed into a 3 × 3 convolution by adding zero-padding. Setting the convolution kernel parameters of the current channel to 1 and the remaining channels to 0, the identity layer can also be transformed into the form of 1 × 1 convolution and further into 3 × 3 convolution. According to the principle of convolutional additivity, the convolutional layers in three branches are fused to form a new 3 × 3 convolution finally, and the bias of this 3 × 3 convolution is the sum of the bias of the previous 1 × 1 and 3 × 3 convolution. The experimental results show that the re-parameterization convolution can effectively improve the super-resolution reconstruction performance compared with the normal convolution.


### 3.3. Self-Calibrated Distillation Block

To recover more high-frequency details by using the low-frequency information of the image, we propose the self-calibrated distillation block (SCDB) to extract features in the deep network. As shown in Figure3, the overall structure of the self-calibrated distillation block is the same as the re-parameterization distillation block, except that the re-parameterization convolution of the feature refinement part is replaced by a self-calibrated block, which can enlarge the receptive field of convolution and thus obtain richer contextual information, which helps to generate realistic HR images.Figure 3
Architecture of our proposed self-calibrated distillation block (SCDB).Inspired by SCNet [39], we improved the SCConv in SCNet by removing its BN layers and embedding a global residual connection to alleviate the pressure during training. The proposed self-calibrated block is shown in Figure 4, where the input features Xn−1 are equally divided into Xn−1′ and Xn−1″ along the channel dimension by a 1 × 1 convolution. Xn−1′ and Xn−1″ are processed in two different branches, and the processed features are concatenated together and summed with Xn−1to obtain the final output Xn.Figure 4
Architecture of our proposed self-calibrated block (SCB).In the upper branch, we first perform an average pooling downsampling operation, a convolutional feature transformation operation, and a bilinear interpolation upsampling operation on the input featuresXn−1', and then we sum it with the input feature to generate the spatial attention feature map. The spatial attention feature map is fused with the transformed Xn−1′, and the output Xn' of this branch is obtained after a 3 × 3 convolution. The whole process can be expressed as follows:(8)Xn'=f3f2Xn−1'∗σfUpf1fDownXn−1'+Xn−1',wherefUp· andfDown·denote the mean pooling upsampling operation and the bilinear interpolation upsampling operation, respectively. f1·, f2·, and f3· are 3 × 3 convolutions, and σ· is the sigmoid function.In the lower branch, the input featureXn−1″ is fed into a 3 × 3 convolution layer to obtain the output Xn″. Xn″ is concatenated with the output Xn′ of the upper branch and summed with the feature Xn−1 containing a large amount of low-frequency information to generate the output Xn of the n-th self-calibrated block. This process can be formulated as follows:(9)Xn″=f4Xn−1″,Xn=catXn″,Xn′+Xn−1,where f4· is the 3 × 3 convolution and cat· represents the concatenation operation along the channel dimensions.


### 3.4. Coordinate Attention Mechanism

The channel attention mechanism (e.g., SE Block in SENet [33]) can effectively improve the reconstruction performance but ignores the locational information, which is critical to capturing the spatial structure of the image in the SR task. CBAM [35] combines channel attention and spatial attention by feedforward propagation, which is likely to cause the loss of locational information. Therefore, to augment the feature representations of the network, we introduce the coordinate attention mechanism [22], which embeds locational information into channel attention and can learn long-range dependencies between spatial locations. It is also well adapted for application to lightweight SR models.As shown in Figure5, given an input X∈ℝ56∗H∗W, 56 is the number of feature channels of the input X, and Hand Wdenote the height and width of the image, respectively. We utilize two spatially pooling kernels (H, 1) and (1, W) to encode each channel along the horizontal and vertical directions, respectively, and the output Y of the c -th channel at height h and width w can be formulated as follows:(10)Ychh=1W∑0≤,i≤WXch,i,Ycww=1H∑0≤j≤HXcj,w.Figure 5
Architecture of coordinate attention module (CAM).The above transformation obtains a pair of feature mapsYh∈ℝ56∗H∗1 and Yw∈ℝ56∗1∗W, and Yh andYware concatenated together and then fed to a 1 × 1 convolution layer to generate the intermediate feature F. It can be described as follows:(11)F=δfconvcatYh,Yw,where cat· represents the concatenation operation along the spatial dimension, fconv· represents the 1 × 1 convolution, and δ· is the h-swish activation function. Then, we split Finto Fhand Fw along the spatial dimension and obtain the attention weights Zh and Zw by two 1 × 1 convolution transformations. The above process can be formulated as follows:(12)Zh=σfhFh,Zw=σfwFw,where σ·is the sigmoid function. Finally, the input featuresXand attention weights Zhand Zware multiplied to generate the output feature Y, which can be formulated as follows:(13)Yci,j=Xci,j∗Zchi∗Zcwj.


## 3.1. Network Framework

The overall framework of the proposed network is illustrated in Figure1, and our RepSCN consists of four parts: feature extraction module, nonlinear feature mapping module, feature fusion part, and reconstruction module. Algorithm 1 formulates the network forward step. Taking ILR and ISR as the input and output of the network, respectively, the feature extraction module is composed of a convolutional layer with 56 channels and 3 × 3 kernel size, which can be expressed as follows:(1)F0=fsILR,where fs· denotes the shallow feature extraction function. After that, the shallow feature F0 is fed to the nonlinear feature mapping module for extracting useful feature information, which consists of three re-parameterization distillation block (RepDB) and three self-calibrated distillation block (SCDB) stacked in a chain-like manner, as shown in the following equation:(2)Fi=fRepDBFi−1,i=1,…,n,Fj=fSCDBFj−1,j=n+1,…,n+m,where fRepDB· and fSCDB· denote the RepDB function and SCDB function, respectively. Fi−1 and Fi represent the input and output features of the i-th RepDB, and similarly, Fj−1 and Fj represent the input and output features of the j−n-th SCDB, respectively. All these intermediate features are aggregated together by a concatenation operation, and then a 1 × 1 and 3 × 3 convolutional layer is used to compress the feature channels and further refine the features. The feature fusion part can be formulated as follows:(3)Ff=ffcatF1,…,Fn,…,Fm,where cat· stands for the concatenation operation of channel dimension andff· is the 1 × 1 convolution and 3 × 3 convolution. Finally, the aggregated featureFfand shallow feature F0 are element-wise summed and upsampled to the target size by the upsampling reconstruction module as follows:Figure 1
Architecture of the proposed framework.Algorithm 1: RepSCN function.
Input: the low-resolution image: IL,R, the shallow feature extraction function: fs(.), the number of RepDB: n = 3, the number of SCDB: m=3, the RepDB function: fRepDB( ), the SCDB function: fSCDB( ), the feature fusion part: ff(cat( )), the reconstruction function: frec(.)Output: the super resolved image: ISR1:
the shallow feature F0 = fs(ILR)2:
//Calculate intermediate featuresFi and Fj3:
fori = 1,...,ndo4:
Fi = fRepDB(Fi−1)5:
end for6:
forj = n + 1,...,n + m do7:
Fj = fSCDB(Fj−1)8:
end for9:
the aggregated featureFf = ff (cat(F1,...,Fn,...,Fn + m))10:
//Reconstruct the super-resolved imageISR11:
ISR = frec (Ff + F0)12:
return ISR(4)ISR=frecFf+F0,where frec· denotes the reconstruction module, which upsamples the image using subpixel convolution to obtain the reconstructed SR image.We denote the training set asIHRt,ILRtt=1T, which contains T iterations of all LR-HR image pairs. The loss function of our RepSCN can be described as(5)ℒθ=1T∑t=1TIHRt−fRepSCNILRt1,where θ represents all learnable parameters in the network and fRepSCN· is our RepSCN function. ∥·∥1 means the L1 norm.


## 3.2. Re-Parameterization Distillation Block

As shown in Figure2, our re-parameterization distillation block (RepDB) consists of cascaded re-parameterization convolutions (RepConv), 1 × 1 convolution, and 3 × 3 convolution for reducing the number of feature channels and coordinate attention mechanism (CAM). The whole module utilizes residual connections to extract valuable features progressively.Figure 2
The left figure shows the architecture of our proposed reparameterization distillation block (RepDB). The right figure presents the reparameterization convolution (RepConv). Here, the number of input channels and the number of output channels both are set to 2, so a convolution layer has four matrices.In Figure2, inspired by RFDN [28], the feature distillation operation is implemented by a 1 × 1 convolution that compresses the feature channels at a rate of 0.5, and the re-parameterization convolution is utilized to refine the features little by little. Given an input feature Fin, the process of the i-th RepDB can be formulated as follows:(6)Fdistill_1i,Frefine_1i=D1iFini,R1iFini,Fdistill_2i,Frefine_2i=D2iFrefine_1i,R2iFrefine_1i,Fdistill_3i,Frefine_3i=D3iFrefine_2i,R3iFrefine_2i,Frefine_4i=R4iFrefine_3i,where Fdistill_ji and Frefine_ji represent the i-th distilled feature and refined feature of the j-th RepDB, respectively. R1i,R2i,R3i represent the three re-parameterization convolutional layers of the i-th RepDB, and R4i is a 3 × 3 convolutional layer to decrease the number of channels of the refined features. Rji represents the j-th 1 × 1 convolutional layer of the i-th RepDB. Then, the distilled features and the final refined feature are concatenated together and added to the input feature Fin to obtain the output feature Fout of the module. It can be expressed as follows:(7)Fout=fCAMfCcatFdistill1i,Fdistill2i,Fdistill3i,Frefine4i+Fin,where cat· is a concatenation operation in channel dimensions and fC· denotes a 1 × 1 convolution used to compress feature channels to the same size as the input features. fCAM· is the coordinate attention mechanism.Since the multibranch structure can lead to high-performance benefits for the network, inspired by RepVGG [38], we propose a re-parameterization convolution (RepConv) based on the RepVGG block but more applicable for SR tasks. Unlike the RepVGG block, as shown in Figure 2, RepConv removes the BN layer that is ineffective for the SR task. In the training phase, RepConv can be represented as y=gx+fx+x, where gx and fx correspond to 1 × 1 convolution and 3 × 3 convolution, respectively. Although we can obtain various receptive fields by employing different convolution kernels on different branches, the multibranch structure is slower during inference and has an increased memory occupation.To address the above problem, we transform the trained multipath model into a single-path model in the inference phase. The specific procedure is shown in Figure2. Given a re-parameterization convolution with the number of input channels and output channels both set to 2, the 1 × 1 convolution can be transformed into a 3 × 3 convolution by adding zero-padding. Setting the convolution kernel parameters of the current channel to 1 and the remaining channels to 0, the identity layer can also be transformed into the form of 1 × 1 convolution and further into 3 × 3 convolution. According to the principle of convolutional additivity, the convolutional layers in three branches are fused to form a new 3 × 3 convolution finally, and the bias of this 3 × 3 convolution is the sum of the bias of the previous 1 × 1 and 3 × 3 convolution. The experimental results show that the re-parameterization convolution can effectively improve the super-resolution reconstruction performance compared with the normal convolution.


## 3.3. Self-Calibrated Distillation Block

To recover more high-frequency details by using the low-frequency information of the image, we propose the self-calibrated distillation block (SCDB) to extract features in the deep network. As shown in Figure3, the overall structure of the self-calibrated distillation block is the same as the re-parameterization distillation block, except that the re-parameterization convolution of the feature refinement part is replaced by a self-calibrated block, which can enlarge the receptive field of convolution and thus obtain richer contextual information, which helps to generate realistic HR images.Figure 3
Architecture of our proposed self-calibrated distillation block (SCDB).Inspired by SCNet [39], we improved the SCConv in SCNet by removing its BN layers and embedding a global residual connection to alleviate the pressure during training. The proposed self-calibrated block is shown in Figure 4, where the input features Xn−1 are equally divided into Xn−1′ and Xn−1″ along the channel dimension by a 1 × 1 convolution. Xn−1′ and Xn−1″ are processed in two different branches, and the processed features are concatenated together and summed with Xn−1to obtain the final output Xn.Figure 4
Architecture of our proposed self-calibrated block (SCB).In the upper branch, we first perform an average pooling downsampling operation, a convolutional feature transformation operation, and a bilinear interpolation upsampling operation on the input featuresXn−1', and then we sum it with the input feature to generate the spatial attention feature map. The spatial attention feature map is fused with the transformed Xn−1′, and the output Xn' of this branch is obtained after a 3 × 3 convolution. The whole process can be expressed as follows:(8)Xn'=f3f2Xn−1'∗σfUpf1fDownXn−1'+Xn−1',wherefUp· andfDown·denote the mean pooling upsampling operation and the bilinear interpolation upsampling operation, respectively. f1·, f2·, and f3· are 3 × 3 convolutions, and σ· is the sigmoid function.In the lower branch, the input featureXn−1″ is fed into a 3 × 3 convolution layer to obtain the output Xn″. Xn″ is concatenated with the output Xn′ of the upper branch and summed with the feature Xn−1 containing a large amount of low-frequency information to generate the output Xn of the n-th self-calibrated block. This process can be formulated as follows:(9)Xn″=f4Xn−1″,Xn=catXn″,Xn′+Xn−1,where f4· is the 3 × 3 convolution and cat· represents the concatenation operation along the channel dimensions.


## 3.4. Coordinate Attention Mechanism

The channel attention mechanism (e.g., SE Block in SENet [33]) can effectively improve the reconstruction performance but ignores the locational information, which is critical to capturing the spatial structure of the image in the SR task. CBAM [35] combines channel attention and spatial attention by feedforward propagation, which is likely to cause the loss of locational information. Therefore, to augment the feature representations of the network, we introduce the coordinate attention mechanism [22], which embeds locational information into channel attention and can learn long-range dependencies between spatial locations. It is also well adapted for application to lightweight SR models.As shown in Figure5, given an input X∈ℝ56∗H∗W, 56 is the number of feature channels of the input X, and Hand Wdenote the height and width of the image, respectively. We utilize two spatially pooling kernels (H, 1) and (1, W) to encode each channel along the horizontal and vertical directions, respectively, and the output Y of the c -th channel at height h and width w can be formulated as follows:(10)Ychh=1W∑0≤,i≤WXch,i,Ycww=1H∑0≤j≤HXcj,w.Figure 5
Architecture of coordinate attention module (CAM).The above transformation obtains a pair of feature mapsYh∈ℝ56∗H∗1 and Yw∈ℝ56∗1∗W, and Yh andYware concatenated together and then fed to a 1 × 1 convolution layer to generate the intermediate feature F. It can be described as follows:(11)F=δfconvcatYh,Yw,where cat· represents the concatenation operation along the spatial dimension, fconv· represents the 1 × 1 convolution, and δ· is the h-swish activation function. Then, we split Finto Fhand Fw along the spatial dimension and obtain the attention weights Zh and Zw by two 1 × 1 convolution transformations. The above process can be formulated as follows:(12)Zh=σfhFh,Zw=σfwFw,where σ·is the sigmoid function. Finally, the input featuresXand attention weights Zhand Zware multiplied to generate the output feature Y, which can be formulated as follows:(13)Yci,j=Xci,j∗Zchi∗Zcwj.


## 4. Experimental Results

### 4.1. Datasets and Evaluation Metrics

In our experiments, we use 800 high-quality RGB training images from the publicly available dataset DIV2K [40] as the training set to train our model. To test the performance of the model, we use four commonly used benchmark datasets: Set5 [41], Set14 [42], BSD100 [43], and Urban100 [44]. Set5, Set14, and BSD100 contain natural scene images, and Urban100 contains only urban scene images. In addition, we evaluated the SR results by calculating the widely used peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) [45] on the Y channel of transformed YCrbr space.


### 4.2. Implementation Details

In the data preprocessing stage, the training set is augmented with data by horizontal and vertical flipping and 90°, 180°, and 270° rotations. Besides, we convert the HR images to the LR training images using a bicubic interpolation downsampling operation in MATLAB.In the training stage, the input image patch size is 64 × 64. We optimize the model using ADAM algorithm with parameters set toβ1 = 0.9, β2 = 0.999, and ϵ=10−8. The batch size is set as 32, and the initial learning rate is set to 5e − 4 and is reduced by half every 200 epochs for a total of 1000 epochs.The ×2, ×3, and ×4 models are trained from scratch when training the final models. The entire network is implemented on the PyTorch framework with an NVIDIA RTX 3080 GPU.


### 4.3. Ablation Study

In this section, we will discuss the differences between our proposed method and IMDN [27]. As shown in Figure 6, we designed four blocks that each forms the body part of the network shown in Figure 1 in a stacked manner, based on which we conducted ablation experiments to validate the effectiveness of each of the proposed blocks. The structure of IMDB, the building block of IMDN, is shown in Figure 6(a). The channel splitting operation limits the number of channels before and after feature extraction, making it difficult to introduce identity connections. Therefore, we designed the feature distillation block (FDB) shown in Figure 6(b), which uses 1 × 1 convolution for channel reduction and is more flexible than IMDB.Figure 6
Structure diagrams of different building blocks used in the ablation study (a) IMDB, (b) FDB, (c) RepDB and (d) SCDB.
(a)(b)(c)(d)Table1 shows the results of the ablation experiments. From the first two rows of Table 1, we observe that FDB improves the performance compared to IMDB, and the PSNR value increases by 0.05 dB on the Urban100 dataset. In the second and third rows, the coordinate attention mechanism (CAM) achieves better experimental results than the contrast-aware channel attention layer (CCA Layer) with approximately the same number of parameters. By comparing the last four rows, we can see that both the re-parameterization convolution (RepConv) and the self-calibrated block (SCB) can bring significant performance benefits to the SR network after replacing the standard convolution, and the combination of the re-parameterization distillation block (RepDB) and the self-calibrated distillation block (SCDB) can maximize this benefit when used together. Thus, we choose 3RepDB + 3SCDB as the main structure of the proposed RepSCN. Comparing the first and last rows, we can observe that RepSCN outperforms IMDN on each data set with the same experimental settings, especially on the Urban100 dataset, where the PSNR value improves by 0.20db and the SSIM value improves by 0.0055.Table 1
Ablation studies of the effects of IMDB, FDB, RepDB, and SCDB with ×4 scale factor on test sets.
MethodScaleParams (K)Set5 PSNR/SSIMSet14 PSNR/SSIMBSD100 PSNR/SSIMUrban100 PSNR/SSIM6IMDB×471532.15/0.894028.56/0.780827.54/0.734825.97/0.78266FDB + CCA×473732.16/0.894328.58/0.781427.55/0.734826.02/0.78316FDB + CAM×474232.17/0.894628.59/0.781627.55/0.735526.05/0.78426RepDB×474232.20/0.894828.61/0.781827.56/0.735626.09/0.78626SCDB×480132.23/0.895128.61/0.781927.57/0.735926.11/0.78633RepDB+3SCDB×477232.22/0.895328.61/0.782027.58/0.736326.17/0.7881


### 4.4. Model Complexity Analysis

The number of model parameters is an important metric to evaluate the complexity of a lightweight network. To directly show that the proposed RepSCN method obtains better SR results with fewer parameters, we compared RepSCN with the following 13 lightweight networks on ×2 Urban100 dataset: SRCNN [12], FSRCNN [13], VDSR [14], DRCN [16], DRRN [17], LapSRN [24], MemNet [25], IDN [20], EDSR-baseline [15], SRMDNF [46], CARN [19], MADNet [47], and IMDN [27]. As shown in Figure 7, we can see that our RepSCN outperforms the other lightweight networks by a large margin while maintaining a modest model size.Figure 7
Trade-off between performance and number of parameters on Urban100 dataset with scale factor ×2.The number of Mult-Adds is another important metric to measure the complexity of the SR model. Figure8 shows the effect of Mult-Adds against PSNR for RepSCN and mainstream lightweight networks on the Urban100 ×2 dataset. We can find that RepSCN achieves better reconstruction results with fewer Mult-Add operations than other lightweight networks.Figure 8
PSNR performance versus Mult-Adds on Urban100 dataset with scale factor ×2.


### 4.5. Comparison with State-Of-The-Arts

To intuitively show the effectiveness and efficiency of the proposed RepSCN model, we compared our RepSCN with various lightweight SR methods, including SRCNN [12], FSRCNN [13], VDSR [14], DRCN [16], LapSRN [24], MemNet [25], IDN [20], EDSR-baseline [15], SRMDNF [46], CARN [19], MADNet [47], IMDN [27], MSICF [48], and MSWSR [49]. To make a fair comparison with the above methods, we train our models individually for the scaling factors of ×2, ×3, and ×4.Table 2
Average PSNR (dB)/SSIM values with the scale factors ×2, ×3, and ×4 on Set5, Set14, BSD100, and Urban100. The best performance is highlighted inred, and the second-best performance is highlighted in blue.
MethodScaleParamsSet5 PSNR/SSIMSet14 PSNR/SSIMBSD100 PSNR/SSIMUrban100 PSNR/SSIMSRCNN [12]×28K36.66/0.954232.45/0.906731.36/0.887929.50/0.8946FSRCNN [13]×213K37.00/0.955832.63/0.908831.53/0.892029.88/0.9020VDSR [14]×2666K37.53/0.958733.03/0.912431.90/0.896030.76/0.9140DRCN [16]×21774K37.63/0.958833.04/0.911831.85/0.894230.75/0.9133LapSRN [24]×2251K37.52/0.959132.99/0.912431.80/0.895230.41/0.9103DRRN [17]×2298K37.74/0.959133.23/0.913632.05/0.897331.23/0.9188MemNet [25]×2678K37.78/0.959733.28/0.914232.08/0.897831.31/0.9195IDN [20]×2553K37.83/0.960033.30/0.914832.08/0.898531.27/0.9196EDSR-baseline [15]×21370K37.99/0.960433.57/0.917532.16/0.899431.98/0.9272SRMDNF [46]×21511K37.79/0.960133.32/0.915932.05/0.898531.33/0.9204CARN [19]×21592K37.76/0.959033.52/0.916632.09/0.897831.92/0.9256IMDN [27]×2694K38.00/0.960533.63/0.917732.19/0.899632.17/0.9283MADNet [47]×2878K37.94/0.960433.46/0.916732.10/0.898831.74/0.9246MSICF [48]×24292K37.89/0.960533.41/0.915332.15/0.899231.47/0.9220MSWSR [49]×21228K37.49/0.958333.23/0.912331.88/0.892931.14/0.9169RepSCN(Ours)×2753K38.01/0.960633.70/0.919232.19/0.899932.36/0.9307SRCNN [12]×38K32.75/0.909029.30/0.821528.41/0.786326.24/0.7989FSRCNN [13]×313K33.18/0.914029.37/0.824028.53/0.791026.43/0.8080VDSR [14]×3666K33.66/0.921329.77/0.831428.82/0.797627.14/0.8279DRCN [16]×31774K33.82/0.922629.76/0.831128.80/0.796327.15/0.8276LapSRN [24]×3502K33.81/0.922029.79/0.832528.82/0.798027.07/0.8275DRRN [17]×3298K34.03/0.924429.96/0.834928.95/0.800427.53/0.8378MemNet [25]×3678K34.09/0.924830.00/0.835028.96/0.800127.56/0.8376IDN [20]×3553K34.11/0.925329.99/0.835428.95/0.801327.42/0.8359EDSR-baseline [15]×31555K34.37/0.927030.28/0.841729.09/0.805228.15/0.8527SRMDNF [46]×31528K34.12/0.925430.04/0.838228.97/0.802527.57/0.8398CARN [19]×31592K34.29/0.925530.29/0.840729.06/0.803428.06/0.8493IMDN [27]×3703K34.36/0.927030.32/0.841729.09/0.804628.17/0.8519MADNet [47]×3930K34.26/0.926230.29/0.841029.04/0.803327.91/0.8464MSICF [48]×34292K34.24/0.926630.09/0.837129.01/0.802427.69/0.8411MSWSR [49]×3—−/−−/−−/−−/−RepSCN(Ours)×3761K34.49/0.927730.38/0.843329.09/0.805428.30/0.8553SRCNN [12]×48K30.48/0.862827.50/0.751326.90/0.710124.52/0.7221FSRCNN [13]×413K30.72/0.866027.61/0.755026.98/0.715024.62/0.7280VDSR [14]×4666K31.35/0.883828.01/0.767427.29/0.725125.18/0.7524DRCN [16]×41774K31.53/0.885428.02/0.767027.23/0.723325.14/0.7510LapSRN [24]×4502K31.54/0.885228.09/0.770027.32/0.727525.21/0.7562DRRN [17]×4298K31.68/0.888828.21/0.772027.38/0.728425.44/0.7638MemNet [25]×4678K31.74/0.889328.26/0.772327.40/0.728125.50/0.7630IDN [20]×4553K31.82/0.890328.25/0.773027.41/0.729725.41/0.7632EDSR-baseline [15]×41518K32.09/0.893828.58/0.781327.57/0.735726.04/0.7849SRMDNF [46]×41552K31.96/0.892528.35/0.778727.49/0.733725.68/0.7731CARN [19]×41592K32.13/0.893728.60/0.780627.58/0.734926.07/0.7837IMDN [27]×4715K32.21/0.894828.58/0.781127.56/0.735326.04/0.7838MADNet [47]×41002K32.11/0.893928.52/0.779927.52/0.734025.89/0.7782MSICF [48]×44292K31.91/0.892328.35/0.775127.46/0.730825.64/0.7692MSWSR [49]×41228K32.01/0.891428.47/0.777627.48/0.731125.78/0.7744RepSCN(Ours)×4772K32.22/0.895328.61/0.782027.58/0.736326.17/0.7881Table 2 shows the number of parameters and objective evaluation metrics values for different models on the four benchmark datasets. We can see that the experimental results of IMDN are better than those of other methods except RepSCN at the upscaling factor of 2×. However, when the scale factor is set to ×4, the experimental results of IMDN are not as well compared with CARN and EDSR because CARN and EDSR with deeper and wider network structures can learn more feature information on large-scale datasets. Our proposed RepSCN has the same depth as IMDN, but by merging RepDB with SCDB, we obtain a larger receptive field and significantly enhance the representational power of the network. As a result, RepSCN achieves the best reconstruction performance on all four benchmark datasets with different upsampling factors, especially on the Set14 and Urban100 datasets, where the average PSNR values improve by 0.05 dB and 0.15 dB compared to IMDN, and the average SSIM values improve by 0.0013 and 0.0034 compared to IMDN, respectively. In summary, our RepSCN achieves the best trade-off against other lightweight methods in terms of performance and parameters.


### 4.6. Qualitative Comparisons

Figure9 shows the subjective visual effect on scale ×4 from the Urban100 dataset. As can be seen from images “Img044” and “Img085,” most of the compared methods suffer from severe blurring artifacts, in contrast to our RepSCN, which produces more accurate lines and reconstructs more high-frequency details. From the image “Img025,” we observe that other methods produce the wrong reconstruction of the image texture direction to some extent. RepSCN can recover clear and correct texture details and generate more realistic visual results.Figure 9
Visual comparisons of RepSCN with other SR methods on Urban100 dataset with scale factor ×4.


## 4.1. Datasets and Evaluation Metrics

In our experiments, we use 800 high-quality RGB training images from the publicly available dataset DIV2K [40] as the training set to train our model. To test the performance of the model, we use four commonly used benchmark datasets: Set5 [41], Set14 [42], BSD100 [43], and Urban100 [44]. Set5, Set14, and BSD100 contain natural scene images, and Urban100 contains only urban scene images. In addition, we evaluated the SR results by calculating the widely used peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) [45] on the Y channel of transformed YCrbr space.


## 4.2. Implementation Details

In the data preprocessing stage, the training set is augmented with data by horizontal and vertical flipping and 90°, 180°, and 270° rotations. Besides, we convert the HR images to the LR training images using a bicubic interpolation downsampling operation in MATLAB.In the training stage, the input image patch size is 64 × 64. We optimize the model using ADAM algorithm with parameters set toβ1 = 0.9, β2 = 0.999, and ϵ=10−8. The batch size is set as 32, and the initial learning rate is set to 5e − 4 and is reduced by half every 200 epochs for a total of 1000 epochs.The ×2, ×3, and ×4 models are trained from scratch when training the final models. The entire network is implemented on the PyTorch framework with an NVIDIA RTX 3080 GPU.


## 4.3. Ablation Study

In this section, we will discuss the differences between our proposed method and IMDN [27]. As shown in Figure 6, we designed four blocks that each forms the body part of the network shown in Figure 1 in a stacked manner, based on which we conducted ablation experiments to validate the effectiveness of each of the proposed blocks. The structure of IMDB, the building block of IMDN, is shown in Figure 6(a). The channel splitting operation limits the number of channels before and after feature extraction, making it difficult to introduce identity connections. Therefore, we designed the feature distillation block (FDB) shown in Figure 6(b), which uses 1 × 1 convolution for channel reduction and is more flexible than IMDB.Figure 6
Structure diagrams of different building blocks used in the ablation study (a) IMDB, (b) FDB, (c) RepDB and (d) SCDB.
(a)(b)(c)(d)Table1 shows the results of the ablation experiments. From the first two rows of Table 1, we observe that FDB improves the performance compared to IMDB, and the PSNR value increases by 0.05 dB on the Urban100 dataset. In the second and third rows, the coordinate attention mechanism (CAM) achieves better experimental results than the contrast-aware channel attention layer (CCA Layer) with approximately the same number of parameters. By comparing the last four rows, we can see that both the re-parameterization convolution (RepConv) and the self-calibrated block (SCB) can bring significant performance benefits to the SR network after replacing the standard convolution, and the combination of the re-parameterization distillation block (RepDB) and the self-calibrated distillation block (SCDB) can maximize this benefit when used together. Thus, we choose 3RepDB + 3SCDB as the main structure of the proposed RepSCN. Comparing the first and last rows, we can observe that RepSCN outperforms IMDN on each data set with the same experimental settings, especially on the Urban100 dataset, where the PSNR value improves by 0.20db and the SSIM value improves by 0.0055.Table 1
Ablation studies of the effects of IMDB, FDB, RepDB, and SCDB with ×4 scale factor on test sets.
MethodScaleParams (K)Set5 PSNR/SSIMSet14 PSNR/SSIMBSD100 PSNR/SSIMUrban100 PSNR/SSIM6IMDB×471532.15/0.894028.56/0.780827.54/0.734825.97/0.78266FDB + CCA×473732.16/0.894328.58/0.781427.55/0.734826.02/0.78316FDB + CAM×474232.17/0.894628.59/0.781627.55/0.735526.05/0.78426RepDB×474232.20/0.894828.61/0.781827.56/0.735626.09/0.78626SCDB×480132.23/0.895128.61/0.781927.57/0.735926.11/0.78633RepDB+3SCDB×477232.22/0.895328.61/0.782027.58/0.736326.17/0.7881


## 4.4. Model Complexity Analysis

The number of model parameters is an important metric to evaluate the complexity of a lightweight network. To directly show that the proposed RepSCN method obtains better SR results with fewer parameters, we compared RepSCN with the following 13 lightweight networks on ×2 Urban100 dataset: SRCNN [12], FSRCNN [13], VDSR [14], DRCN [16], DRRN [17], LapSRN [24], MemNet [25], IDN [20], EDSR-baseline [15], SRMDNF [46], CARN [19], MADNet [47], and IMDN [27]. As shown in Figure 7, we can see that our RepSCN outperforms the other lightweight networks by a large margin while maintaining a modest model size.Figure 7
Trade-off between performance and number of parameters on Urban100 dataset with scale factor ×2.The number of Mult-Adds is another important metric to measure the complexity of the SR model. Figure8 shows the effect of Mult-Adds against PSNR for RepSCN and mainstream lightweight networks on the Urban100 ×2 dataset. We can find that RepSCN achieves better reconstruction results with fewer Mult-Add operations than other lightweight networks.Figure 8
PSNR performance versus Mult-Adds on Urban100 dataset with scale factor ×2.


## 4.5. Comparison with State-Of-The-Arts

To intuitively show the effectiveness and efficiency of the proposed RepSCN model, we compared our RepSCN with various lightweight SR methods, including SRCNN [12], FSRCNN [13], VDSR [14], DRCN [16], LapSRN [24], MemNet [25], IDN [20], EDSR-baseline [15], SRMDNF [46], CARN [19], MADNet [47], IMDN [27], MSICF [48], and MSWSR [49]. To make a fair comparison with the above methods, we train our models individually for the scaling factors of ×2, ×3, and ×4.Table 2
Average PSNR (dB)/SSIM values with the scale factors ×2, ×3, and ×4 on Set5, Set14, BSD100, and Urban100. The best performance is highlighted inred, and the second-best performance is highlighted in blue.
MethodScaleParamsSet5 PSNR/SSIMSet14 PSNR/SSIMBSD100 PSNR/SSIMUrban100 PSNR/SSIMSRCNN [12]×28K36.66/0.954232.45/0.906731.36/0.887929.50/0.8946FSRCNN [13]×213K37.00/0.955832.63/0.908831.53/0.892029.88/0.9020VDSR [14]×2666K37.53/0.958733.03/0.912431.90/0.896030.76/0.9140DRCN [16]×21774K37.63/0.958833.04/0.911831.85/0.894230.75/0.9133LapSRN [24]×2251K37.52/0.959132.99/0.912431.80/0.895230.41/0.9103DRRN [17]×2298K37.74/0.959133.23/0.913632.05/0.897331.23/0.9188MemNet [25]×2678K37.78/0.959733.28/0.914232.08/0.897831.31/0.9195IDN [20]×2553K37.83/0.960033.30/0.914832.08/0.898531.27/0.9196EDSR-baseline [15]×21370K37.99/0.960433.57/0.917532.16/0.899431.98/0.9272SRMDNF [46]×21511K37.79/0.960133.32/0.915932.05/0.898531.33/0.9204CARN [19]×21592K37.76/0.959033.52/0.916632.09/0.897831.92/0.9256IMDN [27]×2694K38.00/0.960533.63/0.917732.19/0.899632.17/0.9283MADNet [47]×2878K37.94/0.960433.46/0.916732.10/0.898831.74/0.9246MSICF [48]×24292K37.89/0.960533.41/0.915332.15/0.899231.47/0.9220MSWSR [49]×21228K37.49/0.958333.23/0.912331.88/0.892931.14/0.9169RepSCN(Ours)×2753K38.01/0.960633.70/0.919232.19/0.899932.36/0.9307SRCNN [12]×38K32.75/0.909029.30/0.821528.41/0.786326.24/0.7989FSRCNN [13]×313K33.18/0.914029.37/0.824028.53/0.791026.43/0.8080VDSR [14]×3666K33.66/0.921329.77/0.831428.82/0.797627.14/0.8279DRCN [16]×31774K33.82/0.922629.76/0.831128.80/0.796327.15/0.8276LapSRN [24]×3502K33.81/0.922029.79/0.832528.82/0.798027.07/0.8275DRRN [17]×3298K34.03/0.924429.96/0.834928.95/0.800427.53/0.8378MemNet [25]×3678K34.09/0.924830.00/0.835028.96/0.800127.56/0.8376IDN [20]×3553K34.11/0.925329.99/0.835428.95/0.801327.42/0.8359EDSR-baseline [15]×31555K34.37/0.927030.28/0.841729.09/0.805228.15/0.8527SRMDNF [46]×31528K34.12/0.925430.04/0.838228.97/0.802527.57/0.8398CARN [19]×31592K34.29/0.925530.29/0.840729.06/0.803428.06/0.8493IMDN [27]×3703K34.36/0.927030.32/0.841729.09/0.804628.17/0.8519MADNet [47]×3930K34.26/0.926230.29/0.841029.04/0.803327.91/0.8464MSICF [48]×34292K34.24/0.926630.09/0.837129.01/0.802427.69/0.8411MSWSR [49]×3—−/−−/−−/−−/−RepSCN(Ours)×3761K34.49/0.927730.38/0.843329.09/0.805428.30/0.8553SRCNN [12]×48K30.48/0.862827.50/0.751326.90/0.710124.52/0.7221FSRCNN [13]×413K30.72/0.866027.61/0.755026.98/0.715024.62/0.7280VDSR [14]×4666K31.35/0.883828.01/0.767427.29/0.725125.18/0.7524DRCN [16]×41774K31.53/0.885428.02/0.767027.23/0.723325.14/0.7510LapSRN [24]×4502K31.54/0.885228.09/0.770027.32/0.727525.21/0.7562DRRN [17]×4298K31.68/0.888828.21/0.772027.38/0.728425.44/0.7638MemNet [25]×4678K31.74/0.889328.26/0.772327.40/0.728125.50/0.7630IDN [20]×4553K31.82/0.890328.25/0.773027.41/0.729725.41/0.7632EDSR-baseline [15]×41518K32.09/0.893828.58/0.781327.57/0.735726.04/0.7849SRMDNF [46]×41552K31.96/0.892528.35/0.778727.49/0.733725.68/0.7731CARN [19]×41592K32.13/0.893728.60/0.780627.58/0.734926.07/0.7837IMDN [27]×4715K32.21/0.894828.58/0.781127.56/0.735326.04/0.7838MADNet [47]×41002K32.11/0.893928.52/0.779927.52/0.734025.89/0.7782MSICF [48]×44292K31.91/0.892328.35/0.775127.46/0.730825.64/0.7692MSWSR [49]×41228K32.01/0.891428.47/0.777627.48/0.731125.78/0.7744RepSCN(Ours)×4772K32.22/0.895328.61/0.782027.58/0.736326.17/0.7881Table 2 shows the number of parameters and objective evaluation metrics values for different models on the four benchmark datasets. We can see that the experimental results of IMDN are better than those of other methods except RepSCN at the upscaling factor of 2×. However, when the scale factor is set to ×4, the experimental results of IMDN are not as well compared with CARN and EDSR because CARN and EDSR with deeper and wider network structures can learn more feature information on large-scale datasets. Our proposed RepSCN has the same depth as IMDN, but by merging RepDB with SCDB, we obtain a larger receptive field and significantly enhance the representational power of the network. As a result, RepSCN achieves the best reconstruction performance on all four benchmark datasets with different upsampling factors, especially on the Set14 and Urban100 datasets, where the average PSNR values improve by 0.05 dB and 0.15 dB compared to IMDN, and the average SSIM values improve by 0.0013 and 0.0034 compared to IMDN, respectively. In summary, our RepSCN achieves the best trade-off against other lightweight methods in terms of performance and parameters.


## 4.6. Qualitative Comparisons

Figure9 shows the subjective visual effect on scale ×4 from the Urban100 dataset. As can be seen from images “Img044” and “Img085,” most of the compared methods suffer from severe blurring artifacts, in contrast to our RepSCN, which produces more accurate lines and reconstructs more high-frequency details. From the image “Img025,” we observe that other methods produce the wrong reconstruction of the image texture direction to some extent. RepSCN can recover clear and correct texture details and generate more realistic visual results.Figure 9
Visual comparisons of RepSCN with other SR methods on Urban100 dataset with scale factor ×4.


## 5. Conclusion

In this work, we propose a lightweight convolutional neural network based on re-parameterization and self-calibrated convolution for SISR. Specifically, we design the re-parameterization distillation module (RepDB) and the self-calibrated distillation module (SCDB) as the building blocks of the SR model. RepDB and SCDB can aggregate local features from different stages to obtain a more powerful feature representation, and re-parameterization convolution (RepConv) and self-calibrated blocks (SCB) are used to extract features at a fine-grained level. Moreover, we introduce the coordinate attention mechanism (CAM) to obtain long-distance dependencies between different locations, which is beneficial to recover more high-frequency details. Numerous experiments show that our method can achieve comparable performance with state-of-the-art lightweight networks. Our RepSCN reconstructs high-quality images with a small number of parameters.In the future, we will explore more efficient re-parameterization strategies and use a single-branch network structure to accelerate model inference, which allows our models to be embedded in edge devices. In addition, we will also apply our approach to other image processing fields such as hyperspectral remote sensing and medical imaging.



---

*Source: 8628402-2022-09-26.xml*