# Cascading and Residual Connected Network for Single Image Superresolution

Authors: Kai Huang; Wenhao Wang; Cheng Pang; Rushi Lan; Ji Li; Xiaonan Luo
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

Convolution neural networks facilitate the significant process of single image super-resolution (SISR). However, most of the existing CNN-based models suffer from numerous parameters and excessively deeper structures. Moreover, these models relying on in-depth features commonly ignore the hints of low-level features, resulting in poor performance. This paper demonstrates an intriguing network for SISR with cascading and residual connections (CASR), which alleviates these problems by extracting features in a small net called head module via the strategies based on the depthwise separable convolution and deformable convolution. Moreover, we also include a cascading residual block (CAS-Block) for the upsampling process, which benefits the gradient propagation and feature learning while easing the model training. Extensive experiments conducted on four benchmark datasets demonstrate that the proposed method is superior to the latest SISR methods in terms of quantitative indicators and realistic visual effects.


---

## Body

## 1. Introduction

Superresolution (SR) image reconstruction is widely used in various applications, such as military surveillance, medical diagnostics [1, 2], remote sensing [3], and video streaming [4, 5]. Single image superresolution (SISR) is aimed at reconstructing a high-resolution (HR) image from its counterpart low-resolution (LR) input, which is an essential and classic task in computer vision. Recently, high-resolution (HR) demand images have boosted. However, physical constraints limit the conduction of high-resolution pictures. A series of successful works brought attention to the research community.The task of recovering HR imagesISR from its counterpart (LR) version ILR is ill-posed. Researchers have made many efforts to this task and invented numerous algorithms, including interpolation-based, reconstruction-based, and learning-based methods [1], respectively.The traditional SISR algorithms, for instance, bicubic interpolation [6], are high-speed while suffering from poor accuracy. It is easy to fail in practice. To limit possible solving space, researchers present more advanced methods, reconstruction-based algorithms [7, 8], by introducing available prior knowledge. These algorithms may restore clear details (i.e., texture details), but extensive experiments show that they degrade sharply when the scale factors increase; subsequently, the algorithms with learnable parameters [9] are proposed to analyze relationships between the ILR image and their counterpart IHR image by training concrete instances [10, 11]. Although such learning-based methods perform very well, the time-consuming optimization problems they involve are very tricky.In recent years, CNNs have been introduced to facilitate the progress of the SISR field because of their excellent feature representation ability. Dong et al. [12] were the first to propose a three-stage convolutional network to solve the SISR problem, which has become a milestone in this field. Since then, the research community has set out to design more complex networks to improve performance. EDSR, a very large network with residual blocks, was presented by Lim et al. [13] and achieved satisfactory performance in both PNSR and SSIM [14]. However, these state-of-the-art methods still have some limitations:
(1)
The state-of-the-art (SotA) models [13] mainly improve the performance by considerably growing the depth and width of the proposed methods. Therefore, massive parameters and increasing resource-consuming problems are inevitable(2)
Many progressive models do not fully take advantage of the hierarchical information from the primary LR images, which are essential for improving visual performancesTo address these shortcomings, we present a model named CASR, exploring two separate strategies to functionally extract features for precise SISR. Figure1 shows the ×4 SR results of our proposed model on dataset DIV2K [15]. First, we propose a small but functional depthwise separable convolution network named head module aimed at more systematic feature extraction.Figure 1
×4 superresolution results of our proposed SR model on dataset DIV2K.
(a)(b)Second, we present another cascaded residual network (CAS-Block) for better feature and gradient propagation. Our proposed method combines features from excessive layers at both the regional and global levels with such architecture. Moreover, a stacking broader local residual connection is applied to exploit the feature of theILR and let the vast low-level mappings be transmitted. This schema unites nonlocal actions to capture remote spatial features from former inputs.As the crucial integrant of the presented method, the CAS-Block includes six subtrunks, each of which consists of two convolutional layers and a nonlinear activation pReLU. Because using the activation function in bottlenecks does affect the performance, we take advantage of channels before the pReLU layer to construct the inverse residual block, resulting in performance improvement.The three main contributions of this article are summarized as follows.(1)
We propose a head module applied with a series of depthwise separable convolution operations for feature extraction. In addition, we replace all existing conventional convolution operations with deformable convolution layers in the module. At the same time, in order to effectively retain the features, we extend the low-dimensional representation to high-dimensional before passing the activation function. This maintains a balance between a large number of parameters and excellent performance(2)
In order to effectually raise feature fusion and gradient propagation, we introduce a cascaded block called CAS-Block. This mechanism allows our network to combine features from diverse layers. Furthermore, such a structure is also used to construct the network and promote its functional expression(3)
We utilize theL1 with the addition of total variance loss LTV instead of the traditional sole L1 loss function, which significantly improves the quality of the reconstruction image ISR. Meanwhile, to obtain better optimization weights, we explored various parameter settings


## 2. Related Works

### 2.1. SISR Using Deep CNNs

In the field of superresolution, compared to conventional image restoration methods, CNN-based models have a stronger feature expression ability and have achieved great success. Dong et al. [12] first proposed an algorithm named SRCNN, which is an end-to-end algorithm based on CNN. It consists of three convolutional layers, and its performance is impressive compared to traditional methods (i.e., sparse coding [7] and bicubic interpolation [6]). Later, the research community designed more intricate CNN architectures and developed more profound networks. For example, in order to grow the depth of the network, VDSR [16] introduced residual learning, and the verification experiment proved that this strategy heightens the SR image qualities and promotes convergence. DRCN [17] uses deep recursion to construct a neural network and uses the same convolution kernel 16 times in the reference network, effectually dropping the number of parameters. He et al. [18], inspired by the ordinary differential equation (ODE), propose an intriguing network named OISR, which provides a new understanding of network designs. It is worth noting that most of these latest methods use interpolated images for input, which will not only cause the details to be too smooth but also boost additional computational cost and time consumption.


### 2.2. Skip Connection

ResNet [19] was the first to adopt the concept of skip connection, and then, the idea was extended to various computer vision tasks, such as image restoration [20] and semantic segmentation [21]. Since it is difficult for ordinary SR networks to construct extremely deep networks, employing various skip connections avoids the gradient vanishing trap and boosts performance. The strategy is roughly divided into two categories, namely, global or local residual connections and dense connections.


#### 2.2.1. Global or Local Residual Connections

In image restoration tasks, LR images are closely related to HR ones. Obtaining the residual feature maps among the image’s pixels can learn the absent high-frequency detailed information. VDSR is the first residual model for superresolution. Extensive experiments have proved that residual learning can enhance reconstruction performance and promote convergence speed. Therefore, this method has been widely used in various computer vision tasks [22].


#### 2.2.2. Dense Connections

A dense connection allows the current layer to connect with all former layers, and the architecture provides more intriguing effects on restoring high-resolution patterns. DenseNet [23] first presents the dense connection in the SR field, starting from the features, achieving better results with fewer parameters through the extreme use of the features.Traditional neural networks are basically unidirectional propagations, and the signals received in the later layers are very weak. To solve this problem, MemNet [20] stacks memory blocks and adds dense connections among each block, which is called the long-term memory model. Such architecture reduces the weight of the entire network, facilitates convergence, and deepens the network.RDN [24] uses a similar architecture, but MemNet does not take all the intermediate feature information, while RDN applies global residual learning to use all of them.Jiang et al. [25] proposed a hierarchical dense network (HDRN) in 2019, which can effectively establish realistic mapping relationships between the LR and HR image, promoting information interaction and representation.Different from the above models, CARN also uses a cascade mechanism at the local and global levels to integrate features from multiple layers, which can reflect input representations at different levels for receiving more information [26]. Haris et al. [27] proposed D-DBPN, which connects the features of the up- and downsampling stages and improves the SR result.


### 2.3. Depthwise Separable Convolution

The cross-channel correlation and spatial correlation of the convolutional layer can be decoupled, and they can be mapped separately to achieve better results. Some lightweight networks, such as MobileNet [28], apply depthwise separable convolution, which is a combination of depthwise (DW) and pointwise (PW) to extract feature maps. Compared with the conventional convolution operation, the number of parameters and operation cost are relatively small. In Figure 2, we use the separable convolution operation in the depth direction in the head module.Figure 2
The architecture of head module is abundantly applied with depthwise separable convolution and deformable convolution operations.⊕ stands for the element-wise additional operation.


### 2.4. Multiscale Learning

So as to utilize computing resources more efficiently and extract more features under the same amount of calculation, Szegedy et al. [29] present the inception module. There are two main contributions of the inception structure: one is to use 1×1 convolution to perform dimensionality reduction; the other is to simultaneously perform convolution and reaggregation on multiple sizes. Inspired by [29] and [30], MSRB [31] was proposed. Multiscale feature fusion and local residual learning can be applied to adaptively detect images of different scales with different sizes of convolution kernel features. The results show that performing different kernel operations can provide better extraction capabilities. However, this method cannot expand more receptive fields and cannot generate more detailed structural information.


### 2.5. Deformable Convolution

Conventional convolution kernels are usually of fixed size (for example,3×3,5×5,and7×7.). The biggest problem with this convolution kernel is that it has poor adaptability to unknown changes and weak generalization ability. In order to solve the object space deformation problem, deformable convolution [32] is proposed to heighten the transformation modeling ability of CNN. Deformable convolution is based on traditional convolution, adding the direction vector of the adjustment convolution kernel to make the shape of the convolution kernel closer to the feature.


### 2.6. Real-World Image Superresolution

In real-world image restoration scenarios, lacking corresponding high-quality references usually conduct poor experimental results. We additionally introduce the naturalness image quality evaluator (NIQE) [33] and Perceptual Index (PI) [22] to perform the evaluation. In fact, these indicators can sensitively reflect content sharpness, detail contrast, and texture diversity. These evaluation indexes have a high consistency with the subjective quality and can effectively reflect the visual quality of images without reference. In particular, the smaller values of NIQE/PI indicate better perceptual quality and clearer content. We intend to apply it to the DRIVE [34] dataset to estimate the restoration capability of the proposed method.


## 2.1. SISR Using Deep CNNs

In the field of superresolution, compared to conventional image restoration methods, CNN-based models have a stronger feature expression ability and have achieved great success. Dong et al. [12] first proposed an algorithm named SRCNN, which is an end-to-end algorithm based on CNN. It consists of three convolutional layers, and its performance is impressive compared to traditional methods (i.e., sparse coding [7] and bicubic interpolation [6]). Later, the research community designed more intricate CNN architectures and developed more profound networks. For example, in order to grow the depth of the network, VDSR [16] introduced residual learning, and the verification experiment proved that this strategy heightens the SR image qualities and promotes convergence. DRCN [17] uses deep recursion to construct a neural network and uses the same convolution kernel 16 times in the reference network, effectually dropping the number of parameters. He et al. [18], inspired by the ordinary differential equation (ODE), propose an intriguing network named OISR, which provides a new understanding of network designs. It is worth noting that most of these latest methods use interpolated images for input, which will not only cause the details to be too smooth but also boost additional computational cost and time consumption.


## 2.2. Skip Connection

ResNet [19] was the first to adopt the concept of skip connection, and then, the idea was extended to various computer vision tasks, such as image restoration [20] and semantic segmentation [21]. Since it is difficult for ordinary SR networks to construct extremely deep networks, employing various skip connections avoids the gradient vanishing trap and boosts performance. The strategy is roughly divided into two categories, namely, global or local residual connections and dense connections.


### 2.2.1. Global or Local Residual Connections

In image restoration tasks, LR images are closely related to HR ones. Obtaining the residual feature maps among the image’s pixels can learn the absent high-frequency detailed information. VDSR is the first residual model for superresolution. Extensive experiments have proved that residual learning can enhance reconstruction performance and promote convergence speed. Therefore, this method has been widely used in various computer vision tasks [22].


### 2.2.2. Dense Connections

A dense connection allows the current layer to connect with all former layers, and the architecture provides more intriguing effects on restoring high-resolution patterns. DenseNet [23] first presents the dense connection in the SR field, starting from the features, achieving better results with fewer parameters through the extreme use of the features.Traditional neural networks are basically unidirectional propagations, and the signals received in the later layers are very weak. To solve this problem, MemNet [20] stacks memory blocks and adds dense connections among each block, which is called the long-term memory model. Such architecture reduces the weight of the entire network, facilitates convergence, and deepens the network.RDN [24] uses a similar architecture, but MemNet does not take all the intermediate feature information, while RDN applies global residual learning to use all of them.Jiang et al. [25] proposed a hierarchical dense network (HDRN) in 2019, which can effectively establish realistic mapping relationships between the LR and HR image, promoting information interaction and representation.Different from the above models, CARN also uses a cascade mechanism at the local and global levels to integrate features from multiple layers, which can reflect input representations at different levels for receiving more information [26]. Haris et al. [27] proposed D-DBPN, which connects the features of the up- and downsampling stages and improves the SR result.


## 2.2.1. Global or Local Residual Connections

In image restoration tasks, LR images are closely related to HR ones. Obtaining the residual feature maps among the image’s pixels can learn the absent high-frequency detailed information. VDSR is the first residual model for superresolution. Extensive experiments have proved that residual learning can enhance reconstruction performance and promote convergence speed. Therefore, this method has been widely used in various computer vision tasks [22].


## 2.2.2. Dense Connections

A dense connection allows the current layer to connect with all former layers, and the architecture provides more intriguing effects on restoring high-resolution patterns. DenseNet [23] first presents the dense connection in the SR field, starting from the features, achieving better results with fewer parameters through the extreme use of the features.Traditional neural networks are basically unidirectional propagations, and the signals received in the later layers are very weak. To solve this problem, MemNet [20] stacks memory blocks and adds dense connections among each block, which is called the long-term memory model. Such architecture reduces the weight of the entire network, facilitates convergence, and deepens the network.RDN [24] uses a similar architecture, but MemNet does not take all the intermediate feature information, while RDN applies global residual learning to use all of them.Jiang et al. [25] proposed a hierarchical dense network (HDRN) in 2019, which can effectively establish realistic mapping relationships between the LR and HR image, promoting information interaction and representation.Different from the above models, CARN also uses a cascade mechanism at the local and global levels to integrate features from multiple layers, which can reflect input representations at different levels for receiving more information [26]. Haris et al. [27] proposed D-DBPN, which connects the features of the up- and downsampling stages and improves the SR result.


## 2.3. Depthwise Separable Convolution

The cross-channel correlation and spatial correlation of the convolutional layer can be decoupled, and they can be mapped separately to achieve better results. Some lightweight networks, such as MobileNet [28], apply depthwise separable convolution, which is a combination of depthwise (DW) and pointwise (PW) to extract feature maps. Compared with the conventional convolution operation, the number of parameters and operation cost are relatively small. In Figure 2, we use the separable convolution operation in the depth direction in the head module.Figure 2
The architecture of head module is abundantly applied with depthwise separable convolution and deformable convolution operations.⊕ stands for the element-wise additional operation.


## 2.4. Multiscale Learning

So as to utilize computing resources more efficiently and extract more features under the same amount of calculation, Szegedy et al. [29] present the inception module. There are two main contributions of the inception structure: one is to use 1×1 convolution to perform dimensionality reduction; the other is to simultaneously perform convolution and reaggregation on multiple sizes. Inspired by [29] and [30], MSRB [31] was proposed. Multiscale feature fusion and local residual learning can be applied to adaptively detect images of different scales with different sizes of convolution kernel features. The results show that performing different kernel operations can provide better extraction capabilities. However, this method cannot expand more receptive fields and cannot generate more detailed structural information.


## 2.5. Deformable Convolution

Conventional convolution kernels are usually of fixed size (for example,3×3,5×5,and7×7.). The biggest problem with this convolution kernel is that it has poor adaptability to unknown changes and weak generalization ability. In order to solve the object space deformation problem, deformable convolution [32] is proposed to heighten the transformation modeling ability of CNN. Deformable convolution is based on traditional convolution, adding the direction vector of the adjustment convolution kernel to make the shape of the convolution kernel closer to the feature.


## 2.6. Real-World Image Superresolution

In real-world image restoration scenarios, lacking corresponding high-quality references usually conduct poor experimental results. We additionally introduce the naturalness image quality evaluator (NIQE) [33] and Perceptual Index (PI) [22] to perform the evaluation. In fact, these indicators can sensitively reflect content sharpness, detail contrast, and texture diversity. These evaluation indexes have a high consistency with the subjective quality and can effectively reflect the visual quality of images without reference. In particular, the smaller values of NIQE/PI indicate better perceptual quality and clearer content. We intend to apply it to the DRIVE [34] dataset to estimate the restoration capability of the proposed method.


## 3. Proposed Approach

### 3.1. Network Architectures

SISR’s algorithm, such as ESPCN [35] and FSRCNN [36], does not take full advantage of low-level feature information. With a deeper structure, there are more parameters. As shown in Figure 3, the proposed CASR consists of three components: (1) head module, (2) cascading block, and (3) upsample module. All we want is the balance between the performance and the cost.Figure 3
Network design for our model. The green cuboid means the head module; the yellow one represents the cascading block, and the last blue rectangle depicts the upsampling process.To better explore the mentioned issues, we adopt two different strategies: (1) original feature extraction and (2) cascading connection structure.


#### 3.1.1. Original Feature Extraction

We depictILR and ISR as the input and output of our models, respectively. Figure 2 illustrates how the head module extracts the original information from LR images:
(1)Fext=HextILR,where Hext· means a series of convolution operations. In the head module, we first replace the conventional one with a depthwise convolution layer for reducing parameters. Through an activation layer, the feature maps are sent to another specific convolution layer, deformable convolution. As we discussed in Section 2, deformable convolution adds an offset to each convolution sampling point, thus achieving free deformation of the sampling grid. Then, after passing through a specific convolutional layer with 1×1 kernel and another pReLU activation function, Fext is sent to the next stage for a higher-level abstraction.


#### 3.1.2. Cascading Connection Structure

Now, we present the CAS-Block. The cascade connection allows information to spread across multiple paths in the network, which greatly enhances feature fusion. It [10] has been widely applied in various computer vision tasks. In Figure 4, the mapping process of our cascade network includes C CAS-Blocks, each with a skip connection:
(2)Hmap0=C0Fext,(3)Hmapi=CiHmap0,where Hmapi presents the output of the Cith CAS-Block. Each CAS-Block contains one group convolution layer (with 3×3 or 1×3 kernel), one traditional convolution layer for adjusting the number of channels, and a pReLU layer. We prefer stacking several kernels with smaller sizes (such as 1×3 and 3×3) to directly applying larger kernels (such as 5×5 and 7×7) for enlarging the receptive field of the feature extraction module and decreasing the number of learnable parameters:
(4)Hmiddle=Cm+1HcasHmapm+Hmapm−1,Hmapm+Hmapm−2,Hmapm−1+Hmapm−2,where Hmapm,Hmapm−1,Hmapm−2 means all the outputs of the middle three CAS-Blocks. Hcas denotes the cascading operation:
(5)Hmap=Cm+2C1+Hmiddle,where Hmap· demotes our proposed mapping function. Finally, we use a common upsampling module to fuse the hierarchical structural features and amplify the image size:
(6)Fup=HupHmap,where Hup· indicates an upscale module. In recent years, many upsampling methods have been proposed, such as [12, 27, 36]. We adopt the postupsampling method, which has been proven effectively outstanding. The process of our model roughly includes three steps. First, taking the low-resolution ILR image as the original input, the feature extraction module obtains the initial features from the low-quality image. Then, these features are delivered to a higher abstraction layer. Finally, we adopt a simple upsampling block, including a convolutional layer, and a pixel-shuffle layer to enlarge the SR image.Figure 4
The architecture of cascading block (CAS-Block).⊕ operator denotes the element-wise additive operation and  means cascading operation.


### 3.2. Total Variation Loss

Aly and Dubois bring the total variation (TV) [37] loss to the SR field in order to suppress noise in generated images, and for imposing spatial smoothness, Yuan et al. also select this TV loss:
(7)LTVI^=1hwc∑i,j,kI∧i,j+1,k−I∧i,j,k2+I∧i+1,j,k−I∧i,j,k2,where I^ depicts the reconstructed HR image, h,w, represent the dimensions of the corresponding feature maps, and c symbolizes the number of channels. On the other hand, although mean square error (MSE) is available, previous work [38] proved that it is not a good choice. Thus, the second loss function is defined as follows:
(8)L=L1+λLTV.We applied these loss functions in the training process of our presented model. From the experiment, we found that adopting theL loss compared with the simple L1 loss, the model achieves better performance, and set λ=1e−4 works well. As shown in Figure 5, the loss L enables the network to generate smoother recovery images, and Figure 6 comparatively illustrates that the combined loss function may produce sharper SR results.Figure 5
×3 SR’s loss function comparison. In the first row, it is thecomic images in the Set14 dataset. The image processed with L1+LTV has precise details in the area around the eyes. The bottom line is the “img020” images in the Urban100 benchmark dataset. This method applies the L1+LTV method to reconstruct the clear details, such as the windows.
(a)(b)(c)(d)Figure 6
PSNR (dB) of training process on scales ×2 and ×4 withL1 and L1+LTV loss, respectively.
(a)(b)(c)(d)


### 3.3. Comparison with Other CNN-Based Methods

#### 3.3.1. Comparison with MSRN

Compared with MSRN, our CASR is different as follows. First, the basic module design is distinct. In MSRN, the multiscale residual block (MSRB) incorporates parallel convolution with multiple feature channels. The output of each multiscale residual block is cascaded together through a hierarchical feature fusion to produce the final result, which leads to a lot of calculations. However, our multiscale modules are branch-based, using regional skip connections and cascades extensively, scaling down parameters. Second, it is the difference in the activation function. MSRN utilizes the ReLU function, while we employ PReLU as the activation function. According to the comparison in Figure7, PReLU optimizes and improves ReLU. Under the premise of almost no increase in the amount of calculation, the PReLU function effectually improves the overfitting problem of the model, accelerates the convergence, and lowers the error. Therefore, our proposed multiscale module owns more effective representation capabilities.Figure 7
Comparison: ReLU versus PReLU. PReLU optimizes and improves ReLU. Under the premise of almost no increase in the amount of calculation, the overfitting problem of the model is effectively improved. The convergence is faster, and the error is lower.


#### 3.3.2. Comparison with MemNet

We summarize the main differences between MemNet [20] and our CASR. The former employs stacking memory blocks and massive shortcuts, while our method avoids extensive dense connections for lowering the number of parameters. What is more, Lim et al. trained their network with the L2 loss, but we prefer L1 loss to L2 loss function. Besides, MemNet regards the interpolated images as input. Contrastively, our proposed method directly extracts hierarchical features from the original LR images upsampled at the end of the process for computational efficiency and SR performance improvement.


## 3.1. Network Architectures

SISR’s algorithm, such as ESPCN [35] and FSRCNN [36], does not take full advantage of low-level feature information. With a deeper structure, there are more parameters. As shown in Figure 3, the proposed CASR consists of three components: (1) head module, (2) cascading block, and (3) upsample module. All we want is the balance between the performance and the cost.Figure 3
Network design for our model. The green cuboid means the head module; the yellow one represents the cascading block, and the last blue rectangle depicts the upsampling process.To better explore the mentioned issues, we adopt two different strategies: (1) original feature extraction and (2) cascading connection structure.


### 3.1.1. Original Feature Extraction

We depictILR and ISR as the input and output of our models, respectively. Figure 2 illustrates how the head module extracts the original information from LR images:
(1)Fext=HextILR,where Hext· means a series of convolution operations. In the head module, we first replace the conventional one with a depthwise convolution layer for reducing parameters. Through an activation layer, the feature maps are sent to another specific convolution layer, deformable convolution. As we discussed in Section 2, deformable convolution adds an offset to each convolution sampling point, thus achieving free deformation of the sampling grid. Then, after passing through a specific convolutional layer with 1×1 kernel and another pReLU activation function, Fext is sent to the next stage for a higher-level abstraction.


### 3.1.2. Cascading Connection Structure

Now, we present the CAS-Block. The cascade connection allows information to spread across multiple paths in the network, which greatly enhances feature fusion. It [10] has been widely applied in various computer vision tasks. In Figure 4, the mapping process of our cascade network includes C CAS-Blocks, each with a skip connection:
(2)Hmap0=C0Fext,(3)Hmapi=CiHmap0,where Hmapi presents the output of the Cith CAS-Block. Each CAS-Block contains one group convolution layer (with 3×3 or 1×3 kernel), one traditional convolution layer for adjusting the number of channels, and a pReLU layer. We prefer stacking several kernels with smaller sizes (such as 1×3 and 3×3) to directly applying larger kernels (such as 5×5 and 7×7) for enlarging the receptive field of the feature extraction module and decreasing the number of learnable parameters:
(4)Hmiddle=Cm+1HcasHmapm+Hmapm−1,Hmapm+Hmapm−2,Hmapm−1+Hmapm−2,where Hmapm,Hmapm−1,Hmapm−2 means all the outputs of the middle three CAS-Blocks. Hcas denotes the cascading operation:
(5)Hmap=Cm+2C1+Hmiddle,where Hmap· demotes our proposed mapping function. Finally, we use a common upsampling module to fuse the hierarchical structural features and amplify the image size:
(6)Fup=HupHmap,where Hup· indicates an upscale module. In recent years, many upsampling methods have been proposed, such as [12, 27, 36]. We adopt the postupsampling method, which has been proven effectively outstanding. The process of our model roughly includes three steps. First, taking the low-resolution ILR image as the original input, the feature extraction module obtains the initial features from the low-quality image. Then, these features are delivered to a higher abstraction layer. Finally, we adopt a simple upsampling block, including a convolutional layer, and a pixel-shuffle layer to enlarge the SR image.Figure 4
The architecture of cascading block (CAS-Block).⊕ operator denotes the element-wise additive operation and  means cascading operation.


## 3.1.1. Original Feature Extraction

We depictILR and ISR as the input and output of our models, respectively. Figure 2 illustrates how the head module extracts the original information from LR images:
(1)Fext=HextILR,where Hext· means a series of convolution operations. In the head module, we first replace the conventional one with a depthwise convolution layer for reducing parameters. Through an activation layer, the feature maps are sent to another specific convolution layer, deformable convolution. As we discussed in Section 2, deformable convolution adds an offset to each convolution sampling point, thus achieving free deformation of the sampling grid. Then, after passing through a specific convolutional layer with 1×1 kernel and another pReLU activation function, Fext is sent to the next stage for a higher-level abstraction.


## 3.1.2. Cascading Connection Structure

Now, we present the CAS-Block. The cascade connection allows information to spread across multiple paths in the network, which greatly enhances feature fusion. It [10] has been widely applied in various computer vision tasks. In Figure 4, the mapping process of our cascade network includes C CAS-Blocks, each with a skip connection:
(2)Hmap0=C0Fext,(3)Hmapi=CiHmap0,where Hmapi presents the output of the Cith CAS-Block. Each CAS-Block contains one group convolution layer (with 3×3 or 1×3 kernel), one traditional convolution layer for adjusting the number of channels, and a pReLU layer. We prefer stacking several kernels with smaller sizes (such as 1×3 and 3×3) to directly applying larger kernels (such as 5×5 and 7×7) for enlarging the receptive field of the feature extraction module and decreasing the number of learnable parameters:
(4)Hmiddle=Cm+1HcasHmapm+Hmapm−1,Hmapm+Hmapm−2,Hmapm−1+Hmapm−2,where Hmapm,Hmapm−1,Hmapm−2 means all the outputs of the middle three CAS-Blocks. Hcas denotes the cascading operation:
(5)Hmap=Cm+2C1+Hmiddle,where Hmap· demotes our proposed mapping function. Finally, we use a common upsampling module to fuse the hierarchical structural features and amplify the image size:
(6)Fup=HupHmap,where Hup· indicates an upscale module. In recent years, many upsampling methods have been proposed, such as [12, 27, 36]. We adopt the postupsampling method, which has been proven effectively outstanding. The process of our model roughly includes three steps. First, taking the low-resolution ILR image as the original input, the feature extraction module obtains the initial features from the low-quality image. Then, these features are delivered to a higher abstraction layer. Finally, we adopt a simple upsampling block, including a convolutional layer, and a pixel-shuffle layer to enlarge the SR image.Figure 4
The architecture of cascading block (CAS-Block).⊕ operator denotes the element-wise additive operation and  means cascading operation.


## 3.2. Total Variation Loss

Aly and Dubois bring the total variation (TV) [37] loss to the SR field in order to suppress noise in generated images, and for imposing spatial smoothness, Yuan et al. also select this TV loss:
(7)LTVI^=1hwc∑i,j,kI∧i,j+1,k−I∧i,j,k2+I∧i+1,j,k−I∧i,j,k2,where I^ depicts the reconstructed HR image, h,w, represent the dimensions of the corresponding feature maps, and c symbolizes the number of channels. On the other hand, although mean square error (MSE) is available, previous work [38] proved that it is not a good choice. Thus, the second loss function is defined as follows:
(8)L=L1+λLTV.We applied these loss functions in the training process of our presented model. From the experiment, we found that adopting theL loss compared with the simple L1 loss, the model achieves better performance, and set λ=1e−4 works well. As shown in Figure 5, the loss L enables the network to generate smoother recovery images, and Figure 6 comparatively illustrates that the combined loss function may produce sharper SR results.Figure 5
×3 SR’s loss function comparison. In the first row, it is thecomic images in the Set14 dataset. The image processed with L1+LTV has precise details in the area around the eyes. The bottom line is the “img020” images in the Urban100 benchmark dataset. This method applies the L1+LTV method to reconstruct the clear details, such as the windows.
(a)(b)(c)(d)Figure 6
PSNR (dB) of training process on scales ×2 and ×4 withL1 and L1+LTV loss, respectively.
(a)(b)(c)(d)


## 3.3. Comparison with Other CNN-Based Methods

### 3.3.1. Comparison with MSRN

Compared with MSRN, our CASR is different as follows. First, the basic module design is distinct. In MSRN, the multiscale residual block (MSRB) incorporates parallel convolution with multiple feature channels. The output of each multiscale residual block is cascaded together through a hierarchical feature fusion to produce the final result, which leads to a lot of calculations. However, our multiscale modules are branch-based, using regional skip connections and cascades extensively, scaling down parameters. Second, it is the difference in the activation function. MSRN utilizes the ReLU function, while we employ PReLU as the activation function. According to the comparison in Figure7, PReLU optimizes and improves ReLU. Under the premise of almost no increase in the amount of calculation, the PReLU function effectually improves the overfitting problem of the model, accelerates the convergence, and lowers the error. Therefore, our proposed multiscale module owns more effective representation capabilities.Figure 7
Comparison: ReLU versus PReLU. PReLU optimizes and improves ReLU. Under the premise of almost no increase in the amount of calculation, the overfitting problem of the model is effectively improved. The convergence is faster, and the error is lower.


### 3.3.2. Comparison with MemNet

We summarize the main differences between MemNet [20] and our CASR. The former employs stacking memory blocks and massive shortcuts, while our method avoids extensive dense connections for lowering the number of parameters. What is more, Lim et al. trained their network with the L2 loss, but we prefer L1 loss to L2 loss function. Besides, MemNet regards the interpolated images as input. Contrastively, our proposed method directly extracts hierarchical features from the original LR images upsampled at the end of the process for computational efficiency and SR performance improvement.


## 3.3.1. Comparison with MSRN

Compared with MSRN, our CASR is different as follows. First, the basic module design is distinct. In MSRN, the multiscale residual block (MSRB) incorporates parallel convolution with multiple feature channels. The output of each multiscale residual block is cascaded together through a hierarchical feature fusion to produce the final result, which leads to a lot of calculations. However, our multiscale modules are branch-based, using regional skip connections and cascades extensively, scaling down parameters. Second, it is the difference in the activation function. MSRN utilizes the ReLU function, while we employ PReLU as the activation function. According to the comparison in Figure7, PReLU optimizes and improves ReLU. Under the premise of almost no increase in the amount of calculation, the PReLU function effectually improves the overfitting problem of the model, accelerates the convergence, and lowers the error. Therefore, our proposed multiscale module owns more effective representation capabilities.Figure 7
Comparison: ReLU versus PReLU. PReLU optimizes and improves ReLU. Under the premise of almost no increase in the amount of calculation, the overfitting problem of the model is effectively improved. The convergence is faster, and the error is lower.


## 3.3.2. Comparison with MemNet

We summarize the main differences between MemNet [20] and our CASR. The former employs stacking memory blocks and massive shortcuts, while our method avoids extensive dense connections for lowering the number of parameters. What is more, Lim et al. trained their network with the L2 loss, but we prefer L1 loss to L2 loss function. Besides, MemNet regards the interpolated images as input. Contrastively, our proposed method directly extracts hierarchical features from the original LR images upsampled at the end of the process for computational efficiency and SR performance improvement.


## 4. Experimental Results

### 4.1. Training Details

We set depthwise separable convolution operations in head module shown in Figure2, which were first illustrated in the Inception net in the proposed model, and were able to reduce the size of the network parameters effectively. Figure 4 graphically illustrates the cascading process occurring. The medial layers’ outcomes are cascaded into the posterior layers and finally assemble in a convolutional trunk consisting of a depthwise separable convolutional operation with three times the input and output features and then thorough a pReLU activation function.We preferL1+LTV to L2 loss as the loss function, though the latter has been generally applied in computer vision tasks because of its intimate relation with PSNR’s calculation. However, the research community recently indicates that L1 loss provides better accuracy and faster convergence; TV loss (LTV) imposes spatial smoothness on reconstructed images. Specifically, we set training patches with a size of 128×128, and batch size=16. We employ 16000 iterations of back-propagation per epoch; we adopt the ADAM [39] optimizer with β1=0.9, β2=0.999, and ε=10−8 for optimization. We set 850 training epochs and the learning rate of all layers to 1×10−4 initially, which will be reduced to half for every 50 epochs. All experiments are run under the PyTorch framework and deployed on NVIDIA RTX 2080Ti GPU.


### 4.2. Datasets

DIV2K [15] is a high-definition dataset containing various image contents. It has 800 training images, 100 verification images, and 100 test images. We employ 800 training images to train the proposed model and randomly select ten validation images as evaluation. In the testing process, we adopt the following benchmark datasets as test datasets: Set5 [40], Set14 [41], B100 [42], and Urban100 [43]. They contain various scenes in real life, such as landscapes, buildings, and people, while the Digital Retinal Images for Vessel Extraction (DRIVE [34]) dataset is a dataset for retinal vessel segmentation. It consists of a total of JPEG 40 color fundus images, including 7 abnormal pathology cases.


### 4.3. Experimental Analyses

#### 4.3.1. Results on Benchmark Datasets

Our proposed method will be compared with the state-of-the-art SR model on two commonly adopted image quality metrics (i.e., PSNR and SSIM). We analyze our methods with several progressive networks: (1) bicubic, (2) SRCNN [12], (3) VDSR [16], (4) LapSRN [44], (5) DRCN [17], (6) DRRN [8], (7) MemNet [20], (8) RDN [24], (9) HDRN [25], (10) OISR [18], and (11) IDN [45]. As described in the technical literature, these methods were evaluated on four aforementioned datasets. Table 1 lists the performance of all mentioned algorithms. Our networks are much better than the comparison model in variant scale factors except for RDN. On some datasets, the performance of CASR is completely close to RDN, while the consumption of RDN is much larger in the meantime. We will particularly discuss it later.Table 1
Public benchmark test results (PSNR (dB)/SSIM). Bold illustrates the best result, and the underline indicates the second-best ones.
MethodScaleParamsMulti-addsSet5 [40]Set14 [41]B100 [42]Urban100 [43]SRCNN [12]×257K52.7G36.66/0.954232.42/0.906331.36/0.887929.50/0.8946VDSR [16]×2665K612G37.52/0.958633.01/0.912331.89/0.896030.76/0.9140LapSRN [44]×2813K29.9G37.51/0.959033.07/0.912931.89/0.895930.40/0.9100DRCN [17]×21774K17974G37.43/0.958733.23/0.913731.29/0.891130.22/0.9023DRRN [8]×2297K6796.9G37.74/0.952332.79/0.912131.67/0.890030.14/0.9112MemNet [20]×2677K2662.4G37.83/0.962333.15/0.919231.99/0.900030.51/0.9177RDN [24]×222.12M5096.2G38.30/0.961634.10/0.921832.40/0.902233.09/0.9368HDRN [25]×2——37.75/0.959033.49/0.915032.03/0.898031.87/0.9250OISR [18]×241.91M9656.5G37.98/0.960433.58/0.917232.18/0.899632.09/0.9281IDN [45]×2590K174.1G38.10/0.960133.91/0.919432.31/0.901232.92/0.9269CASR (ours)×2501K113G38.49/0.960733.97/0.920432.33/0.901732.90/0.9244SRCNN [12]×357K52.7G32.75/0.909029.28/0.820928.41/0.786326.24/0.7963VDSR [16]×3665K612.6G33.66/0.921329.77/0.831428.82/0.797627.14/0.8279LapSRN [44]×3813K29.9G33.82/0.920729.89/0.830428.82/0.795027.07/0.8298DRCN [17]×31.77M17974G33.84/0.922629.76/0.831128.80/0.796327.15/0.8276DRRN [8]×3297K6796.9G34.03/0.924429.96/0.834928.95/0.800427.53/0.8378MemNet [20]×3677K2662.4G34.09/0.959030.07/0.842929.89/0.811028.41/0.8610RDN [24]×322.31M2281.2G34.78/0.930030.67/0.848229.33/0.810529.00/0.8683HDRN [25]×3——34.24/0.924030.23/0.840028.96/0.804027.93/0.8490OISR [18]×344.86M4590.1G34.43/0.927330.33/0.842029.10/0.805328.20/0.8534IDN [45]×3590K105.6G34.46/0.928230.52/0.846229.25/0.809328.80/0.8653CASR (ours)×3597K52G34.49/0.929730.57/0.845029.29/0.809928.90/0.8704SRCNN [12]×457K52.7G30.48/0.862827.50/0.751326.90/0.710124.52/0.7221VDSR [16]×4665K612.6G31.35/0.883028.02/0.768027.29/0.722625.18/0.7540LapSRN [44]×4813K149.4G31.54/0.885028.19/0.772027.32/0.727025.21/0.7560DRCN [17]×41774K17974G31.53/0.884628.02/0.767027.23/0.723325.14/0.7510DRRN [8]×4297K6796.9G31.68/0.888828.21/0.772027.38/0.728425.44/0.7638RDN [24]×422.27M1309.2G32.61/0.900328.92/0.789327.80/0.743426.82/0.8069HDRN [25]×4——32.23/0.896028.58/0.781027.53/0.737026.09/0.7870OISR [18]×444.27M2962.5G32.21/0.895028.63/0.782227.58/0.736426.14/0.7874MemNet [20]×4677K2662.4G31.74/0.889328.26/0.772327.40/0.728625.50/0.7630IDN [45]×4590K81.8G32.46/0.896828.80/0.787627.71/0.742026.64/0.8033CASR (ours)×4597K51G32.67/0.900628.96/0.789927.77/0.742826.67/0.8057Compared with other methods, on dataset Set5, our proposed method performs better at all scales. Especially on the ×2 superresolution, the reconstructed image contains very clear texture details.Our method performs well in image superresolution restoration tasks of various scales on the dataset Set14. Specifically, the best performance of the benchmark SR model is RDN [24] and IDN [45], which reach 30.67/0.8482 and 30.52/0.8462 on PSNR/SSIM metrics, respectively, at the ×3 scale; our model is 0.1 db lower than the former and 0.05 db higher than the latter.As mentioned earlier, the dataset B100 contains many real-world images. As seen in Figure8, the vase image recovered by our method has clearer edges, reaching 32.33 db. Figure 8 demonstrates visual comparisons on dataset B100 and Urban100 with scales ×2 and ×4, respectively.Figure 8
Comparison of reconstructed image details on the benchmark dataset. From top to bottom, the ×2 superresolution experiment on B100, the ×3 and ×4 superresolution experiment on Urban100. Our presented method reconstructs SR images with richer details.The Urban100 dataset consists of 100 pictures of various buildings, which usually contain clear edges and rich textures. So, according to [24], RDN is expected to perform well on superresolution tasks, reaching 33.09 db on ×2. Our method acts well at ×3 and ×4 superresolution tasks, reaching 28.90 db and 26.67 db, respectively, which is approximately 0.1 db and 0.15 db lower than RDN, while CASR costs much less than the competitors.


#### 4.3.2. Comparison Results on Time Complexity

Besides, we have provided a comparison of the model’s efficiency in terms of time complexity on public dataset Set14 (taking ×4 as an instance), as tabulated in Table2. The table intuitively shows that the CASR model achieves a similar competitive experiment result compared to VDSR [16] and RDN [24], reaching 28.96/0.7899 on PSNR/SSIM metrics, while spending less time (0.1017 s on a single image) and costing the least resource on processing image restoration. We may conclude that our proposed CASR model takes the least time consumption and adopts acceptable parameters compared with VDSR and RDN.Table 2
Time consumption and experimental performance in dataset Set14 with the scale 4.
MethodScaleParamsMulti-addsRuntimes (s)PSNR/SSIMVDSR [16]×4665K612.6G0.452328.02/0.7680RDN [24]×422.27M1309.2G0.211428.92/0.7893CASR (ours)×4597K51G0.101728.96/0.7899


#### 4.3.3. Superresolution on Real-World Images

Table3 indicates that our proposed CASR method is highly competitive, achieving the lowest average values of NIQE/PI on benchmark dataset Set14 with scale factor ×4. Figure 9 illustrates the visual image restoration comparison with several SR methods on the real-world image chip. Results visually show that our method, compared to others, achieves better restorative performance. It not only achieves competitive PI and NIQE values but also improves more pleasant visual quality in terms of image, edge, texture, color, and feature-rich regions. Besides, as shown in Figure 10, the restorative performance on the larger scale, e.g., ×4, is also acceptable. The vessel in the retinal image is more clear than the competitors, and the edge of the retina is also sharp as we expected. Considering that the whole experiment was designed and conducted on dataset DIV2K, a supervised public dataset with ground truth images, which is acceptable and compromised, we believe that could provide a further research direction, exploring a more realistic oriented image SR process with a better degradation kernel on a real-world image dataset.Table 3
Average NIQE and PI values on public dataset Set14 with scale factor ×4.
MetricsSRCNN [12]VDSR [16]IDN [45]CASR (ours)NIQE [33]6.6245.7446.4724.506PI [22]5.9295.1215.4483.859Figure 9
Visual image comparisons with different SR methods on real-world imagechip with scale ×2.Figure 10
The qualitative comparisons between different SR methods on the digital retinal image for vessel extraction (DRIVE) dataset with scale ×4.


### 4.4. Ablation Study

In order to further explore the details of the experiment, we design 2 ablation experiments: one is to investigate the influence of different dilation factors on deformable convolution, and the other is the experiment of different loss functions’ effects.


#### 4.4.1. Study of the Deformable Convolution

Figure11 illustrates two training processes with variant dilation scales. We examine whether the dilation scale of deformable convolution would affect recovery performance or not. As is shown in Figure 11, with epochs rising, both training results grow as well, while the model with dilation two would achieve better performance but cause a worse fluctuation. We also compare the effect of different scale factors on the experimental performance, as shown in Table 4. It can be learned that with the same scale factor ×2, our proposed method, which replaces the convenient convolution with deformable convolution, would achieve better results on both datasets Set5 and B100. With the dilation factor enlarging, the performances go better. This result mainly occurs since the operation may effectively and dynamically expand the receptive field. Because different input feature maps may correspond to objects with different deformation scales, for some tasks, it is essential to adaptively determine the ratio or receptive field size.Figure 11
Comparable training results (PSNR) with scale factor ×2 under variable dilations; blue result illustrates the training process without dilation while red one shows deformable convolution with dilation 2.Table 4
The results of ablation studies on deformable convolutions with distinct dilation factors on dataset Set5 and B100.
ScaleDeformConvDilationSet5 (PSNR)B100 (PSNR)2×—37.3431.83✓137.5430.99✓233.9731.904×—31.9526.76✓132.3226.64✓232.5326.44


#### 4.4.2. Study of the Loss Function

To examine the effect of the mentioned loss functions, we design an ablation experiment to explore it. Expressed formally, let the first model be “L1” and the other one be L1+LTV (i.e., using the enhanced loss function L1+LTV). We tried different combinations of scale factor and loss function to examine which one would achieve better performance on dataset DIV2K, as demonstrated in Figure 6 and Table 5. Afterward, the validation process on dataset Set14 and Urban100 proves that the enhanced loss function actually results in a clearer image with more details in Figure 5.Table 5
Effect of the variant loss functions on the Set14 and Urban100 benchmark datasets, with a scale factor of ×2 and ×4, respectively.
ScaleLoss functionSet14Urban1002L1L1+LTV33.3833.9731.6231.904L1L1+LTV31.9528.9725.7626.65


## 4.1. Training Details

We set depthwise separable convolution operations in head module shown in Figure2, which were first illustrated in the Inception net in the proposed model, and were able to reduce the size of the network parameters effectively. Figure 4 graphically illustrates the cascading process occurring. The medial layers’ outcomes are cascaded into the posterior layers and finally assemble in a convolutional trunk consisting of a depthwise separable convolutional operation with three times the input and output features and then thorough a pReLU activation function.We preferL1+LTV to L2 loss as the loss function, though the latter has been generally applied in computer vision tasks because of its intimate relation with PSNR’s calculation. However, the research community recently indicates that L1 loss provides better accuracy and faster convergence; TV loss (LTV) imposes spatial smoothness on reconstructed images. Specifically, we set training patches with a size of 128×128, and batch size=16. We employ 16000 iterations of back-propagation per epoch; we adopt the ADAM [39] optimizer with β1=0.9, β2=0.999, and ε=10−8 for optimization. We set 850 training epochs and the learning rate of all layers to 1×10−4 initially, which will be reduced to half for every 50 epochs. All experiments are run under the PyTorch framework and deployed on NVIDIA RTX 2080Ti GPU.


## 4.2. Datasets

DIV2K [15] is a high-definition dataset containing various image contents. It has 800 training images, 100 verification images, and 100 test images. We employ 800 training images to train the proposed model and randomly select ten validation images as evaluation. In the testing process, we adopt the following benchmark datasets as test datasets: Set5 [40], Set14 [41], B100 [42], and Urban100 [43]. They contain various scenes in real life, such as landscapes, buildings, and people, while the Digital Retinal Images for Vessel Extraction (DRIVE [34]) dataset is a dataset for retinal vessel segmentation. It consists of a total of JPEG 40 color fundus images, including 7 abnormal pathology cases.


## 4.3. Experimental Analyses

### 4.3.1. Results on Benchmark Datasets

Our proposed method will be compared with the state-of-the-art SR model on two commonly adopted image quality metrics (i.e., PSNR and SSIM). We analyze our methods with several progressive networks: (1) bicubic, (2) SRCNN [12], (3) VDSR [16], (4) LapSRN [44], (5) DRCN [17], (6) DRRN [8], (7) MemNet [20], (8) RDN [24], (9) HDRN [25], (10) OISR [18], and (11) IDN [45]. As described in the technical literature, these methods were evaluated on four aforementioned datasets. Table 1 lists the performance of all mentioned algorithms. Our networks are much better than the comparison model in variant scale factors except for RDN. On some datasets, the performance of CASR is completely close to RDN, while the consumption of RDN is much larger in the meantime. We will particularly discuss it later.Table 1
Public benchmark test results (PSNR (dB)/SSIM). Bold illustrates the best result, and the underline indicates the second-best ones.
MethodScaleParamsMulti-addsSet5 [40]Set14 [41]B100 [42]Urban100 [43]SRCNN [12]×257K52.7G36.66/0.954232.42/0.906331.36/0.887929.50/0.8946VDSR [16]×2665K612G37.52/0.958633.01/0.912331.89/0.896030.76/0.9140LapSRN [44]×2813K29.9G37.51/0.959033.07/0.912931.89/0.895930.40/0.9100DRCN [17]×21774K17974G37.43/0.958733.23/0.913731.29/0.891130.22/0.9023DRRN [8]×2297K6796.9G37.74/0.952332.79/0.912131.67/0.890030.14/0.9112MemNet [20]×2677K2662.4G37.83/0.962333.15/0.919231.99/0.900030.51/0.9177RDN [24]×222.12M5096.2G38.30/0.961634.10/0.921832.40/0.902233.09/0.9368HDRN [25]×2——37.75/0.959033.49/0.915032.03/0.898031.87/0.9250OISR [18]×241.91M9656.5G37.98/0.960433.58/0.917232.18/0.899632.09/0.9281IDN [45]×2590K174.1G38.10/0.960133.91/0.919432.31/0.901232.92/0.9269CASR (ours)×2501K113G38.49/0.960733.97/0.920432.33/0.901732.90/0.9244SRCNN [12]×357K52.7G32.75/0.909029.28/0.820928.41/0.786326.24/0.7963VDSR [16]×3665K612.6G33.66/0.921329.77/0.831428.82/0.797627.14/0.8279LapSRN [44]×3813K29.9G33.82/0.920729.89/0.830428.82/0.795027.07/0.8298DRCN [17]×31.77M17974G33.84/0.922629.76/0.831128.80/0.796327.15/0.8276DRRN [8]×3297K6796.9G34.03/0.924429.96/0.834928.95/0.800427.53/0.8378MemNet [20]×3677K2662.4G34.09/0.959030.07/0.842929.89/0.811028.41/0.8610RDN [24]×322.31M2281.2G34.78/0.930030.67/0.848229.33/0.810529.00/0.8683HDRN [25]×3——34.24/0.924030.23/0.840028.96/0.804027.93/0.8490OISR [18]×344.86M4590.1G34.43/0.927330.33/0.842029.10/0.805328.20/0.8534IDN [45]×3590K105.6G34.46/0.928230.52/0.846229.25/0.809328.80/0.8653CASR (ours)×3597K52G34.49/0.929730.57/0.845029.29/0.809928.90/0.8704SRCNN [12]×457K52.7G30.48/0.862827.50/0.751326.90/0.710124.52/0.7221VDSR [16]×4665K612.6G31.35/0.883028.02/0.768027.29/0.722625.18/0.7540LapSRN [44]×4813K149.4G31.54/0.885028.19/0.772027.32/0.727025.21/0.7560DRCN [17]×41774K17974G31.53/0.884628.02/0.767027.23/0.723325.14/0.7510DRRN [8]×4297K6796.9G31.68/0.888828.21/0.772027.38/0.728425.44/0.7638RDN [24]×422.27M1309.2G32.61/0.900328.92/0.789327.80/0.743426.82/0.8069HDRN [25]×4——32.23/0.896028.58/0.781027.53/0.737026.09/0.7870OISR [18]×444.27M2962.5G32.21/0.895028.63/0.782227.58/0.736426.14/0.7874MemNet [20]×4677K2662.4G31.74/0.889328.26/0.772327.40/0.728625.50/0.7630IDN [45]×4590K81.8G32.46/0.896828.80/0.787627.71/0.742026.64/0.8033CASR (ours)×4597K51G32.67/0.900628.96/0.789927.77/0.742826.67/0.8057Compared with other methods, on dataset Set5, our proposed method performs better at all scales. Especially on the ×2 superresolution, the reconstructed image contains very clear texture details.Our method performs well in image superresolution restoration tasks of various scales on the dataset Set14. Specifically, the best performance of the benchmark SR model is RDN [24] and IDN [45], which reach 30.67/0.8482 and 30.52/0.8462 on PSNR/SSIM metrics, respectively, at the ×3 scale; our model is 0.1 db lower than the former and 0.05 db higher than the latter.As mentioned earlier, the dataset B100 contains many real-world images. As seen in Figure8, the vase image recovered by our method has clearer edges, reaching 32.33 db. Figure 8 demonstrates visual comparisons on dataset B100 and Urban100 with scales ×2 and ×4, respectively.Figure 8
Comparison of reconstructed image details on the benchmark dataset. From top to bottom, the ×2 superresolution experiment on B100, the ×3 and ×4 superresolution experiment on Urban100. Our presented method reconstructs SR images with richer details.The Urban100 dataset consists of 100 pictures of various buildings, which usually contain clear edges and rich textures. So, according to [24], RDN is expected to perform well on superresolution tasks, reaching 33.09 db on ×2. Our method acts well at ×3 and ×4 superresolution tasks, reaching 28.90 db and 26.67 db, respectively, which is approximately 0.1 db and 0.15 db lower than RDN, while CASR costs much less than the competitors.


### 4.3.2. Comparison Results on Time Complexity

Besides, we have provided a comparison of the model’s efficiency in terms of time complexity on public dataset Set14 (taking ×4 as an instance), as tabulated in Table2. The table intuitively shows that the CASR model achieves a similar competitive experiment result compared to VDSR [16] and RDN [24], reaching 28.96/0.7899 on PSNR/SSIM metrics, while spending less time (0.1017 s on a single image) and costing the least resource on processing image restoration. We may conclude that our proposed CASR model takes the least time consumption and adopts acceptable parameters compared with VDSR and RDN.Table 2
Time consumption and experimental performance in dataset Set14 with the scale 4.
MethodScaleParamsMulti-addsRuntimes (s)PSNR/SSIMVDSR [16]×4665K612.6G0.452328.02/0.7680RDN [24]×422.27M1309.2G0.211428.92/0.7893CASR (ours)×4597K51G0.101728.96/0.7899


### 4.3.3. Superresolution on Real-World Images

Table3 indicates that our proposed CASR method is highly competitive, achieving the lowest average values of NIQE/PI on benchmark dataset Set14 with scale factor ×4. Figure 9 illustrates the visual image restoration comparison with several SR methods on the real-world image chip. Results visually show that our method, compared to others, achieves better restorative performance. It not only achieves competitive PI and NIQE values but also improves more pleasant visual quality in terms of image, edge, texture, color, and feature-rich regions. Besides, as shown in Figure 10, the restorative performance on the larger scale, e.g., ×4, is also acceptable. The vessel in the retinal image is more clear than the competitors, and the edge of the retina is also sharp as we expected. Considering that the whole experiment was designed and conducted on dataset DIV2K, a supervised public dataset with ground truth images, which is acceptable and compromised, we believe that could provide a further research direction, exploring a more realistic oriented image SR process with a better degradation kernel on a real-world image dataset.Table 3
Average NIQE and PI values on public dataset Set14 with scale factor ×4.
MetricsSRCNN [12]VDSR [16]IDN [45]CASR (ours)NIQE [33]6.6245.7446.4724.506PI [22]5.9295.1215.4483.859Figure 9
Visual image comparisons with different SR methods on real-world imagechip with scale ×2.Figure 10
The qualitative comparisons between different SR methods on the digital retinal image for vessel extraction (DRIVE) dataset with scale ×4.


## 4.3.1. Results on Benchmark Datasets

Our proposed method will be compared with the state-of-the-art SR model on two commonly adopted image quality metrics (i.e., PSNR and SSIM). We analyze our methods with several progressive networks: (1) bicubic, (2) SRCNN [12], (3) VDSR [16], (4) LapSRN [44], (5) DRCN [17], (6) DRRN [8], (7) MemNet [20], (8) RDN [24], (9) HDRN [25], (10) OISR [18], and (11) IDN [45]. As described in the technical literature, these methods were evaluated on four aforementioned datasets. Table 1 lists the performance of all mentioned algorithms. Our networks are much better than the comparison model in variant scale factors except for RDN. On some datasets, the performance of CASR is completely close to RDN, while the consumption of RDN is much larger in the meantime. We will particularly discuss it later.Table 1
Public benchmark test results (PSNR (dB)/SSIM). Bold illustrates the best result, and the underline indicates the second-best ones.
MethodScaleParamsMulti-addsSet5 [40]Set14 [41]B100 [42]Urban100 [43]SRCNN [12]×257K52.7G36.66/0.954232.42/0.906331.36/0.887929.50/0.8946VDSR [16]×2665K612G37.52/0.958633.01/0.912331.89/0.896030.76/0.9140LapSRN [44]×2813K29.9G37.51/0.959033.07/0.912931.89/0.895930.40/0.9100DRCN [17]×21774K17974G37.43/0.958733.23/0.913731.29/0.891130.22/0.9023DRRN [8]×2297K6796.9G37.74/0.952332.79/0.912131.67/0.890030.14/0.9112MemNet [20]×2677K2662.4G37.83/0.962333.15/0.919231.99/0.900030.51/0.9177RDN [24]×222.12M5096.2G38.30/0.961634.10/0.921832.40/0.902233.09/0.9368HDRN [25]×2——37.75/0.959033.49/0.915032.03/0.898031.87/0.9250OISR [18]×241.91M9656.5G37.98/0.960433.58/0.917232.18/0.899632.09/0.9281IDN [45]×2590K174.1G38.10/0.960133.91/0.919432.31/0.901232.92/0.9269CASR (ours)×2501K113G38.49/0.960733.97/0.920432.33/0.901732.90/0.9244SRCNN [12]×357K52.7G32.75/0.909029.28/0.820928.41/0.786326.24/0.7963VDSR [16]×3665K612.6G33.66/0.921329.77/0.831428.82/0.797627.14/0.8279LapSRN [44]×3813K29.9G33.82/0.920729.89/0.830428.82/0.795027.07/0.8298DRCN [17]×31.77M17974G33.84/0.922629.76/0.831128.80/0.796327.15/0.8276DRRN [8]×3297K6796.9G34.03/0.924429.96/0.834928.95/0.800427.53/0.8378MemNet [20]×3677K2662.4G34.09/0.959030.07/0.842929.89/0.811028.41/0.8610RDN [24]×322.31M2281.2G34.78/0.930030.67/0.848229.33/0.810529.00/0.8683HDRN [25]×3——34.24/0.924030.23/0.840028.96/0.804027.93/0.8490OISR [18]×344.86M4590.1G34.43/0.927330.33/0.842029.10/0.805328.20/0.8534IDN [45]×3590K105.6G34.46/0.928230.52/0.846229.25/0.809328.80/0.8653CASR (ours)×3597K52G34.49/0.929730.57/0.845029.29/0.809928.90/0.8704SRCNN [12]×457K52.7G30.48/0.862827.50/0.751326.90/0.710124.52/0.7221VDSR [16]×4665K612.6G31.35/0.883028.02/0.768027.29/0.722625.18/0.7540LapSRN [44]×4813K149.4G31.54/0.885028.19/0.772027.32/0.727025.21/0.7560DRCN [17]×41774K17974G31.53/0.884628.02/0.767027.23/0.723325.14/0.7510DRRN [8]×4297K6796.9G31.68/0.888828.21/0.772027.38/0.728425.44/0.7638RDN [24]×422.27M1309.2G32.61/0.900328.92/0.789327.80/0.743426.82/0.8069HDRN [25]×4——32.23/0.896028.58/0.781027.53/0.737026.09/0.7870OISR [18]×444.27M2962.5G32.21/0.895028.63/0.782227.58/0.736426.14/0.7874MemNet [20]×4677K2662.4G31.74/0.889328.26/0.772327.40/0.728625.50/0.7630IDN [45]×4590K81.8G32.46/0.896828.80/0.787627.71/0.742026.64/0.8033CASR (ours)×4597K51G32.67/0.900628.96/0.789927.77/0.742826.67/0.8057Compared with other methods, on dataset Set5, our proposed method performs better at all scales. Especially on the ×2 superresolution, the reconstructed image contains very clear texture details.Our method performs well in image superresolution restoration tasks of various scales on the dataset Set14. Specifically, the best performance of the benchmark SR model is RDN [24] and IDN [45], which reach 30.67/0.8482 and 30.52/0.8462 on PSNR/SSIM metrics, respectively, at the ×3 scale; our model is 0.1 db lower than the former and 0.05 db higher than the latter.As mentioned earlier, the dataset B100 contains many real-world images. As seen in Figure8, the vase image recovered by our method has clearer edges, reaching 32.33 db. Figure 8 demonstrates visual comparisons on dataset B100 and Urban100 with scales ×2 and ×4, respectively.Figure 8
Comparison of reconstructed image details on the benchmark dataset. From top to bottom, the ×2 superresolution experiment on B100, the ×3 and ×4 superresolution experiment on Urban100. Our presented method reconstructs SR images with richer details.The Urban100 dataset consists of 100 pictures of various buildings, which usually contain clear edges and rich textures. So, according to [24], RDN is expected to perform well on superresolution tasks, reaching 33.09 db on ×2. Our method acts well at ×3 and ×4 superresolution tasks, reaching 28.90 db and 26.67 db, respectively, which is approximately 0.1 db and 0.15 db lower than RDN, while CASR costs much less than the competitors.


## 4.3.2. Comparison Results on Time Complexity

Besides, we have provided a comparison of the model’s efficiency in terms of time complexity on public dataset Set14 (taking ×4 as an instance), as tabulated in Table2. The table intuitively shows that the CASR model achieves a similar competitive experiment result compared to VDSR [16] and RDN [24], reaching 28.96/0.7899 on PSNR/SSIM metrics, while spending less time (0.1017 s on a single image) and costing the least resource on processing image restoration. We may conclude that our proposed CASR model takes the least time consumption and adopts acceptable parameters compared with VDSR and RDN.Table 2
Time consumption and experimental performance in dataset Set14 with the scale 4.
MethodScaleParamsMulti-addsRuntimes (s)PSNR/SSIMVDSR [16]×4665K612.6G0.452328.02/0.7680RDN [24]×422.27M1309.2G0.211428.92/0.7893CASR (ours)×4597K51G0.101728.96/0.7899


## 4.3.3. Superresolution on Real-World Images

Table3 indicates that our proposed CASR method is highly competitive, achieving the lowest average values of NIQE/PI on benchmark dataset Set14 with scale factor ×4. Figure 9 illustrates the visual image restoration comparison with several SR methods on the real-world image chip. Results visually show that our method, compared to others, achieves better restorative performance. It not only achieves competitive PI and NIQE values but also improves more pleasant visual quality in terms of image, edge, texture, color, and feature-rich regions. Besides, as shown in Figure 10, the restorative performance on the larger scale, e.g., ×4, is also acceptable. The vessel in the retinal image is more clear than the competitors, and the edge of the retina is also sharp as we expected. Considering that the whole experiment was designed and conducted on dataset DIV2K, a supervised public dataset with ground truth images, which is acceptable and compromised, we believe that could provide a further research direction, exploring a more realistic oriented image SR process with a better degradation kernel on a real-world image dataset.Table 3
Average NIQE and PI values on public dataset Set14 with scale factor ×4.
MetricsSRCNN [12]VDSR [16]IDN [45]CASR (ours)NIQE [33]6.6245.7446.4724.506PI [22]5.9295.1215.4483.859Figure 9
Visual image comparisons with different SR methods on real-world imagechip with scale ×2.Figure 10
The qualitative comparisons between different SR methods on the digital retinal image for vessel extraction (DRIVE) dataset with scale ×4.


## 4.4. Ablation Study

In order to further explore the details of the experiment, we design 2 ablation experiments: one is to investigate the influence of different dilation factors on deformable convolution, and the other is the experiment of different loss functions’ effects.


### 4.4.1. Study of the Deformable Convolution

Figure11 illustrates two training processes with variant dilation scales. We examine whether the dilation scale of deformable convolution would affect recovery performance or not. As is shown in Figure 11, with epochs rising, both training results grow as well, while the model with dilation two would achieve better performance but cause a worse fluctuation. We also compare the effect of different scale factors on the experimental performance, as shown in Table 4. It can be learned that with the same scale factor ×2, our proposed method, which replaces the convenient convolution with deformable convolution, would achieve better results on both datasets Set5 and B100. With the dilation factor enlarging, the performances go better. This result mainly occurs since the operation may effectively and dynamically expand the receptive field. Because different input feature maps may correspond to objects with different deformation scales, for some tasks, it is essential to adaptively determine the ratio or receptive field size.Figure 11
Comparable training results (PSNR) with scale factor ×2 under variable dilations; blue result illustrates the training process without dilation while red one shows deformable convolution with dilation 2.Table 4
The results of ablation studies on deformable convolutions with distinct dilation factors on dataset Set5 and B100.
ScaleDeformConvDilationSet5 (PSNR)B100 (PSNR)2×—37.3431.83✓137.5430.99✓233.9731.904×—31.9526.76✓132.3226.64✓232.5326.44


### 4.4.2. Study of the Loss Function

To examine the effect of the mentioned loss functions, we design an ablation experiment to explore it. Expressed formally, let the first model be “L1” and the other one be L1+LTV (i.e., using the enhanced loss function L1+LTV). We tried different combinations of scale factor and loss function to examine which one would achieve better performance on dataset DIV2K, as demonstrated in Figure 6 and Table 5. Afterward, the validation process on dataset Set14 and Urban100 proves that the enhanced loss function actually results in a clearer image with more details in Figure 5.Table 5
Effect of the variant loss functions on the Set14 and Urban100 benchmark datasets, with a scale factor of ×2 and ×4, respectively.
ScaleLoss functionSet14Urban1002L1L1+LTV33.3833.9731.6231.904L1L1+LTV31.9528.9725.7626.65


## 4.4.1. Study of the Deformable Convolution

Figure11 illustrates two training processes with variant dilation scales. We examine whether the dilation scale of deformable convolution would affect recovery performance or not. As is shown in Figure 11, with epochs rising, both training results grow as well, while the model with dilation two would achieve better performance but cause a worse fluctuation. We also compare the effect of different scale factors on the experimental performance, as shown in Table 4. It can be learned that with the same scale factor ×2, our proposed method, which replaces the convenient convolution with deformable convolution, would achieve better results on both datasets Set5 and B100. With the dilation factor enlarging, the performances go better. This result mainly occurs since the operation may effectively and dynamically expand the receptive field. Because different input feature maps may correspond to objects with different deformation scales, for some tasks, it is essential to adaptively determine the ratio or receptive field size.Figure 11
Comparable training results (PSNR) with scale factor ×2 under variable dilations; blue result illustrates the training process without dilation while red one shows deformable convolution with dilation 2.Table 4
The results of ablation studies on deformable convolutions with distinct dilation factors on dataset Set5 and B100.
ScaleDeformConvDilationSet5 (PSNR)B100 (PSNR)2×—37.3431.83✓137.5430.99✓233.9731.904×—31.9526.76✓132.3226.64✓232.5326.44


## 4.4.2. Study of the Loss Function

To examine the effect of the mentioned loss functions, we design an ablation experiment to explore it. Expressed formally, let the first model be “L1” and the other one be L1+LTV (i.e., using the enhanced loss function L1+LTV). We tried different combinations of scale factor and loss function to examine which one would achieve better performance on dataset DIV2K, as demonstrated in Figure 6 and Table 5. Afterward, the validation process on dataset Set14 and Urban100 proves that the enhanced loss function actually results in a clearer image with more details in Figure 5.Table 5
Effect of the variant loss functions on the Set14 and Urban100 benchmark datasets, with a scale factor of ×2 and ×4, respectively.
ScaleLoss functionSet14Urban1002L1L1+LTV33.3833.9731.6231.904L1L1+LTV31.9528.9725.7626.65


## 5. Conclusion

This paper presents two novel CNN architectures, namely, head module and CAS-Block, to improve the SISR performance. Compared with the state-of-the-art (SotA) CNN-based algorithms, our presented head module considers low-level feature expression by applying depthwise separable convolution and deformable convolution, which is demonstrated to not only effectively extract the patterns but also reduce the parameter size. At the same time, the CAS-Block employs a global residual connection and abundantly utilizes cascading connections to capture remote spatial features from former inputs. Extensive experiments have illustrated that our presented model has effectively improved both the quality of the reconstructed images and the processing speed compared with the SotA methods in terms of quantitative indicators and realistic visual effects.



---

*Source: 5579090-2021-10-21.xml*