Single infrared image super‐resolution based on lightweight multi‐path feature fusion network

IET Image ProcessingEarly View ORIGINAL RESEARCH PAPEROpen Access Single infrared image super-resolution based on lightweight multi-path feature fusion network Fei Mo, Fei Mo orcid.org/0000-0002-4851-3011 Guangdong Provincial Key Laboratory of Cyber-Physical System, School of Automation, Guangdong University of Technology, Guangzhou, China School of Computer, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this authorHeng Wu, Corresponding Author Heng Wu heng.wu@foxmail.com Guangdong Provincial Key Laboratory of Cyber-Physical System, School of Automation, Guangdong University of Technology, Guangzhou, China School of Computer, Guangdong University of Technology, Guangzhou, China Correspondence Heng Wu, Guangdong Provincial Key Laboratory of Cyber-Physical System, School of Automation, Guangdong University of Technology, Guangzhou 510006, China. Email: heng.wu@foxmail.com Shaojuan Luo, School of Chemical Engineering and Light Industry, Guangdong University of Technology, Guangzhou 510006, China. Email: kesjluo@gdut.edu.cnSearch for more papers by this authorShuo Qu, Shuo Qu Guangdong Provincial Key Laboratory of Cyber-Physical System, School of Automation, Guangdong University of Technology, Guangzhou, China School of Computer, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this authorShaojuan Luo, Corresponding Author Shaojuan Luo kesjluo@gdut.edu.cn School of Chemical Engineering and Light Industry, Guangdong University of Technology, Guangzhou, China Correspondence Heng Wu, Guangdong Provincial Key Laboratory of Cyber-Physical System, School of Automation, Guangdong University of Technology, Guangzhou 510006, China. Email: heng.wu@foxmail.com Shaojuan Luo, School of Chemical Engineering and Light Industry, Guangdong University of Technology, Guangzhou 510006, China. Email: kesjluo@gdut.edu.cnSearch for more papers by this authorLianglun Cheng, Lianglun Cheng Guangdong Provincial Key Laboratory of Cyber-Physical System, School of Automation, Guangdong University of Technology, Guangzhou, China School of Computer, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this author Fei Mo, Fei Mo orcid.org/0000-0002-4851-3011 Guangdong Provincial Key Laboratory of Cyber-Physical System, School of Automation, Guangdong University of Technology, Guangzhou, China School of Computer, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this authorHeng Wu, Corresponding Author Heng Wu heng.wu@foxmail.com Guangdong Provincial Key Laboratory of Cyber-Physical System, School of Automation, Guangdong University of Technology, Guangzhou, China School of Computer, Guangdong University of Technology, Guangzhou, China Correspondence Heng Wu, Guangdong Provincial Key Laboratory of Cyber-Physical System, School of Automation, Guangdong University of Technology, Guangzhou 510006, China. Email: heng.wu@foxmail.com Shaojuan Luo, School of Chemical Engineering and Light Industry, Guangdong University of Technology, Guangzhou 510006, China. Email: kesjluo@gdut.edu.cnSearch for more papers by this authorShuo Qu, Shuo Qu Guangdong Provincial Key Laboratory of Cyber-Physical System, School of Automation, Guangdong University of Technology, Guangzhou, China School of Computer, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this authorShaojuan Luo, Corresponding Author Shaojuan Luo kesjluo@gdut.edu.cn School of Chemical Engineering and Light Industry, Guangdong University of Technology, Guangzhou, China Correspondence Heng Wu, Guangdong Provincial Key Laboratory of Cyber-Physical System, School of Automation, Guangdong University of Technology, Guangzhou 510006, China. Email: heng.wu@foxmail.com Shaojuan Luo, School of Chemical Engineering and Light Industry, Guangdong University of Technology, Guangzhou 510006, China. Email: kesjluo@gdut.edu.cnSearch for more papers by this authorLianglun Cheng, Lianglun Cheng Guangdong Provincial Key Laboratory of Cyber-Physical System, School of Automation, Guangdong University of Technology, Guangzhou, China School of Computer, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this author First published: 24 February 2022 https://doi.org/10.1049/ipr2.12454AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Single infrared (IR) image super-resolution methods can help to reduce the cost and difficulty in manufacturing IR sensors for the imaging system. However, the deep learning-based image SR methods need to build a complex network and thus consume a lot of computational power, which limits the application of SR technology on devices with low computing resources in practice. To solve this problem, the authors present a lightweight multi-path feature fusion network (MFFN) for the single infrared (IR) image SR. A multi-path feature fusion block (MFFB) is developed to extract and fuse multiple and discriminative features in a recursive feedback way. Specifically, the multiple features are refined via the linear feature extraction branch, shared-source residual feature extraction branch, and channel attention branch in MFFB. Finally, the authors reconstruct the high-resolution IR images from the low-resolution counterpart based on the refined multiple features. The experimental results demonstrate that MFFN achieves high-quality single infrared image SR and shows superiority over previous methods for several scale factors (e.g. ×2, ×3, and ×4). MFFN has potential applications in the mobile infrared imaging system. 1 INTRODUCTION Infrared imaging, due to the advantages of well anti-interference, powerful penetration, and long imaging distance, has wide applications in military detection, medical imaging, security, and surveillance [1-3]. However, due to the physical limitation of the infrared (IR) image sensors and noise interference, the IR images captured by the previous IR imaging devices usually encounter the problems of low spatial resolution, low contrast, blur edges, and loss of high-frequency details [3, 4]. Over the past few years, various methods have been proposed to improve the quality of IR images, such as the IR image denoising, IR and visible (VI) image fusion, IR image super-resolution (SR), etc. Among the existing methods, the IR image SR method is an excellent approach to enhance the quality of low-resolution (LR) IR images because of the high efficiency and low requirements on the IR imaging system. Generally, the IR image SR method can be divided into three types, single image SR (SISR), multi-frame image SR (MISR), and IR-VI fusion-based SR (VIFSR). Compared with MISR and VIFSR, SISR requires the least resource because only the LR IR image is needed. SISR is a downstream computer vision technique that tries to reconstruct a high-resolution (HR) image from its LR version. However, SISR is a highly ill-posed problem since a single LR image may correspond to multiple potential HR images. This problem exists in both VI and IR image SR areas. Recently, in the VI image SR area, numerous methods [5-18] have been developed to find the most likely solutions. The existing VI SR methods can be roughly grouped into three categories: interpolation-based [5], prior-based [6], and learning-based methods [9-20]. Among them, the interpolation-based methods, such as the nearest neighbour, bilinear and bicubic interpolations, are the simplest and fastest. Yet, they fail to recover the details of the image efficiently, and the images are often super-resolved with blurred edges and artefacts. Unlike the interpolation-based methods, the prior-based methods use additional statistical image priors to reproduce the HR images. Most recently, deep learning-based methods [7, 8] attract much attention. The deep learning (DL) algorithms, especially the convolutional neural networks (CNN) [35], have shown prominent performance and superiority over conventional SR methods. As a pioneering work, Dong et al. [12] firstly introduced a three-layer CNN, namely SRCNN, which demonstrated the effectiveness of CNN for image SR. Later, Kim et al. [13] increased the depth of CNN network to 20 layers in VDSR for extracting more informational features from LR input and the performance of VDSR exceeded the SRCNN. Meanwhile, Kim et al. also designed a deep-recursive convolutional network (DRCN) [14] by incorporating the recursive learning strategy for reducing the model parameters. Similarly, Tai et al. [15] proposed a deep recursive residual network (DRRN) which combined the recursive learning and residual connections to economize the parameters and outperformed the DRCN. The EDSR [16] further pushed the depth of CNN to more than 160 layers and achieved better performance at the cost of heavy computation and numerous parameters. Unlike [16], Li et al. [17] designed a feedback block that consists of a series of iterative up- and down-sampling layers. However, these methods [12-17] are not suitable for resource-constrained equipment like mobile devices. Accordingly, in [18], Hui et al. presented IDN, where the intermediate features were unequally split into two parts along the channel dimension. As a result, IDN achieved good performance at a modest size. Different from VI image SR methods [12-18], the IR image SR methods encounter several problems: (1) Previous DL-based models did not make full use of multiple intermediate features, and they mainly focused on designing deeper and wider networks to capture more informative features; (2) the existing SR methods rarely pay attention to the inter-dependency across various channels; (3) previous lightweight models do not attain a good trade-off between model size (parameters) and performance. To address the above-mentioned issues, we propose a lightweight multi-path feature fusion network (MFFN) for the IR image SR. Specifically, we design a multi-path feature fusion block (MFFB) that is utilized as the basic feature extraction block ensembled in a recursive feedback manner to reduce the number of parameters. The MFFB adaptively learns the inter-dependencies among different channels to recalibrate the features for better feature representation. To ease the training difficulty and extract more discriminative features, we construct a shared-source residual feature extraction branch (SRFEB). Furthermore, we integrate another linear feature extraction branch (LFEB) to extract linear features. Then, we aggregate multiple features via feature fusion module to filter the abundant information. Finally, we upsample the extracted informative features to the desired size through the reconstruction part. Compared with the previous state-of-the-art SR methods (e.g., VDSR, DRRN, IDN, and SRFBN), MFFN is effective for image SR and more lightweight. Besides, MFFN can extract multiple discriminative features and refines them. The main contributions of this paper are described as follows: We propose a lightweight multi-path feature fusion network (MFFN) which is built by MFFB in a recursive feedback way for accurate and fast infrared image SR. We design a MFFB to explore multiple features. The multiple feature exploration is achieved by three paths, that is, the LFEB, SRFEB, and channel attention branch (CAB). Comprehensive ablation study shows that MFFB can aggregate multiple features and reconstruct high-quality IR SR images. Extensive experiments demonstrate that the MFFN achieves a good trade-off between the performance and computational cost. Due to the advantages of low computational cost and high running speed, MFFN can be used in many application areas, such as medical imaging, military detection, and satellite imaging. 2 RELATED WORK During the past decade, many SISR methods have been reported to improve the quality of the VI and IR LR images [12, 13, 21–34, 40, 41]. Here we briefly review SR works related to CNN-based algorithms and attention mechanism. 2.1 Deep CNN-based image SR Currently, CNN-based methods have been widely studied in image SR. For instance, the SRCNN [12] and VDSR [13] shown a great success in VI image SR. Nevertheless, the LR input of these methods is interpolated to the desired size before feeding into networks, which results in inflexibility and heavy computation cost. To solve this problem, Dong et al. [24] directly adopted the original LR image as input and introduced a deconvolution layer (aka transposed convolution layer) to implement the final upsampling reconstruction. On the contrary, Shi et al. [31] replaced deconvolution with an efficient sub-pixel convolutional layer to remove the checkerboard artefact. However, these methods simply stack building blocks in a chain way, and fail to adequately utilize information from different weight layer. To improve these issues, Zhang et al. proposed the residual dense network (RDN) [32] by introducing dense connections into the residual block to aggregating all the features from the preceding layers. Similarly, MSRN [46] and MDCN [47] both utilized the multi-scale features to improve the performance of the model. Recently, the lightweight model SRFBN [17] and IDN [18] were developed to reduce the parameters of the network. Additionally, Ahn et al. [33] designed the CARN-M for mobile devices by using a cascading network architecture at the cost of dropping the PSNR. 2.2 DL-based IR image SR For IR image SR, Choi et al. [23] proposed the DL thermal IR image enhancement network (TEN), where the visible spectral dataset was adopted to train the network. However, TEN failed to achieve favourable SR results due to the shallow depth. In 2017, Sun et al. [25] used a zoom mechanism to increase the nonlinear mapping of network. Unlike [23] and [25], Han et al. [26] proposed a deep network that used both infrared and visible images as input to recover HR IR images. Meanwhile, Zhang et al. [27] combined compressive sensing and DL into an architecture for IR image SR. Later, He et al. [28] designed a cascaded network (CDN_MRF) with various receptive fields. However, the IR LR images were interpolated to the size of ×8 images before feed into the network, leading to the extra computation costs. Furthermore, a deep convolutional generative adversarial network (DCGAN) [29] was reported to enhance the resolution of face images captured by the IR camera. Similarly, Kuang et al. [30] presented an image enhancement conditional GAN (IE-CGAN) that was used to improve the resolution of LR IR images. 2.3 Attention mechanism The attention mechanism is aiming at extracting more useful information in features, and has been widely used in various computer vision tasks [36-42]. Hu et al. [36] introduced squeeze-and-excitation (SE) block to treat channel-wise feature differently according to the respective weight responses, showing its effectiveness on image classification. Woo et al. presented the CBAM [37] that modified the SE block to emphasize meaningful features along channel and spatial attention mechanisms. Subsequently, Wang et al. [38] proposed the non-local module to generate the wide attention map by calculating the correlation matrix between each spatial point in the feature map. Wang et al. [39] proposed a residual attention network (RAN) to improve the performance of image classification. In the field of image SR, Dai et al. [40] exploited the second-order attention mechanism to explore more powerful feature expression by using second-order feature statistics. Zhang et al. [41] reported the RCAN that explored deeper CNN-based architecture with the skip connection channel attention mechanism. This paper proposes a lightweight MFFN for high-quality IR image SR based on the DL IR image SR and attention mechanism. An MFFB that consists of LFEB, SRFEB and CAB is designed to extract multiple features. The effectiveness of MFFN is validated by extensive experiments. 3 MULTI-PATH FEATURE FUSION NETWORK In this section, we describe the details of MFFN. We use the long-range residual learning in the identity branch with the upsampling module, that is, the bilinear interpolation operation. Additionally, we build a recursive MFFB structure in the trunk, where the parameters are shared for constructing the lightweight network. 3.1 Network architecture As shown in Figure 1, the MFFN includes three parts, the shallow features extracted module (SFEM), MFFB, and reconstruction module (RM). The recursive part is shown in the solid blue box of Figure 1 [17]. Here, the recursive part means that the input of the MFFB is the concatenation of the input of the current iteration and the output of the last iteration. With the recursive part, the network can be decomposed into T $T$ iteration, where the input of MFFB is the same for each iteration t = { 1 , 2 , … , T } $t = \{ 1,2, \ldots ,T\} $ . Note that the weight parameters are shared in the recursive manner to build the lightweight architecture. FIGURE 1Open in figure viewerPowerPoint The network structure of the proposed MFFN. Conv, convolution with filter 3 × 3; Upsample, bilinear upsample kernel Assuming that I L R ${I_{LR}}$ and I S R ${I_{SR}}$ are the input and output of MFFNet, respectively. Here, the HR IR image I S R ${I_{SR}}$ is directly reconstructed from the LR IR image I L R ${I_{LR}}$ , expressed as: I S R = H M F F N ( I L R ) , \begin{equation}{I_{SR}} = {H_{MFFN}}({I_{LR}}),\end{equation} (1)where H M F F N ( · ) ${H_{MFFN}}( \cdot )$ denotes the implicit function of the MFFN. For the SFEM, inspired by [15, 33, 40, 41], only one 3 × 3 $3 \times 3$ convolutional layer is adopted to extract the shallow feature F 0 ${F_0}$ from the given LR IR image (size 40 × 40 $40 \times 40$ pixels) for saving parameters: F 0 = H S F E M ( I L R ) , \begin{equation}{F_0} = {H_{SFEM}}({I_{LR}}),\end{equation} (2)where H S F E M ( · ) ${H_{SFEM}}( \cdot )$ represents the convolutional operation. Then, the extracted shallow feature F 0 ${F_0}$ is concatenated with the hidden state of each iteration and is fed into the MFFB for the discriminative feature extraction. At the t th $t{\rm{th}}$ iteration, the input of the MFFB is the concatenation of the hidden state from the former iteration h t − 1 $h^{t - 1}$ and the extracted shallow feature F 0 ${F_0}$ . Thus, the procedure of the MFFB can be formulated as: F 1 t = H M F F B ( Φ [ F 0 , h t − 1 ] ) , \begin{equation}F_{_1}^t = {H_{MFFB}}(\Phi [{F_0},{h^{t - 1}}]),\end{equation} (3)where H M F F B ( · ) ${H_{MFFB}}( \cdot )$ denotes the discriminative feature extraction function of MFFB, F 1 t $F_{_1}^t$ is the output of the MFFB at t th $t{\rm{th}}$ iteration, and Φ [ · ] $\Phi [ \cdot ]$ represents the channel-wise concatenation of F 0 ${F_0}$ and h t − 1 $h^{t - 1}$ . In this way, the deep features are recursively extracted and refined through the MFFB. The final RM only contains two layers, a 3 × 3 $3 \times 3$ convolution layer, and a parameter-free sub-pixel convolution layer. With RM, the LR features F 1 t $F_{_1}^t$ are upscaled to the desired size, given by: F r e c t = H R e c o n ( F 1 t ) , \begin{equation}F_{rec}^t = {H_{Recon}}(F_1^t),\end{equation} (4)where H R e c o n ( · ) ${H_{Recon}}( \cdot )$ represents the reconstruction function of the RM and F r e c t $F_{rec}^t$ is the output feature of RM at t th $t{\rm{th}}$ iteration. The output SR image I S R ${I_{SR}}$ of MFFN is the summation of the HR features and the upsampled image produced by the bilinear upsample kernel, expressed as: I S R t = F r e c t + f b i l i n e a r ( I L R ) , \begin{equation}I_{SR}^t = F_{rec}^t + {f_{bilinear}}({I_{LR}}),\end{equation} (5)where f b i l i n e a r ( · ) ${f_{bilinear}}( \cdot )$ represents the bilinear upsample kernel. As MFFN is constructed in the recursive framework, the L 1 ${L_1}$ loss function is chosen for network optimization. Given an IR image training set { ( I L R i , I H R i ) } i = 1 N $\{ (I_{LR}^i,I_{HR}^i)\} _{i = 1}^N$ , the training of the MFFN is minimized by the following loss function: L ( Θ ) = arg min 1 T ∑ t = 1 T I S R t − I H R t 1 , \begin{equation}L(\Theta ) = \arg \min \frac{1}{T}\sum_{t = 1}^T {{{\left\| {I_{SR}^t - I_{HR}^t} \right\|}_1}} ,\end{equation} (6)where N $N$ , Θ $\Theta $ , T $T$ , and ∥ · ∥ 1 ${\| \cdot \|_1}$ are the number of LR-HR image pairs, optimal parameters of the network model, total number of recursions, and L 1 ${L_1}$ norm, respectively. The loss function is optimized by the Adam optimizer [43]. More details of the training are described in Section 3.3. 3.2 Multi-path feature fusion block (MFFB) As illustrated in Figure 2, the MFFB is mainly composed of four parts: LFEB, SRFEB, CAB, and multiple feature fusion (MFF). Firstly, the shallow feature F 0 ${F_0}$ and the high-level feature h t − 1 $h^{t - 1}$ from the former iteration is concatenated as the input of MFFB at the t th $t{\rm{th}}$ iteration, F c o n c a t = [ F 0 , h t − 1 ] , \begin{equation}{F_{concat}} = [{F_0},{h^{t - 1}}],\end{equation} (7)where F c o n c a t ${F_{concat}}$ is the concatenated features, and [ · ] $[ \cdot ]$ signifies the operation of concatenation. F c o n c a t ${F_{concat}}$ is then passed through a bottleneck layer implemented by a 1 × 1 $1 \times 1$ convolution to reduce the computational cost. Here, the high-level feature h 0 ${h^0}$ is equal to the shallow feature F 0 ${F_0}$ when t = 1 $t{\rm{ = }}1$ . Therefore, the compressed feature F B N ${F_{BN}}$ is obtained by: F B N = H c o n v ( F c o n c a t ) , \begin{equation}{F_{BN}} = {H_{conv}}({F_{concat}}),\end{equation} (8)where H c o n v ( · ) ${H_{conv}}( \cdot )$ denotes the 1 × 1 $1 \times 1$ convolutional operation. After that, the F B N ${F_{BN}}$ is fed to the above-mentioned three branches (LFEB, SRFEB, and CAB) for further discriminative feature extraction. FIGURE 2Open in figure viewerPowerPoint The structure of the proposed MFFB. Concat, concatenation; Conv, convolution with filter 3 × 3; GAP, Global average pooling. LFEB, the linear feature extraction branch; SRFEB, the linear feature extraction branch; CAB, channel attention branch As for the LFEB, a 3 × 3 $3 \times 3$ convolutional layer without using any activation function is utilized to explore the linear features [44]. The extracted features f L F E B ${f_{LFEB}}$ of LFEB is expressed as: f L F E B = H c o n v ( F B N ) , \begin{equation}{f_{LFEB}} = {H_{conv}}({F_{BN}}),\end{equation} (9)where H c o n v ( · ) ${H_{conv}}( \cdot )$ denotes the convolutional operation. The SRFEB includes two consecutive shared-source residual units, and each of them contains a 1 × 1 $1 \times 1$ convolutional layer. The output feature f S R F E B ${f_{SRFEB}}$ is obtained by: f S R F E B = H c o n v ( H c o n v ( F B N ) + F B N ) + F B N . \begin{equation}{f_{SRFEB}} = {H_{conv}}({H_{conv}}({F_{BN}}) + {F_{BN}}) + {F_{BN}}.\end{equation} (10) Generally, the channel attention (CA) is used to exploit the feature interdependencies across various channels. Most of the previous works utilize CA as a part of module instead of incorporating it as an independent feature extractor. Unlike the previous works, the CAB is proposed to capture the interdependencies among different channels and is used to assign more weights to the discriminative features and fewer weights to its counterpart. Meanwhile, CAB is adopted to extract informative features and thus assign more weights to the features. Note that the attention-based features are sent into a 3 × 3 $3 \times 3$ convolution for further feature exploration. As illustrated in Figure 2, in the CAB, we firstly utilize global average pooling (GAP) to turn the channel-wise global spatial features into a channel descriptor. Next to the GAP is the 1 × 1 $1 \times 1$ and 3 × 3 $3 \times 3$ convolution layers. Besides, we adopt a gating mechanism via sigmoid function to fully exploit channel-wise dependencies from the extracted information. The above-mentioned process of CAB can be written as f S i g = f ( H c o n v ( H c o n v ( H G A P ( F B N ) ) ) ) , \begin{equation}{f_{Sig}} = f({H_{conv}}({H_{conv}}({H_{GAP}}({F_{BN}})))),\end{equation} (11)where f ( · ) $f( \cdot )$ , f S i g ${f_{Sig}}$ and H G A P ( · ) ${H_{GAP}}( \cdot )$ denote the sigmoid function, output of sigmoid function, and global average pooling function, respectively. Then the final channel attention features f C A B ${f_{CAB}}$ are obtained by rescaling the input: f C A B = f S i g ⊗ F B N , \begin{equation}{f_{CAB}} = {f_{Sig}} \otimes {F_{BN}},\end{equation} (12)where ⊗ $ \otimes $ denotes the element-wise multiplication. Subsequently, in MFF, the outputs of the three branches are concatenated, followed by a bottleneck layer ( 1 × 1 $1 \times 1$ convolution), as shown in Figure 2. The mathematical model of MFFB is defined as: F M F F = H M F F ( [ f L F E B , f S R F E B , f C A B ] ) , \begin{equation}{F_{MFF}} = {H_{MFF}}([{f_{LFEB}},{f_{SRFEB}},{f_{CAB}}]),\end{equation} (13)where H M F F ( · ) ${H_{MFF}}( \cdot )$ denotes the proposed MFFB, f L F E B ${f_{LFEB}}$ , f S R F E B ${f_{SRFEB}}$ and f C A B ${f_{CAB}}$ are the output of the three branches, respectively. Overall, MFFB benefits from the way of parameter sharing and full use of different types of information while reaching a good trade-off between the parameter and performance. 3.3 The proposed algorithm The training process of the proposed is shown in Algorithm 1. Algorithm 1. MFFN training process Input: HR-LR image pair { I L R i , I H R i } i = 1 N $\{ I_{LR}^i,I_{HR}^i\} _{i = 1}^N$ . Output: SR image { I S R i } i = 1 N $\{ I_{{\rm{S}}R}^i\} _{i = 1}^N$ . Parameter initialization: Batch size b $b$ , learning rate η $\eta $ , and Θ $\Theta $ , iteration T $T$ , and set i = 1 $i = 1$ . Step 1: The SFEM takes LR image { I L R i } $\{ I_{LR}^i\} $ as input, F 0 ${F_0}$ is extracted by Equation (2). Step 2: Calculate the upsampled image by bilinear upsample kernel, i.e., f b i l i n e a r ( I L R ) ${f_{bilinear}}({I_{LR}})$ . Step 3: Loop Start Step 3.1: For i = 1 , 2 , … , N $i = 1,2, \ldots ,N$ do Step 3.1.1: For t = 1 , 2 , … , N $t = 1,2, \ldots ,N$ do Step 3.1.2: F 1 t i $F_1^{ti}$ is extracted by Equation (3). Step 3.1.3: F R E C t i $F_{REC}^{ti}$ is extracted by Equation (4). Step 3.1.4: I S R t i $I_{SR}^{ti}$ is extracted by Equation (5). Step 3.1.5: Calculate by Equation (6) to update Θ $\Theta $ with Adam optimizer. Step 3.1.6: End for Step 3.2: End for Step 3: Loop End 3.4 Implementation details In MFFN, the number of filters in each convolution layer is set to 48. The reduction rate in the CAB is set as eight for saving model parameters. The PReLU [45] is employed as the activation function after each convolution layer except the convolution layers in the LFEB. During the MFFN training, the number of iterations is set as four. 3.5 Comparisons 3.5.1 Difference to deep recursive residual network (DRRN) DRRN mainly combines recursive learning with the residual network as the recursive block (RB) to reach the goal of reducing parameters for image SR. Moreover, the RB is simply stacked by several times. There are three different points between DRRN and MFFN. Firstly, the building block of DRRN is organized in a single-path manner, while MFFN incorporates MFFB that contains four pathways as the basic block. Secondly, the recursive learning only appears in RB, leading to a local parameter-sharing. However, MFFN applies the recursive learning to the entire network, where the parameters are shared across the whole network. Thirdly, MFFN uses a channel attention mechanism to capture the interdependencies between different feature channels, while no attention mechanism is adopted in DRRN. 3.5.2 Difference to super-resolution feedback network (SRFBN) The main differences between SRFBN and MFFN are described as follows. The first one lies in the building block. SRFBN mainly establishes the feedback block by combining the repeated upsampling and downsampling strategies with the dense connection. MFFN uses the MFFB to extract and refine the discriminative information instead. The second one is that the channel attention is employed in MFFN to enhance the discriminative ability of the network. CAB is designed as a branch in MFFB rather than a module in the trunk, which greatly improve the performance with fewer parameters and computation cost. However, the attention mechan