## 3 Proposed Method
### Multi-Scale Feature Fusion Module (MFFM)  
The importance of multi-scale feature information in the image reconstruction process has been proven by the successive pyramid super-resolution networks [13; 20]. However, the pyramid methods mentioned above reconstruct high-resolution images only from adjacent layers, limiting the FSR performance. To further utilize the multi-scale feature information and enable the network with better feature representation capabilities, we introduce the Multi-scale Feature Fusion Module (MFFM), which is shown on the bottom side of Figure 1.  
The first step of the MFFM is to unify the size of multi-scale feature maps. Noticing that the magnification scale of adjacent layers in the proposed method is always \(2\), we introduce a \(3\times 3\) convolution layer with a stride of \(2\) and a \(6\times 6\) transposed convolution layer with a stride and padding of \(2\) for \(/2\) down-scale and \(\times 2\) up-scale, processes respectively. Please note that the \(/M\) down-scale and \(\times N\) up-scale mentioned here represent the M times downsampling and N times upsampling, respectively. Furthermore, for a larger magnification scale like \(/4\) or \(\times 4\), double \(3\times 3\) convolution or \(6\times 6\) transposed convolution layers are applied, etc. Considering that the MFFM is trained for residual compensation for multi-scale feature maps, a single convolution or transposed convolution layer is applied here to modify the feature map size instead of the PRUM/PRDM for simplicity. After resizing the multi-scale feature maps to a uniform size, they are concatenated to undergo a full connection layer and then fed to the CA network to highlight the essential channels. Finally, the well-handled multi-scale features are integrated with the target feature map layer from the encoding stage.  
The Pixel-Related Up/Downsample Module (PRUM/PRDM) aims to establish direct relationships among adjacent pixels for better face structure preservation and further strengthen the overall image reconstruction performance, whose structure is illustrated in Figure 5.  
The PRDM consists of three layers. Firstly, a \(3\times 3\) convolutional layer with a padding of 1 and a stride of 2 is applied to simulate the image degradation process. The kernel size here is set to \(3\times 3\) due to the fact that multiple \(3\times 3\) kernels from the multi-scale encoding-decoding network can simulate large kernel sizes while reducing computational complexity. After that, a Pixel Unshuffle layer instead of a convolutional layer is utilized to preserve as much original pixel information as possible. Finally, a full connection layer is applied to extract the vital image features. After the PRDM, the feature map channel doubles while the feature map size halves.  
To establish direct relationships among adjacent pixels in the upsampling process, we introduce the PRUM, whose central part is the Pixel-Related Deconvolution (PRD) layer. The structure of the module is illustrated in Figure 4(b). As we all know, pixel-Os obtained from /2 down-scale are not always their original values due to the complicated degradation process. Therefore, we first applied a \(2\times 2\) convolutional layer to obtain the real pixel-As. Then, the diagonal pixel-Bs are calculated by their corresponding As with another \(3\times 3\) convolutional layer with a padding of 1. After the above processing, the Cs and Ds are surrounded by known pixels and can be obtained by two separate \(3\times 3\) convolutional layers. Based on the relative pixel position, the calculated pixel-As, Bs, Cs, and Ds are highly related and structured, which assists the PRUM in establishing direct relationships among adjacent pixels. After the PRUM, the feature map channel halves while the feature map size doubles, which is the exact opposite of the PRDM in the encoding stage.