# Progressive Multi-Scale Fusion Network for Light Field Super-Resolution
## 3 Progressive Multi-Scale Fusion Network
### Progressive Feature Fusion Block (PFFB)  
After the feature extraction, the purpose of the PFFB is to implicitly align the disparities between the reference-view and auxiliary-view features. Moreover, this block can effectively exploit the complementary information among LF views. Here, we propose a structure of encoder and decoder, which embeds a progressive receptive field. Figure 2d shows that the basic and core component is the feature-fusion block (FFB), which contains three DMFMs. With this structure, it can map the pairs of features from the reference view and auxiliary views to higher dimensions for fusion by the encoder, and the receptive fields with different scales are suitable for extracting distinctive information from feature maps with different sizes. In this paper, we take \(t\)-th DMFMs to perform complementary information fusion. Without loss of generality, the determination of the number of DMFMs is demonstrated in Section 4.4. Figure 4: Architecture of RFB. This block is the basic component of ResRFB. Figure 3: Illustration of supplementing complementary information. Here, a \(L_{LR}\) (\(L_{LR}\in\mathbb{R}^{3\times 3\times H\times W}\)) is used as an example. We choose the reference view (\(U=2,V=2\)), and remaining views are used as auxiliary views. For better comprehension, complementary information from different auxiliary views is visually represented as stars with different colours. Note that the information is added to the blue box in Ref image. The result are shown in the zoomed-in region. As shown in Figure 2d, the reference-view feature is stacked respectively with the auxiliary-view features to construct the feature pairs, and these pairs are first concatenated and fused by a \(1\times 1\) convolution layer in succession. The output of this convolution is \(G_{1}^{\text{fu,k}}\). Here, we use \(t\)-th DMFMs to generate a fusion feature, which can bring great benefits to implicit feature alignment and feature fusion. The details of the FFB are shown in Figure 5. Specifically, each DMFM has a dual-branch structure mainly consisting of the encoder and decoder convolutions to execute upscaling and downscaling operations. In each branch, we insert two convolutions with different receptive fields in front of the encoder and decoder convolutions to capture the multi-scale information. Then, we concatenate these two branches to obtain the informative high-level feature, \(G_{1}^{1,k}\). This feature can be specifically expressed as  
\[G_{1}^{1,k}=f_{\text{DMFM}}^{1}\Big{(}G_{1}^{\text{fu,k}}\Big{)}, \tag{4}\]  
where \(f_{\text{DMFM}}^{1}\) represents the operation of our first DMFB, \(k=1,2,\dots\), and the output feature is \(G_{1}^{1,k}\in\mathbb{R}^{C\times H\times W}\). In our PMFN, three DMFMs are used with a dense connection to fully exploit multi-level features and preserve important information. For the second DMFM, we concatenate \(G_{1}^{\text{fu,k}}\) and \(G_{1}^{1,k}\) as input and feed them into a \(1\times 1\) convolution to process features. That is,  
\[G_{1}^{1^{*},k}=f_{\text{Conv}}^{1}\Big{(}\Big{[}G_{1}^{\text{fu,k}},G_{1}^{1, k}\Big{]}\Big{)}, \tag{5}\]  
where \(f_{\text{Conv}}^{1}\) is a \(1\times 1\) convolution. Similarly, we can obtain the output of the second DMFM, which is  
\[G_{1}^{2,k}=f_{\text{DMFM}}^{2}\Big{(}G_{1}^{1^{*},k}\Big{)}, \tag{6}\]  
where \(f_{\text{DMFM}}^{2}\) denotes the second DMFM. Due to the structure of dense connections, the input of the third DMFM consists of three parts: \(G_{1}^{\text{fu,k}}\), \(G_{1}^{1^{*},k}\), and \(G_{1}^{2,k}\). We use a \(1\times 1\) convolution to process these features and feed the output into the third DMFM, which can be denoted by  
\[G_{1}^{2^{*},k}=f_{\text{Conv}}^{2}\Big{(}\Big{[}G_{1}^{\text{fu,k}},G_{1}^{ 1^{*},k},G_{1}^{2,k}\Big{]}\Big{)}, \tag{7}\] \[G_{1}^{3,k}=f_{\text{DMFM}}^{2}\Big{(}G_{1}^{2^{*},k}\Big{)},\]  
Figure 5: Architecture of the three DMFMs in the FFB. Specifically, the encoder-and-decoder structure consists of a convolution and a deconvolution.