# Progressive Multi-Scale Fusion Network for Light Field Super-Resolution
## 3 Progressive Multi-Scale Fusion Network
### Overview  
Our PMFN network is shown in Figure 2, which is designed as an all-to-one structure. This network consists of three parts: feature extraction (\(f_{\text{FE}}\)), feature fusion (\(f_{\text{FF}}\)), and reconstruction (\(f_{\text{RC}}\)). Specifically, the residual receptive field block (ResRFB) is designed to extract the shallow features of each SAI, which consists of two RFBs. Given \(L_{\text{LR}}\) as input, the \(L_{\text{LR}}\) is first fed to a \(1\times 1\) convolution to generate the initial features. These features are processed by the ResRFB and residual block (ResBlock), respectively. In this part, the shallow features (\(G_{1}\)) are extracted by the \(f_{\text{FE}}\), which can be expressed by the following,  
\[G_{1}=f_{\text{FE}}(L_{\text{LR}}), \tag{1}\]  
where \(G_{1}\in\mathbb{R}^{N\times\mathbb{C}\times H\times W}\) represents the shallow features, \(C\) is the feature depth, and \(N=U\times V\) is the number of SAIs. We divide the \(G_{1}\) into a two part reference-view feature and several auxiliary-view features, which can be expressed as \(G_{1}^{\text{ref}}\in\mathbb{R}^{\mathbb{C}\times H\times W}\) and \(G_{1}^{\text{aux}\times}\in\mathbb{R}^{\mathbb{C}\times H\times W}\). Specifically, \(G_{1}^{\text{ref}}\) is arbitrarily selected from \(G_{1}\) and \(G_{1}^{\text{aux}\times}\) are composed of the remaining features of \(G_{1}\). The number of \(G_{1}^{\text{ref}}\) is 1 and the number of \(G_{1}^{\text{aux}\times k}\) is \(N-1\). Then, the \(G_{1}^{\text{ref}}\) combined with each of \(G_{1}^{\text{aux}\times k}\) is put into the PFFB, respectively. Notably, the key component of our network is the \(f_{\text{FF}}\) (PFFB). Through this component, the fusion feature (\(G_{2}\)) is generated, connected with informative features of different auxiliary views, i.e.,  
\[G_{2}=f_{\text{FF}}\Big{(}\Big{[}G_{1}^{\text{ref}},G_{1}^{\text{aux},1}\Big{]},\dots,\Big{[}G_{1}^{\text{ref}},G_{1}^{\text{aux},k}\Big{]}\Big{)}, \tag{2}\]  
where \([\cdot]\) denotes the connection operation, and \(G_{2}\in\mathbb{R}^{\mathbb{C}\times H\times W}\) represents the final fusion feature. After generating the \(G_{2}\), the feature enhancement block (FEB) is used to bridge the gap between the obtained reference view and the given HR reference view. Then, the residual map (\(G_{3}\)) is fine-tuned. This block is very useful to obtain more distillation of valid information and promote more HR reconstruction. The architecture of the FEB is shown in Figure 2e. Finally, the \(G_{3}\) adds a coarse HR image processed by the bicubic interpolation (_Up_) to generate an HR image (\(L_{\text{HR}}^{\text{ref}}\)). This process can be simply expressed as  
\[\begin{split} G_{3}&=f_{\text{RC}}(G_{2}),\\ L_{\text{HR}}^{\text{ref}}&=G_{3}+Up\Big{(}L_{ \text{LR}}^{\text{ref}}\Big{)},\end{split} \tag{3}\]  
where \(G_{3}\) denotes the residual map, and \(L_{\text{LR}}^{\text{ref}}\in\mathbb{R}^{1\times H\times W}\) is an LR reference image, respectively. For other auxiliary views (\(L_{\text{HR}}^{\text{aux}}\)), they will share the same weights of this network and be combined with different \(L_{\text{LR}}^{\text{aux}}\) processed by the bicubic interpolation. Eventually, all the HR SAIs are generated, hence the \(L_{\text{HR}}\).  
In summary, our all-to-one structure can capture the absent information in the Ref image through other Aux images, which is shown in Figure 3. Moreover, the all-to-all structure can not find unique information from individual views in that the average error over all views is used to optimise during the process of training network [13]. Thus each view in our structure can directly and efficiently incorporate the information from all views.