# Progressive Multi-Scale Fusion Network for Light Field Super-Resolution
## 1 Introduction  
Light field (LF) cameras are able to record the spatial and angular information of a scene. This camera technology has been successfully used in many applications, such as VR [1, 2], 3D reconstruction [3, 4], and saliency detection [5, 6]. However, the resolution limitation of camera sensors inhibits the development of LF imaging technology. The higher the angular resolution required of the LF, the lower the spatial resolution obtained for each view. Consequently, LF super-resolution (LFSR) algorithms are widely investigated to retrieve high-resolution (HR) information from low-resolution (LR) images. Compared with the single image SR (SISR) [7, 8], LFSR achieves SR images by using angular and spatial information from different LF views. Based on the complex geometrical structure of LF images, some traditional methods based on disparities have been proposed [9, 10]. However, the performance of these methods relies on the accuracy of disparity estimation, and their computational costs are very high. Although optimisation-based methods are constantly researched, obtaining accurate disparities from LR sub-aperture images (SAIs) is still challenging. With the development of deep learning, a straightforward way is fine-tuning the network parameters of SISR. However, these methods can hardly preserve the complex 4D structure of LF. Recently, learning-based methods [11, 12, 13, 14, 15, 16] are utilised to effectively integrate the spatial and angular information, and improve the SR reconstruction. Thework in [11] could reconstruct the final high-resolution (HR) SAIs, which simply used the stacked horizontal and vertical directional views. After the structure of the proposed residual network, [PERSON] et al. [12] proposed the resLFs, which could extract complementary information by stacking the SAIs in four directions. Recently, [PERSON] et al. [13] proposed an all-to-one (ATO) model to generate SR LF images, which combined the reference view and surrounding auxiliary views via combinatorial geometry embedding. Moreover, [PERSON] et al. [14] utilised a different convolution to extract spatial and angular information for LFSR. More recently, [PERSON] et al. [15] designed a view and channel attention to fuse hierarchical features and distillate valid information, which could reconstruct HR LF images. [PERSON] et al. [16] proposed a deformable convolutional network to address the problem of disparities among LF images. Although most of these networks can incorporate spatial and angular information to achieve high accuracy for LF reconstruction, the disparities among different SAIs are still under-investigated. Furthermore, due to the occlusions and non-Lambertian reflections in LF, an image from one view may contain distinctive details compared with images from other views, as shown in Figure 1. The structures of all-to-all networks are not well utilised informative information from auxiliary views for further performance improvement. Consequently, there are two problems existing in LFSR methods, which align the disparities among LF views and supplement the sufficient complementary information. In order to handle these issues, we propose a progressive multi-scale fusion network, PMFN, which has an encoder-and-decoder structure with a progressive multi-scale convolution. Our method is designed to fully use the complementary information from all auxiliary images and implicitly address the problem of disparities among the reference view and auxiliary views. Specifically, we propose a progressive feature fusion block (PFFB) with three dual-branch multi-scale fusion modules (DMFMs). Each DMFM has a dual-branch structure. The two branches of DMFB interact with the extraction feature with different receptive fields, respectively. Then, we concatenate the output features of these two branches with informative information. The PFFB is mainly constructed from three DMFMs. These DMFMs adopt a dense skip connection to strengthen the long-term information from previous DMFMs.