## 2 Materials and Methods
### Proposed Coupled CNN Fusion Approach  
The proposed CNN-based fusion method can be summarized in Figure 1, in which two CNN networks with different numbers of convolutional layers are applied to HS and MS image patches, respectively. In practice, the HS image will be first upscaled to the size of the MS image by bilinear interpolation as mentioned above. Afterwards, both HS and MS image patches are selected by partitioning the two images through a pixel-by-pixel fashion with a stride 1 \(\times\) 1 and fed into the two sub-networks, respectively. As a matter of fact, MS image has such a higher spatial resolution than HS images that they are substantially subject to different spatial scales. Therefore, the input of our network is composed of different spatial and spectral sizes of HS and MS image patches, which can be denoted by \(p_{H}\times p_{H}\times\Lambda\) and \(p_{M}\times p_{M}\times\lambda\), \(p_{H}<p_{M}\) and \(\Lambda>\lambda\) in general in order to fully exploit the structure information of HR image and spectral information of HS images. For the HS and MS sub-networks, different numbers of convolutional layers are applied on the image patches, respectively, which can be generally determined as [(\(p_{H}\)-1)/2] and [(\(p_{M}\)-1)/2], respectively, where [\(\cdot\)] denotes the largest integer that is less than or equal to the given number. Each convolutional layer is followed by a batch normalization (BN) layer and an active function with a rectified linear unit (ReLU), individually. Therefore, the response of the \(l\)-th layer can be denoted as  
\[A^{l}=max\Big{(}\mathbf{BN}\Big{(}\mathbf{W}^{l}*\mathbf{A}^{l-1}+\mathbf{b}^{l}\Big{)},0\Big{)} \tag{4}\]  
where \(\mathbf{W}^{l}\) and \(\mathbf{b}^{l}\) are the weight and bias matrices of \(l\)-th layer; \(\mathbf{A}^{l}\) denotes the output of \(l\)-th layer; \(*\) denotes the convolution operation; BN(\(\cdot\)) denotes the batch normalization operation, which is used to overcome the issue of internal covariate shift and accelerate the training of network. Consequently, both sub-networks yield a 1 \(\times\) 1 size of output feature maps. Typically, in this paper, the patch sizes of HS and MS images are 5 \(\times\) 5 \(\times\)\(\Lambda\) and 9 \(\times\) 9 \(\times\)\(\lambda\), respectively, thus the numbers of convolutional layers are 2 and 4, respectively. Figure 1: The main framework of the proposed method. For both sub-networks, the convolutional filter sizes are all set to 3 \(\times\) 3, which is commonly used by related works, and the convolutional filter numbers are set to 32. Subsequently, the output feature maps of both HS and MS sub-networks are concatenated to form 64 feature maps and fed into an additional convolutional layer with \(\Lambda\) filters and 1 \(\times\) 1 filter size to predict the residuals, resulting in 1 \(\times\) 1 \(\times\) \(\Lambda\) values for each pixel. The residuals, namely the predicted details, which are denoted by \(\mathbf{D}_{k}\), \(k\) = 1, 2,..., \(\Lambda\), are finally added to the primary HS image to reconstruct the expected HR HS image. It should be noted that the input patches of HS and MS sub-networks include the neighbor block of a pixel, and the output of the network is the residuals of the current pixel. In this way, the network will automatically utilize the collaborative spatial and spectral information of HS and MS images, respectively, and infer the details of the current pixel accurately. In order to effectively train the network, both HS and MS images are decimated by the factor \(s\) (\(s\) = \(M/m\)). Suppose the decimated HS image is denoted by \(\mathbf{X}^{\text{low}}\)\(\in\)\(\mathbb{R}^{m\times n\times\Lambda}\). As mentioned above, \(\mathbf{X}^{\text{low}}\) will first be upscaled to the size of the decimated MS image by bilinear interpolation. For conciseness, it is also denoted by \(\mathbf{X}^{\text{low}}\), namely \(\mathbf{X}^{\text{low}}\)\(\in\)\(\mathbb{R}^{m\times n\times\Lambda}\).