# Learning an Efficient Convolution Neural Network for Pansharpening
## 2 Related Work  
### Linear Models in Pansharpening  
The observed satellite imageries MS (\(\mathbf{m}\)) and PAN (\(\mathbf{p}\)) are assumed as degraded observations of the desired high-resolution MS (HRMS), and the degradation process can be modelled as:  
\[\left\{\begin{array}{l}\mathbf{m}=(\mathbf{x}*\mathbf{k})\downarrow_{4}+\varepsilon_{ \text{MS}}\\ \mathbf{p}=\mathbf{x}*\mathbf{H}+\varepsilon_{\text{PAN}}\end{array}\right. \tag{1}\]where, \(\mathbf{x}*\mathbf{k}\) represents the convolution between the desired HRMS (\(\mathbf{x}\)) and a blurring kernel \(\mathbf{k}\), \(\downarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\) is a subsequent down-sampling operator with a scale of 4; \(\mathbf{H}\) is a spectral response matrix, which down-samples HRMS along the spectrum, \(\varepsilon_{\mathbf{MS}}\) and \(\varepsilon_{\mathbf{PAN}}\) are additive noise. Accordingly, the MBO method addresses pansharpening by forming an optimization function as:  
\[\hat{\mathbf{x}}=\arg\underset{\mathbf{x}}{min}\Big{\{}\alpha\|\mathbf{y}-\mathbf{k}*\mathbf{x}\|_ {2}^{2}+\beta\|G\Big{(}\mathbf{p}-\sum_{i=1}^{I}\omega_{i}\mathbf{x_{i}}\Big{)}\|_{2}^ {2}+\gamma\varphi(\mathbf{x})\Big{\}} \tag{2}\]  
in which, \(\hat{\mathbf{x}}\) and \(\mathbf{y}\) denote the pansharpened result and LRMS respectively; \(I\) is the number of spectral bands of \(\mathbf{x}\), \(\mathbf{x}_{i}\) is the \(i\)-th band of \(\mathbf{x}\), and \(\omega\) represents an \(I\)-dimensional probability weight vector that satisfies \(\sum_{i=1}^{I}\omega_{i}=1\) and indicates the linear nature of this model. \(G\) is a spatial difference operator to focus on high-frequency content, and \(\varphi\) denotes a prior term, which is used to regularize the solution space. The trade-off parameters \(\alpha\), \(\beta\), and \(\gamma\) are used to balance the contribution of each term in the model.  
For the CS-based pansharpening family, a low-resolution PAN image is formed by combining the available LRMS linearly. The generated low PAN image is transformed into another space, assuming the spatial structure is separated from spectral component. Subsequently, the extracted spatial information is replaced with the PAN image, and the fusion process is completed by transforming the data into the original space. A general formulation of CS fusion is given by:  
\[\hat{\mathbf{x}}_{k}=\mathbf{y}_{k}+\psi_{k}\Big{(}\mathbf{p}-\sum_{i=1}^{I}\omega_{i}\mathbf{ y}_{i}\Big{)} \tag{3}\]  
in which, \(\hat{\mathbf{x}}_{k}\) and \(\mathbf{y}_{k}\) are the \(k\)-th band of \(\hat{\mathbf{x}}\) and \(\mathbf{y}\), and \(\psi_{k}\) denotes the injection gain of the \(k\)-th band. For the MRA-based approach, the contribution of \(\mathbf{p}\) to \(\hat{\mathbf{x}}\) is achieved via linear decomposition, and the general form of MRA pansharpening method is defined as:  
\[\hat{\mathbf{x}}_{k}=\mathbf{y}_{k}+\phi_{k}(\mathbf{p}-\mathbf{p}*\mathbf{h}) \tag{4}\]  
where \(\mathbf{h}\) is the corresponding decomposition operator, and \(\phi_{k}\) denotes injection gain of the \(k\)-th band.  
All the above-mentioned approaches extract structural information of the specific bandwidth from the PAN image with linear models and inject the extracted spatial details into the corresponding LRMS band. However, the spectral coverage of PAN and LRMS images are not fully overlapped and information extracted in a linear manner may lead to spectral distortions. Furthermore, the transformation from LRMS to HRMS is complex and highly nonlinear such that linear fusion models can rarely achieve satisfactory accuracy. In order to further improve the fusion performance, a nonlinear model is needed to fit the merging process. Therefore, deep-learning-based methods are taken into consideration.