# A Spectral-Aware Convolutional Neural Network for Pansharpening
## 1 Introduction
Recently, convolutional neural networks (CNNs) have witnessed a sudden surge of interest in image resolution enhancement tasks such as super-resolution [36] and pansharpening [37], Pansharpening, to some degree, can be regarded as a super-resolution task, where pansharpening is generally a multiple input, single output problem, while in a traditional image, super-resolution is a single input, single output process. In [38], [PERSON] et al. proposed a super-resolution CNN (SRCNN) to reconstruct high-spatial-resolution images directly from low-spatial-resolution images in an end-to-end manner. Following the basic thread of SRCNN, [PERSON] et al. introduced CNN into pansharpening for the first time and proposed a pansharpening CNN (PNN) [39]. After that, [PERSON] et al. designed a deep residual pansharpening neural network (DRPNN) [40] and [PERSON] et al. proposed two detail injection-based pansharpening CNNs (DiCNNs), i.e. DiCNN1 and DiCNN2 [41]. Although the aforementioned CNN-based pansharpening methods achieve better performances than traditional methods, there are still two problems that have not been well resolved. First, most previous CNN-based methods do not focus on consideration of strong spectral correlations that occur in MS/HS data and thus may incur spectral distortion in the fusion results, which means that some spectra (or pixels) of the recovered high-resolution MS/HS images deviate from those of the ideal high-resolution MS/HS images. Second, most existing CNN-based methods treat pansharpening merely as a black-box process and lack clear interpretability. To deal with these problems, we propose a spectral-aware pansharpening neural network (SA-PNN) in this study. The major contributions of our work can be summarized as follows:  
* We create a two-branch structure, including the detail branch and the approximation branch, in SA-PNN which coincides with the interpretation of a detailed injection. The detail branch, which originated from catenation of the LR-MBO image and the PAN image, mainly fulfills the spatial detail resolution, while the approximation branch, which inputs merely the LR-MBOimage, collaborates with the detail branch to inject the details to yield the final HR-MBO image. Therefore, our SA-PNN follows the concept of detail injection, which is a routine that inspired many classical pansharpening methods and hence gains clear interpretability. * We use a spectral-aware strategy in SA-PNN to alleviate spectral distortion. The strategy involves mainly two aspects. On the one hand, spatial feature transforms (SFTs) are for the first time introduced into the pansharpening task to adjust spectra of the processed data adaptively to the observed scene. On the other hand, 3D convolution operations are used in the approximation branch which naturally accommodate the data resolving along the spectral dimension. Since spectral-aware strategies are adopted in SA-PNN, spectral processing or prediction is strengthened; thus, spectral distortion is expected to be reduced. The rest of this paper is organized as follows. Section 2 briefly introduces the related backgrounds. Section 3 describes the proposed SA-PNN model in detail. The experimental results and analyses are presented in Section 4. Finally, the conclusion is presented in Section 5.