# A Spectral-Aware Convolutional Neural Network for Pansharpening
## 3 Proposed Method  
In this section, we develop a novel pansharpening neural network, called SA-PNN, which not only has clear interpretability but also is able to effectively alleviate spectral distortion. The overall structure of our SA-PNN is graphically shown in Figure 1. As can be observed, the network comprises two major branches. One is the detail branch which extracts spatial details; the other is the approximation branch to extract approximations which collaborate the detailed information from the detail branch to build the final HR-MBO image. The detail branch takes the stacked PAN image of size \(H_{P}\times W_{P}\) and the LR-MBO image of size \(H_{M}\times W_{M}\times C\) as its input while using 2D convolutions to fulfill convolutional layer operations, which is designed with PAN image being assumed to contain most of useful spatial details and 2D convolutions being able to effectively collect spatial information. In contrast, the approximation branch takes only the LR-MBO image as its input and distills spectral approximation which collaborates the detailed information yielded in the detail branch to produce the final HR-MBO image of size \(H_{P}\times W_{P}\times C\). Since 2D convolutions are unable to effectively build representation along the spectral dimension, they thus may incur spectral distortion; in such an approximation branch, 3D convolutions are used in convolutional layers to strengthen the spectral processing. The detail branch and the approximate branch form the basis of the detailed injection-based structure of our SA-PNN. Apart from the 3D convolution operation for convolution layers, the SFT strategy is also introduced in the approximate branch of SA-PNN (as show in Figure 1). SFT is originally designed to acquire the semantic categorical prior for color image enhancement [43]. We argue here that SFT can be used to adaptively adjust spectra in terms of the observed scene, to play a crucial role in the detail injection; and for the first time (to the best of our knowledge), to introduce it into pansharpening task. More specifically, in our SA-PNN, SFT provides an affine transformation for the convolutional feature maps based on a modulation parameter pair \((\alpha,\beta)\), where \(\alpha\) and \(\beta\) are parameter maps conditioned on the observed scene. The transformation is carried out by a scaling and shifting operation as follows:  
\[\mathbf{SFT}(F|\alpha,\beta)=\alpha\odot F+\beta, \tag{3}\]  
where \(\mathbf{F}\) denotes the feature maps and \(\odot\) represents element-wise multiplication. As shown in Figure 2, for the \(l\)th layer in the approximation branch, we first obtain a modulation parameter pair \((\alpha^{l},\beta^{l})\) from the feature maps of the \(l\)th convolutional layer \(\mathbf{D}^{l}\) (the \(l\)th green cube in Figure 2). Then, we apply the element-wise affine transformation to the feature maps \(\mathbf{F}^{l}\) (the \(l\)th blue cubes in Figure 2) according to (4):  
\[\widetilde{\mathbf{F}}_{k}^{l}=\alpha_{k}^{l}\odot\mathbf{F}_{k}^{l}+\beta_{k }^{l}, \tag{4}\]  
where \(\alpha_{k}^{l}\) and \(\beta_{k}^{l}\) denote the \(k\)th modulation parameter pair at the \(l\)th layer, \(\mathbf{F}_{k}^{l}\) represents the \(k\)th feature map of the the approximate branch at the \(l\)th layer, and \(\widetilde{\mathbf{F}}_{k}^{l}\) indicates the result after detail injection. It is noteworthy that both \(\alpha_{k}^{l}\) and \(\beta_{k}^{l}\) are 2D data while \(\mathbf{F}_{k}^{l}\) is volumetric data; thus, the operation in (4) actually applies the affine transformation to each band of \(\mathbf{F}_{k}^{l}\) according to the same parameter pair \((\alpha_{k}^{l},\beta_{k}^{l})\). As determined by the detail branch which carries high spatial details, \(\alpha_{k}^{l}\) and \(\beta_{k}^{l}\) carry spatial information in response to the observed scene. Therefore, with the calculation in (4), the feature map data are modulated spatially conditioned on the observed scene.