# Panchromatic and Hyperspectral Image Fusion Using Ratio Residual Attention Networks
## 3 Methods
### Residual Attention Network  
A significant feature of the residual network architecture is the skip identity connections within the residual blocks, which allow the input signal to propagate directly from any lower layer to higher layers during forward propagation [28]. For a given input, the weight layers of the convolutional neural network perform transformation functions on the input, as depicted in Figure 2. In the residual unit, the network's input \(x\) is directly added to the transformed output, yielding the result as shown in the following equation:  
\[h(x)=f(x)+x. \tag{7}\]  
Then, the result is activated by an activation function. Since the input bypasses the transformation layer, this connection is referred to as a \"skip identity connection\". Consequently, the transformation function in the residual unit is decomposed into an identity term and a residual term, where the residual map needs to be learned, as illustrated in the figure on the right. Figure 2: Standard and residual modules commonly used in neural networks. While the residual connection shown on the right helps address vanishing gradient issues and facilitates deep network training, this structure is well-established. In this work, we further enhance it with spatial attention mechanisms to selectively emphasize important spatial regions during hyperspectral image fusion. In the previous section, the initial ratio image obtained through the ratio transformation method was found to be rough and insufficient to accurately depict the degraded panchromatic image. Therefore, in this section, a residual attention network is designed to fine-tune the initial ratio image. The network consists of two cascaded residual spatial attention modules for extracting spatial features. The residual spatial attention module is composed of convolution operations, spatial attention operations, and skip connections. As shown in Figure 3, the input to the residual spatial attention module is \(F_{n-1}\), and the output is \(F_{n}\circ F_{n-1}\). First, two \(\mathrm{S}\times\mathrm{S}\) convolution layers are applied, and the output feature \(U\in\mathbb{R}^{H\times\mathrm{W}\times\mathrm{C}}\), where \(\mathrm{S}\times\mathrm{S}\) represents the size of the convolution kernel. Let \(U=\left[u^{1,1},u^{1,2},\ldots,u^{p,q},\ldots,u^{H,W}\right]\), where \(u^{p,q}\in\mathbb{R}^{\mathrm{C}}\) represents the feature vector at the spatial position \((p,q)\). After feature extraction with the two convolution layers, a spatial attention operation is performed. \(M_{sa}\) represents the spatial attention mask, which adaptively assigns weights to the extracted feature matrix. To generate the spatial attention mask, a \(1\times 1\) convolution layer with weight \(w_{sa}\) operates on \(U\). Then, a Sigmoid activation function \(\sigma(\cdot)\) is applied to rescale the attention map to the range \([0,1]\). This process can be expressed as follows:  
\[M_{sa}=\sigma(w_{sa}*U), \tag{8}\]  
where \(*\) represents the convolution operation. Using the generated mask, the feature set \(U\) can be spatially recalibrated, as expressed by the following equation:  
\[U_{sa}=f_{sa}(U,M_{sa}), \tag{9}\]  
where \(f_{sa}()\) represents the element-wise multiplication operation between the spatial positions of \(U\) and their corresponding spatial attention weights, which is the output of the spatial attention mechanism. Additionally, each residual spatial attention module employs skip connections, enabling the network to effectively combine both the initial and high-level features. Therefore, the output of the \(n\)-th residual spatial attention module is obtained by the following equation:  
\[F_{n}=F_{n-1}+U_{sa}. \tag{10}\]  
In the residual spatial attention network, the kernel sizes of the two cascaded residual spatial attention modules are \(5\times 5\) and \(3\times 3\), respectively. The number of channels in the corresponding extracted feature maps is 512 and 256, respectively.