## 3 Proposed Method
### Feature Extraction Module  
In the DCMFN, we introduced the attention mechanism. The attention mechanism addresses the spatial and spectral inconsistencies through spatial attention, spectral attention, and multi-modal fusion. Spatial attention generates feature maps, calculates position-based weights, and can adaptively adjust to deal with spatial differences such as the shift of object positions. Spectral attention extracts spectral features, assigns band-based weights, and resolves spectral variation issues, such as those caused by lighting conditions or sensors. Multi-modality fusion combines spatial and spectral information by leveraging the attention mechanism and adjusts the weights of each type of information according to the task requirements, which is crucial for tasks like object detection and material classification. This comprehensive approach effectively overcomes the challenges posed by the spatial and spectral inconsistencies in the data analysis. Figure 1(a) illustrates the proposed framework referred to as the PFEM. Drawing inspiration from [PERSON] et al. (2020), we design the PFEM to incorporate four Self-Attention Blocks (SABs), which is shown in Figure 3. SABs serve as an effective mechanism for feature extraction, particularly for sequential and image data. Each SAB consists of two-layer normalization processes, a self-attention layer, and two multi-layer perceptron (MLP) layers. Figure 3 shows the specific structure of SAB. It computes similarity scores (attention weights) among the elements in the sequence. These scores are then used to take a weighted  
Figure 1: Architecture of the dual-stream cross-modality fusion network. average of the input features, generating a contextual representation for each position. The calculation is as follows:  
\[Att_{self}(Q,K,V)=Softmax(\frac{QK^{T}}{\sqrt{d}})V \tag{1}\]  
The basic idea is to determine the attention weights by calculating the similarity between the query and the key, and then applying those weights to the value to obtain a weighted output. In both PFEM and MFEM, multiple SABs are used to extract features from the PAN and MS images, respectively. In each SAB, the embedding dimension is set to 64, with 8 attention heads, each having a dimension of 8. The window size is set to 4. For the MLP layer, the input is first normalized, and the input feature dimension is 64. The feedforward network has an input dimension of 64, a hidden layer dimension of 256, and an output dimension of 64. Finally, a residual connection is applied by adding the input and the output of the feedforward network. The SAB blocks capture long-range dependencies through a shifted window mechanism. In this mechanism, positional encoding becomes crucial, as it allows the model to maintain spatial awareness while performing attention calculations within local windows. To address this, we introduce the relative indices matrix, which encodes the relative distances between positions within the window. We adjust the positional encoding matrix to avoid negative indexing by adding (W\(-\)1) (where (W) is the window size). This ensures that positional encoding can consistently provide effective spatial relationships as the window shifts across the image. In the case where relative indices are used, the shifting of the attention window can create gaps or cause ambiguity in how distant pixels or patches are spatially related. To resolve this issue, we introduce positional encoding through a learnable embedding matrix. The pos_embedding initializes a 2D positional encoding matrix, which helps the model learn spatial relationships across the image. This matrix, with a size of (7 \(\times\) 7) for a window size (W = 4), represents relative position embeddings within the attention window, allowing the model to capture spatial information even when working with shifted windows. The application of multiple SABs in feature extraction offers several advantages.