# Multispectral Target Detection Based on Deep Feature Fusion of Visible and Infrared Modalities
## 3 Proposed Methodology
### Hybrid Fusion Mechanism
#### 3.2.1 Cross-Modal Feature Enhancement  
In this study, we introduce the CFE module to improve complementary feature extraction between RGB and thermal modalities. This module enables a single modality to learn feature information from the other modality at a global level using multiple cross  
Figure 3: The Attention-Enhanced Feature Fusion (AEFF) framework.  
Figure 2: Architecture of the YOLO-MEDet model. For performance considerations, the C3-PSA module is applied only to the first C3 layer of the model.  
attention layers and residual structures. To fully leverage cross-modal complementarity, we implement the CFE module separately on the RGB and thermal branches, training their parameters independently. This strategy yields two distinct enhancement paths, allowing each modality to capture complementary information without parameter sharing.  
Specifically, for an input RGB feature map \(F_{R}\) and a thermal feature map \(F_{T}\in\mathbb{R}^{H_{X}W\times C}\), we first flatten both and incorporate a learnable position embedding to obtain positional information, which is represented as tokens \(T_{R}\) and \(T_{T}\in\mathbb{R}^{HW\times C}\). Given that RGB-thermal image pairs are often imperfectly aligned, we employ separate CFE modules for the RGB and thermal branches to extract and enhance complementary information. The output of the CFE module for the thermal branch is given by  
\[\hat{T}_{T}=\mathcal{F}_{\text{CFE-}T}(T_{R},T_{T}) \tag{1}\]  
where \(T_{R}\) and \(T_{T}\) represent the RGB and thermal tokens input to the module, respectively. Here, \(\hat{T}_{T}\) denotes the augmented thermal features, and \(\mathcal{F}_{\text{CFE-}T}(\cdot)\) represents the CFE module for the thermal branch.  
Within the CFE module, the thermal token \(T_{T}\) is first projected onto the value matrix \(V_{T}\) and key matrix \(K_{T}\), while the RGB token \(T_{R}\) is projected onto the query matrix \(Q_{R}\) as follows:  
\[V_{T}=T_{T}W^{V},\quad K_{T}=T_{T}W^{K},\quad Q_{R}=T_{R}W^{Q} \tag{2}\]  
where \(W^{V},W^{K},W^{Q}\in\mathbb{R}^{C\times C}\) are learnable linear mappings. Next, the correlation matrix is computed via a dot product operation, which is normalized using the softmax function to measure the similarity between RGB and thermal features, and subsequently multiplied by \(V_{T}\), yielding  
\[Z_{T}=\text{softmax}\left(\frac{Q_{R}K_{T}^{T}}{\sqrt{D_{K}}}\right)\cdot V_{T}. \tag{3}\]  
To capture cross-modal dependencies from different perspectives, we employ a multihead cross-attention mechanism with eight parallel heads. The resulting matrix \(Z_{T}\) is projected back to the original feature space through a nonlinear transformation and combined with residual connectivity to produce the fused features as follows:  
\[T_{T}^{\prime}=\alpha\cdot Z_{T}W^{O}+\beta\cdot T_{T} \tag{4}\]  
where \(W^{O}\in\mathbb{R}^{C\times C}\) is the projection matrix, and \(\alpha\) and \(\beta\) are learnable residual coefficients. Finally, a feedforward network (FFN) comprising two fully connected layers further enhances global information modeling:  
\[\hat{T}_{T}=\gamma\cdot T_{T}^{\prime}+\delta\cdot\text{FFN}(T_{T}^{\prime}) \tag{5}\]  
where \(\gamma\) and \(\delta\) are additional learnable coefficients, which are initialized to 1 during training. Similarly, a corresponding CFE module is applied to the RGB branch, with the output defined as  
\[\hat{T}_{R}=\mathcal{F}_{\text{CFE-}R}(T_{R},T_{T}) \tag{6}\]  
Through these two independent enhancement paths, the enhanced RGB features \(\hat{T}_{T}\) and thermal features \(\hat{T}_{R}\) effectively capture complementary global information, significantly improving the model's cross-modal characterization capability and robustness.