# RMFNet: Redetection Multimodal Fusion Network for RGBT Tracking
## 3 Our Method
### The Fusion Strategy  
Visible and thermal infrared images have different ways of perceiving and expressing target information and have different advantages under other conditions. Modality fusion can comprehensively utilize the benefits of both modalities, improve the precision and robustness of object tracking, enhance the tracker's resistance to environmental changes and noise, and improve the tracking speed by reducing unnecessary calculations. This paper employs two modality fusion methods, medium-term feature-level fusion and late-term decision-level fusion. (1) Medium-Term Feature-Level Fusion  
The feature-level fusion tracking algorithm first requires feature extraction using convolutional neural networks for visual and thermal infrared images, respectively, and then feature fusion using some designed fusion rules to obtain fused features of the two modalities, then tracking using this fused feature. As shown in Figure 2, our fusion branch aims to exploit the shared information between the two modalities fully. To better utilize the shared information, we perform feature-level fusion operations on the two features outputted by the fusion branch. In this paper, we propose a multimodal feature fusion module based on ECA attention (EFM), which can adaptively compute the contribution of the two modality features to target tracking and obtain the weighted fusion feature based on the relative reliability between modalities. The specific details of the EFM module are illustrated in Figure 3. The EFM module can reduce noise and redundant information in the fusion process, obtain more robust feature representation, and improve tracking accuracy. Figure 2: RMFNet architecture, including thermal infrared branch, visible light branch, and fusion branch. Inspired by the attentional fusion module proposed in AFF [26], we suggest an improved feature fusion module based on ECA attention [7]. We use fast one-dimensional convolution to capture inter-channel interactions effectively, without reducing the number of channels, and with fast computational speed and minimal memory occupation. Unlike the fusion strategy in AFF, we do not add the two features and then weigh the resulting fused feature. Instead, we first calculate the global feature representation of the two features, then the global features of the two global feature representations obtained separately, and then calculate the weights after stitching the two feature channels together. The input of the EFM module comprises two features that need to be weighted and fused. Firstly, perform global average pooling on the two features separately to extract their global information, denoted as \(f_{c}^{t}\in\mathbb{R}\) and \(f_{c}^{r}\in\mathbb{R}\), respectively. The formula is as follows:  
\[f_{c}^{t}=\frac{1}{W\times H}\sum_{j=1}^{W}\sum_{k=1}^{H}X_{\text{f}}^{c}(j,k) \tag{1}\]  
\[f_{c}^{r}=\frac{1}{W\times H}\sum_{i=1}^{W}\sum_{k=1}^{H}X_{\text{r}}^{c}(i,k) \tag{2}\]  
\(W\) and \(H\) denoted of dimensions of the input features. One-dimensional fast convolution is then applied to \(f_{c}^{t}\) and \(f_{c}^{r}\) with a kernel size of 3, which allows the network to capture interactions between the 3 adjacent channels. The two global feature representations are then concatenated along the channel dimension, and the channel weights are calculated through a softmax operation. Finally, the two features are weighted and fused using the obtained weights, resulting in a fused feature more conducive to subsequent tracking. \[f_{c}^{a}=[\mathcal{F}_{1D}^{r}(f_{c}^{t}),\mathcal{F}_{1D}^{t}(f_{c}^{r})] \tag{3}\]  
\[w^{t}=\sigma((f_{c}^{a})[0;0:d]) \tag{4}\]  
\[w^{r}=\sigma((f_{c}^{a})[0;d:2d]) \tag{5}\]  
\[X_{f}=X_{r}\times w^{r}+X_{t}\times w^{t} \tag{6}\]  
Figure 3: Detail diagram of the EFM module, \(X_{r},X_{t},X_{f}\), indicating the available optical, thermal infrared, and fusion features, respectively.