# Multi-Sensor Data Fusion Method Based on Self-Attention Mechanism
## 3 Method
### Fusion Module  
The feature fusion module in the CNN-SA was built based on the self-attention mechanism, and its structure is illustrated in Figure 4. First, the four feature vectors obtained from the point cloud and image feature extraction modules were concatenated. These feature vectors include the point cloud feature vector \(\text{F}_{\text{P}}^{\text{pool}}\), control point feature vector \(\text{F}_{\text{P}}^{\text{point}}\), RGB image feature vector \(\text{F}_{\text{x}}^{\text{rgb}}\), and depth image feature vector \(\text{F}_{\text{x}}^{\text{depth}}\). After concatenation, the resulting feature vector \(\text{F}^{\text{concat}}\) undergoes a specified dimensional transpose operation, resulting in \(\text{F}_{\text{train}}^{\text{concat}}\). Finally, the concatenated and transposed feature vector was input into the self-attention mechanism module, enabling it to adaptively adjust the importance of each input feature and enhance the overall expressive power of the network. It is worth noting that in the process of connecting feature vectors, the order in which they are connected does not have an effect on the final result.  
The input features \(\text{F}^{\text{concat}}\) and \(\text{F}_{\text{train}}^{\text{concat}}\) for the self-attention mechanism were calculated as follows, where concat represents the concatenation operation along the specified channel dimension and transpose represents the transpose operation along the specified dimension.  
\[\text{F}^{\text{concat}}=\text{concat}(\text{F}_{\text{P}}^{\text{pool}},\text {F}_{\text{P}}^{\text{point}},\text{F}_{\text{x}}^{\text{rgb}},\text{F}_{\text {x}}^{\text{depth}}) \tag{8}\]  
\[\text{F}_{\text{train}}^{\text{concat}}=\text{transpose}(\text{F}^{\text{concat}}) \tag{9}\]  
During forward propagation, the attention scores, which indicate the importance of each sequence, are calculated as follows: \(\text{weight}_{\text{init}}\) and \(\text{bias}_{\text{init}}\) represent the initialized weights and bias terms, respectively.  
\[\text{scores}=(\text{F}_{\text{train}}^{\text{concat}}\times\text{weight}_{ \text{init}})+\text{bias}_{\text{init}} \tag{10}\]  
Figure 4: Feature fusion module.  
The application of Softmax to normalize the attention scores yields attention weights, which ensure that the elements of the feature vectors are within the range [0, 1] along the specified dimension and that they sum up to 1. The calculation is as follows:  
\[\text{weight}_{\text{soft}}=\frac{\exp(\text{scores}_{\text{i}})}{\sum_{\text{j}} \exp(\text{scores}_{\text{j}})} \tag{11}\]  
The self-attention vector after weighted summation is obtained through the following calculation by multiplying the \(\text{weight}_{\text{soft}}\) and \(\text{F}_{\text{ran}}^{\text{concat}}\) summed along the specified dimension to obtain the final self-attention vector.  
\[\text{weight}=\sum\left(\text{weight}_{\text{soft}}\times\text{F}_{\text{ tran}}^{\text{concat}}\right) \tag{12}\]  
The final output-fused feature \(\text{F}^{\text{output}}\) is calculated as follows: it combines self-attention-weighted results with the original feature vectors.  
\[\text{F}^{\text{output}}=\text{F}^{\text{concat}}+\text{weight} \tag{13}\]  
The feature fusion module, implemented based on the self-attention mechanism, fuses four different types of input features, which enables the network to utilize different types of feature information more effectively and improves multi-source multi-modal data fusion.