# Two-Stage Fusion-Based Audiovisual Remote Sensing Scene Classification
## 3 Methods
### Two-Stage Hybrid Fusion Strategy  
Integrating data from various modalities is the aim of multimodal fusion. Early fusion (feature-level fusion) and late fusion (decision-level fusion) are the two primary multimodal fusion strategies. However, because the sparsity of the features extracted by various modalities may differ, their distributions may differ significantly. Therefore, it may not be possible to achieve the best fusion results by simply concatenating multimodal features into an input-level joint feature vector. Additionally, because decision-level fusion does not take into account the relationship between feature sets, it is not possible to fully utilize the similarity and complementarity of features, and it may not produce satisfactory fusion results.  
Figure 4: The proposed visual module.  
As shown in Figure 5, we propose a two-stage hybrid fusion strategy, which consists of feature-level fusion and decision-level fusion. This module fuses information from both audio and visual modalities in two stages and trains jointly to achieve a more-robust solution.  
In the feature-level fusion stage, we designed the adaptive distribution weighted fusion module to obtain fusion embedding. First, we calculated the Softmax results of audio embedding and visual embedding. Both embeddings share the same dimensions (batch_size, 768). The normalized weight (or importance) of the channel is calculated by computing the Softmax of the i-th channel of embedding [65], as shown in Equation (1).  
\[S(e_{i})=\frac{\exp(e_{i})}{\sum_{i}\exp(e_{i})} \tag{1}\]  
where \(e_{i}\) is the \(i\)-th channel of embedding.  
Then, calculate the weight of audio embedding (\(f_{1}\)) and visual embedding (\(f_{2}\)) for fusion embedding (\(f_{3}\)) by calculating the matrix kernel norm (\(\|\;S\|_{*}\)). Finally, the weight of the embedding is obtained by taking the weighted average of the nuclear norm. The calculation formula is given by Equations (2) and (3).  
\[W_{1} =\frac{\varphi(\|\;S(e_{i})^{1}\|_{*})}{\varphi(\|\;S(e_{i})^{1}\|_{*})+ \varphi(\|\;S(e_{i})^{2}\|_{*})} \tag{2}\] \[W_{2} =\frac{\varphi(\|\;S(e_{i})^{2}\|_{*})}{\varphi(\|\;S(e_{i})^{1}\| _{*})+\varphi(\|\;S(e_{i})^{2}\|_{*})} \tag{3}\]  
where \(S(e_{i})^{j},j\in[1,2]\) is the weight after the Softmax operation for the feature \(f_{j},j\in[1,2]\) and \(\varphi(\cdot)\) can be any arbitrary functions; we will discuss the effect of different function settings on the results in detail in the later experimental part. The final fusion embedding is then given by \(f=W_{1}\times f_{1}+W_{2}\times f_{2}\).  
After obtaining the audio, visual, and fusion embeddings, map the three embeddings to the labels for classification using a fully connected layer with Softmax activation to produce the audio result, visual result, and fusion result vectors, each of which has a batch size of 13.  
Figure 5: Our two-stage hybrid fusion strategy.  
We weighted and fused the output results of the audio module, visual module, and feature-level fusion stage in the decision-level fusion stage to maximize the use of the two types of single-mode information and weighted fusion information. These outputs are all probability distributions that were obtained using a Softmax activation function on a fully connected layer. The calculation process is shown in Equation (4). \(\phi(\cdot)\) can be any arbitrary functions; we will discuss the effect of different function settings on the results in detail in the later experimental part.  
\[R=\phi(R_{1},R_{2},R_{3}) \tag{4}\]