# Two-Stage Fusion-Based Audiovisual Remote Sensing Scene Classification
## 4 Experiments
### Audiovisual Experiments  
Here, we compare the results of ADVANCE [66], SoundingEarth [68], and our two-stage hybrid fusion strategy and show the quantitative results of different fusion strategies on three metrics (precision, recall, F1-score). The ADVANCE scheme transfers sound event knowledge to aerial scene recognition tasks to improve scene recognition performance. The SoundingEarth solution learns semantic representation between the audio and image modalities through self-supervised learning. Next, in our two-stage hybrid fusion schemes, we selected the \(max()\) and \(sum()\) functions to determine \(\varphi(\cdot)\) and \(\phi(\cdot)\) for feature fusion and decision fusion.  
From Table 4, the following observations can be made. First, audiovisual experiments can achieve better results than those based only on audio or visual information, suggesting that fusing information from the audio and visual modalities can improve single-mode classification. Secondly, for the comparison of the fusion methods, our two-stage fusion strategy  
Figure 8: Confusion matrix for our image systems, with true labels on the vertical axis and predicted values on the horizontal axis.  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Methods** & **Precision** & **Recall** & **F1-Score** \\ \hline ADVANCE (ResNet-101) [66] & 74.05 \(\pm\) 0.42 & 72.79 \(\pm\) 0.62 & 72.85 \(\pm\) 0.57 \\ SoundingEarth (ResNet-18) [68] & 87.09 \(\pm\) 0.19 & 87.07 \(\pm\) 0.11 & 86.92 \(\pm\) 0.16 \\ SoundingEarth (ResNet-50) [68] & 83.97 \(\pm\) 0.15 & 83.88 \(\pm\) 0.14 & 83.84 \(\pm\) 0.17 \\
**ViT-B/16 (Ours)** & **87.64 \(\pm\) 0.38** & **87.52 \(\pm\) 0.49** & **87.61 \(\pm\) 0.41** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results (%) \(\pm\) STD of the experiments on visual systems.  
further improved the effect compared with the ADVANCE method and SoundingEarth method, which only use feature fusion.  
The confusion matrix of our two-stage hybrid fusion strategy's classification results is shown in Figure 9. For the majority of categories, we can see that the result of classification tended to be good. In certain scenes (forest, harbor, residential), it worked flawlessly. Grassland and orchard were the scenes with the worst classification. In the test set for grassland, six samples were classified as forest, six as shrubland, and three as farmland; in the test set for orchard, eight samples were classified as forests and three were classified as farmland. The dataset's obvious long tail distribution and the insufficient sample size for the grassland and orchard scenes prevented the model from learning more-useful distinguishing features, according to preliminary analysis. The fact that the scene's audio events were too similar could also be a factor. In the grassland, forest, and shrubland scenes, we discovered that there were numerous similar sound events, such as birdsong, water, etc., through manual inspection. The vast amounts of green vegetation in both scenes, which are indistinguishable to humans, further resembled one another, creating a visual similarity between the two.