# Two-Stage Fusion-Based Audiovisual Remote Sensing Scene Classification
## 4 Experiments
### Audio Experiments  
Here, we compare the results of four schemes, ADVANCE [66], SoundingEarth [68], attention + CNN, and AiT, to demonstrate the effectiveness of the schemes based only on Transformer. Among them, the acoustic characteristics of the four schemes are common (use the log-Mel spectrogram). The audio path of the ADVANCE solution uses the AudioSet pre-trained ResNet-50 to model the sound content. The second solution uses batch triplet loss, which has the advantage of combining classical triplet loss training with recent contrastive learning methods. The third scheme consists of a single layer of self-attention and four layers of convolution. The fourth option is the audio Transformer introduced in Section 3.2.  
In order to prevent purely memorizing the input data and introducing more diversity into the training samples, we used frequency mask and time mask data augmentation [69] for the audio spectrogram, so that the AiT model can learn more-robust feature representations, thereby improving the generalization performance on the validation dataset. Specifically, before extracting the spectrogram features, we applied the FrequencyMasking (freqm) and TimeMasking (timem) transformations. The parameters, freqm and timem, represent the respective widths of the continuous frequency interval and time window to be masked. In addition, we also normalized the feature dimension of the spectrogram, so that the dataset was normalized to 0 mean and standard deviation.  
Table 2 shows the results of different audio system approaches. We can observe that the method using the attention + CNN architecture is superior to the baseline (ADVANCE) scheme based on the CNN architecture on all three indicators. Specifically, the results of the attention + CNN architecture (\(33.56\%\pm 0.24\) on the F1-score) are \(4.57\%\) higher than the ADVANCE method (\(28.99\%\pm 0.51\) on the F1-score) using only CNN, but it has no advantage or is even worse than the SoundingEarth scheme. According to the preliminary analysis, the attention + CNN structure does not receive any additional dataset pre-training, whereas the SoundingEarth model is pre-trained on a large audiovisual dataset. From the AiT scheme, we can observe that, in the case of using a pre-trained model, only the Transformer-based method (\(41.9\%\pm 0.11\) on the F1-score) is \(2.89\%\) higher than the ADVANCE and SoundingEarth methods (\(39.01\%\pm 0.17\) on the F1-score), which also use the transfer learning method.  
The confusion matrix for our the audio classification model AiT is shown in Figure 7. When only audio samples were used in the training, the ability to classify audio scenes was poor. Although forest and residential scenes with a large number of samples had better classification results, this also led to natural scenes (such as shrubland, orchard, and grassland) and scenes of human life (such as farmland and sports land) having poor classification results. As a consequence, there were some limitations when classifying scenes solely based on land ambient audio.  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Methods** & **Precision** & **Recall** & **F1-Score** \\ \hline ADVANCE (ResNet-50) [66] & 30.46 \(\pm\) 0.66 & 32.99 \(\pm\) 1.20 & 28.99 \(\pm\) 0.51 \\ SoundingEarth (ResNet-18) [68] & 37.91 \(\pm\) 0.31 & 38.36 \(\pm\) 0.25 & 37.69 \(\pm\) 0.19 \\ SoundingEarth (ResNet-50) [68] & 39.13 \(\pm\) 0.22 & 39.96 \(\pm\) 0.14 & 39.01 \(\pm\) 0.17 \\ Attention + CNN & 33.41 \(\pm\) 0.21 & 34.42 \(\pm\) 0.41 & 33.56 \(\pm\) 0.24 \\
**AiT (Ours)** & **41.8 \(\pm\) 0.15** & **42.8 \(\pm\) 0.61** & **41.9 \(\pm\) 0.11** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results (%) \(\pm\) STD of the experiments on audio systems.