# Two-Stage Fusion-Based Audiovisual Remote Sensing Scene Classification
## 4 Experiments
### Experimental Setup and Evaluation Metrics  
All experiments were implemented using Pytoch 1.12. The models were trained on an NVIDIA RTX 3080 (12 GB) GPU in a single machine. The details during training were as follows. In the audio module, we used the pretrained weights of a DeiT-base 384 [67], which is trained with CNN knowledge distillation, with 384 \(\times\) 384 images, and has 87M parameters. In the visual module, we used a ViT-B/16 [64] (with 70 M parameters) as the backbone. We removed the original fully connected layers of the two Transformers. In addition, the pretrained parameters of ImageNet-21K were loaded on the DeiT-base 384 and ViT-B/16 (except the fully connected layer). When training TFAVCNet, we chose the Adam optimizer with a relatively small learning rate of \(1\times 10^{-4}\) and a weight decay of \(1\times 10^{-3}\) to optimize the parameters of TFAVCNet, as both backbones have been pre-trained from external knowledge. In the optimizer, we improved the stability of the model by taking a 10-epoch warm-up strategy. Considering the limited GPU memory resource, we set the batch size to 64 for the audio experiment and to 32 for the image.  
Splitting strategy: The ADVANCE dataset was employed for the evaluation, where 70% image-sound pairs were for training, 10% for validation, and 20% for testing. All the test results are the mean results of 10 splits in order to reduce random influence.  
Evaluation metrics: To quantitatively evaluate the performance of each module, we adopted the weighted average precision, recall, and F-score metrics with the standard deviation (\(\pm\)STD) for the evaluation. The distribution of the data can have an impact on accuracy in tasks with imbalanced classes, as the model may be biased towards prediction if the number of samples from one class is significantly higher than the number of samples from another class. Most categories receive high accuracy ratings, while minority categories receive an insufficient performance evaluation. Metrics like the precision, recall, and F-score may be more useful in this situation because they give more in-depth information to analyze the model's performance across various categories:  
1. The accuracy of model predictions is determined by the precision. It shows how many of the samples that the model predicted to be positive are actually positive.
2. Another crucial evaluation metric is the recall, which is used to determine how well the model is able to identify all real positive examples. A high recall means that the model can capture positive examples more accurately.
3. The F-score combines the precision and recall to provide an indicator for comprehensively evaluating model performance. The F-score is the weighted average of the precision and recall, where the weight factor can be adjusted according to the task requirements. The most-common F-score is the F1-score, which gives equal weight to the precision and recall.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Scene Category of ADVANCE** & **Total** \\ \hline Airport & 185 \\ Beach & 216 \\ Bridge & 280 \\ Farmland & 430 \\ Forest & 862 \\ Grassland & 150 \\ Harbor & 508 \\ Lake & 357 \\ Orchard & 207 \\ Residential & 1056 \\ Shrubland & 405 \\ Sports land & 159 \\ Train station & 260 \\ \hline \hline \end{tabular}
\end{table}
Table 1: ADVANCE dataset scene category distribution.