# Two-Stage Fusion-Based Audiovisual Remote Sensing Scene Classification
## 4 Experiments
### Log-Mel Spectrogram  
Different acoustic features will have different effects on the accuracy of scene classification. Here, we chose the log-Mel spectrogram, which is widely used in the field of speech and sound processing. \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Methods** & **Precision** & **Recall** & **F1-Score** \\ \hline ADVANCE (ResNet + Cross-Task Transfer) [66] & 75.25 \(\pm\) 0.34 & 74.79 \(\pm\) 0.34 & 74.58 \(\pm\) 0.25 \\ SoundingEarth (ResNet-18) [68] & 89.59 \(\pm\) 0.27 & 89.52 \(\pm\) 0.29 & 89.5 \(\pm\) 0.12 \\ SoundingEarth (ResNet-50) [68] & 88.9 \(\pm\) 0.21 & 88.85 \(\pm\) 0.34 & 88.83 \(\pm\) 0.27 \\
**TFAVCNet (Ours)** & **89.9 \(\pm\) 0.14** & **89.85 \(\pm\) 0.36** & **89.83 \(\pm\) 0.21** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results (%) \(\pm\) STD of the experiments on audiovisual systems. Figure 9: Confusion matrix for our audiovisual systems, with true labels on the vertical axis and predicted values on the horizontal axis. The log-Mel spectrogram provides more-detailed spectrum information than the traditional spectrogram. It is divided into frames on the time axis, and each frame represents the energy distribution of different frequencies over a period of time. The purpose of logarithmic transformations, usually applied to Mel spectrograms, is to enhance details in low-energy regions such that the features are more evenly distributed on a logarithmic scale. The calculation process of the log-Mel spectrogram is shown in Formula (5). \[Mel(f)=2595*\log_{10}(1+\frac{f}{700}) \tag{5}\]  
where \(f\) is the frequency expressed in Hertz and \(Mel(f)\) is the corresponding Mel frequency. In addition, we performed some experimental analysis on two important parameters in the calculation of the spectrogram: hop length and number of Mel filters, to determine the optimal hyperparameters. The results are shown in Table 5.  
Hop length is measured on samples here, and 441 samples corresponded to 10 ms at a sampling rate of 44,100. In this experiment, we can see that a hop length of 400 samples (about 10 ms) had a higher average accuracy overall. Hop length is important, because it directly affects how \"wide\" the image is, so while a hop length of 5 ms might have a slightly higher accuracy than 10 ms, the images will be twice as large and, therefore, take up twice as much GPU memory. If hop length determines the width of the spectrogram, then the height is determined by the number of Mel filters. After first producing a regular STFT using an FFT size of, say, 2048, the result is a spectrum with 1025 FFT bins, varying over time. When we convert to a Mel spectrogram, those bins are logarithmically compressed down to n_mels bands. This experiment showed that 32 is definitely too few, but other than that, it did not affect the accuracy too much. A value of 128, which is the default with the baseline, gave the highest mean accuracy. Figure 10 shows the log-Mel spectrogram for the two audio scene classes we created using the librosa library, with hop_length set to 400 and the number of Mel filters set to 128. The plotted log-Mel spectrogram shows the characteristics of the audio signal in time and frequency. Some information can be observed from the spectrogram:  
* Time information: Each time step on the x-axis corresponds to an audio frame, which is the time axis of the audio signal. The audio's temporal development is visible. * Frequency information: The frequency band of the Mel filter is shown on the y-axis, which is typically scaled in the Mel frequency. The Mel frequency is a frequency scale that is connected to human auditory perception and more closely resembles the features of the auditory system in humans. * Energy information: Each time frame's logarithmic energy for each Mel frequency band is represented by a different color. The relative strength of various frequency components in the audio signal can be represented using this.