## 3 Experiments and Results
### Experiments on the MHGSA System
Figure 7: Fine- and coarse-grained categories of the extracted VGGSound-DR dataset. In multimodal mode, we assign a baseline model to vision and audio features and concatenate their feature vectors as the input of the fully connected layer classifier. We used this approach to construct ResNet-50-MM and EfficientNet-b1-MM as baseline models. On the other hand, we used both visual and audio feature extraction modules and compared fully connected layers (MHSA) and RG-GCN (MHGSA) as classifiers. As shown in Table 3, the highest results were achieved by the MHGSA, obtaining 77.6% accuracy with 34M parameters. The graph-based model outperforms the traditional fully connected layer by about 4M higher number of parameters in the multimodal mode, proving the effectiveness of multimodal hierarchical graph learning. Different from the visual-only and audio-only cases, the introduction of a graph-based approach improves more significantly in the multimodal case. This also confirms our idea of constructing a multimodal hierarchical graph, i.e., modeling the relationship between events across modalities using a graph learning approach, in order to improve the model's ability to discriminate between different events. We can see the improved performance of the model when using multimodal information. Compared to VO and AO, all models show a substantial increase in accuracy with a doubled number of parameters in the multimodal mode, with the highest increases being in MHGSA at 10.9% and 9.8%. From the perspective of situation awareness, more comprehensive perception and deeper understanding are important goals. In contrast to baseline models, the proposed MHGSA uses audio-visual modalities to obtain a more comprehensive perception, whereas multi-granularity classification allows the model to have a richer understanding of the environment. The classification results of the events with different granularities provide a richer understanding of the environment, enabling disaster response. Table 4 illustrates the precision metrics for every coarse- and fine-grained category evaluated from the MHGSA in different cases. The results from the coarse-grained categories intuitively show that the visual and audio modalities have different representational capabilities in different categories. For example, visual-only models are much more precise in the 'Nature Disasters' category than audio-only models, which is the opposite in the 'Disaster Alerts' category. For the fine-grained categories, the multimodal model obtained the highest accuracy in the vast majority of the categories, especially the 'rocket launch' category, which obtained 92.5% with less than 75% precision for both VO and AO. This indicates the rationality of building a multimodal situational awareness system, i.e., using different modalities for complementary information. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Mode** & **Model** & \
\begin{tabular}{c} **Avg. Acc.** \\ **(S.D.)** \\ \end{tabular} & **Best Acc.** &
\begin{tabular}{c} **Avg. Time** \\ **(ms/Video)** \\ \end{tabular} & **Params** \\ \hline \multirow{4}{*}{Audio-Only} & ResNet-50 & 64.8 (\(\pm\)0.2)\% & 65.1\% & 49.2 & 26M \\  & Eff.Net-b1 & 65.3 (\(\pm\)0.3)\% & 65.8\% & 52.6 & 8M \\  & PANNs & 51.9 (\(\pm\)0.5)\% & 52.6\% & 42.6 & 75M \\  & MHSA-AO & 66.7 (\(\pm\)0.4)\% & 67.2\% & 52.8 & 15M \\  & MHGSA-AO & 67.3 (\(\pm\)0.4)\% & 67.8\% & 74.7 & 17M \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Experiment on VGGSound-DR in audio-only mode.