## 3 Experiments and Results
### Experiments on the MHGSA System  
The MHGSA system, based on the two previously mentioned multi-branch feature extraction modules for different modalities, uses a residual gated graph convolutional neural network based on hierarchical graphs as multimodal feature fusion. We extracted 12 categories from the original VGGSound dataset and categorized them into three categories as coarse granularity labels (Figure 7). We referred the extracted dataset as VGGSound for Disaster Response (VGGSound-DR). We conducted three experiments to compare the vision-only, audio-only, and multimodal cases. ResNet50 [25], EfficientNet-b1 [23] ResNet3D [33], and PANNs [34] are used as baselines to explore the performance of our models. It is worth mentioning that the proposed models are constructed differently in terms of classifiers due to the use of a multi-branch structure. We refer to the proposed feature extractor without graph learning as MHSA and to the model that uses the proposed graph-based approach as MHGSA. In addition, the comparison of MHSA and MHGSA demonstrates the effectiveness of the graph-based approach in the multimodal case. The average accuracy and its standard deviation, the best accuracy, the total number of trainable parameters, and the average inference time for one video clip are shown to reveal the performance of models. As indicated in Table 1, in the visual-only mode (VO), the best classification accuracy of the MHSA-VO using 10 frames as input and fully connected layers as classifiers is higher than that of EfficientNet-b1 (57.2%), ResNet-50 (54.8%), and R3D-18 (58.5%), at 65.3%, and the number of parameters is much lower than that of ResNet-50 and R3D-18. This suggests that the multi-branch structure is effectively utilized to the coarse-grained classification task to improve the model's performance on the fine-grained classification task. Multi-frame input significantly improves the performance of the model. MHSA-VO improves accuracy by more than 5% when using multiple frames as input over the single-frame case. To reduce the computational cost, we first trained MHSA-VO using single-frame inputs and froze the trained parameters for multi-frame training. Thus, the multi-frame input does not affect the number of parameters but only the inference time. The MHSA-VO adds hierarchical graph construction and three layers of RG-GCN for classification compared to MHSA-VO, which provides a slight improvement in accuracy, at 66.7%. This indicates that introducing a single-modal event relational hierarchical graph in the visual-only mode contributes to the model's performance in disaster event classification. It is worth mentioning that MHSA-VO serves as a subset of MHGSA-VO; the training process of MHGSA-VO introduces and freezes the pre-training parameters of MHSA-VO, i.e., it only trains the graph-related parameters. In audio-only mode (AO), as shown in Table 2, MHGSA-AO achieved the highest accuracy of 67.8%. It improved 0.6% over MHSA-AO without graph learning and 2.7%, 2.0%, and 15.2% over ResNet-50, EfficientNet-b1, and PANNs, respectively. This indicates that hierarchical graph construction and learning play a positive role when applied to concepts with multiple granularities. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Mode**} & \multirow{2}{*}{**Model**} & **Avg. Acc.** & \multirow{2}{*}{**Best Acc.**} & **Avg. Time** & \multirow{2}{*}{**Params**} \\  & & **(S.D.)** & & **(ms/Video)** & \\ \hline \multirow{6}{*}{Visual-Only} & ResNet-50 & 54.1 (\(\pm\)0.5)\% & 54.8\% & 13.1 & 26M \\  & Eff.Net-b1 & 57.2 (\(\pm\)0.6)\% & 58.0\% & 18.2 & 8M \\ \cline{1-1}  & R3D-18 * & 58.2 (\(\pm\)0.2)\% & 58.5\% & 10.4 & 33M \\ \cline{1-1}  & MHSA-VO * & 59.4 (\(\pm\)0.5)\% & 60.0\% & 18.8 & 15M \\ \cline{1-1}  & MHSA-VO * & 64.9 (\(\pm\)0.4)\% & 65.3\% & 37.8 & 15M \\ \cline{1-1}  & MHGSA-VO * & 66.2 (\(\pm\)0.4)\% & 66.7\% & 46.1 & 17M \\ \hline \hline \end{tabular}
* Model using multiple frames.  
\end{table} 
Table 1: Experiment on VGGSound-DR in visual-only mode.