## 2 Proposed Methodology
### Hierarchical Graph for Disaster Event Classification  
#### 2.2.1 Hierarchical Graph Construction  
A multimodal hierarchical graph consisting of events is built after having all the node features of the fine-grained disaster event \(E_{f}\) and the coarse-grained disaster event \(E_{c}\) in visual and audio features. It aims to learn the relationships between different event nodes and update the node features employing the subsequent graph convolutional learning. Inspired by [17], we define a hierarchical graph containing multimodal multi-granularity event nodes. They are all constructed in a fully connected manner. There is a graph \(G=(V,E)\), where \(V\) and \(E\) denote the nodes and edges, respectively. The initial connection of nodes can be represented as follows:  
\[A_{i,j}=1,\;i\in N,\;j\in N \tag{23}\]  
where \(A\) is the adjacency matrix of the graph and \(N\) denotes the number of nodes in the graph. This type of connection requires each node to be self-connected, so the number of edges contained in a single graph is indicated below:  
\[|E|=\frac{N(N+1)}{2}, \tag{24}\]  
where \(|\cdot|\) denotes the base of a set, i.e., the number of elements in the set.  
The hierarchical graph construction is visually portrayed in Figure 5, where blue nodes correspond to coarse-grained event categories, and yellow nodes signify fine-grained ones. Visual features are represented by circular nodes, whereas audio features are denoted by square nodes. The visual and audio features are output by their corresponding feature extraction module and are concatenated outside the module to form node features for the graph construction.  
Figure 4: Audio feature extraction module for constructing multi-grained audio event nodes features.  
Figure 5: Multimodal hierarchical graph construction.  
Based on the proposed visual and audio feature extraction modules, we construct a node for each disaster event and assign a node feature for it. In order to independently learn the node features for each type of event, node features are constructed from the corresponding granularity branches using linear layers, as described in Equations (8) and (9) for visual features and Equations (21) and (22) for audio features.  
For single-modal hierarchical graphs, i.e., \(G_{Single}^{Visual}\) (visual only) or \(G_{Single}^{Audio}\) (audio only), their node features are obtained from the proposed multi-branch feature extraction module for corresponding modality. A single modality hierarchical graph \(G_{single}\) can be represented as  
\[G_{Single}^{m}=\left(V_{Coarse}^{m}\cup V_{Finer}^{m},E_{Single}^{m}\right),m \in[Visual,\ Audio], \tag{25}\]  
where \(N_{i}^{m}\) denotes the node features. This allows subsequent GCN to learn the subordination relationship of coarse-grained and fine-grained events.  
The proposed multimodal hierarchical graph \(G_{Multi}\), composed using all the event nodes of different modalities, makes it have twice as many nodes as event categories.  
\[G_{Multi}=\left(V^{Visual}\cup\ V^{Audio},\ E_{Single}^{Visual}\cup E_{Single}^{ Audio}\cup E_{Multi}^{Visual+Audio}\right), \tag{26}\]  
where \(V_{Visual}\) and \(V_{Audio}\) denote the nodes from \(G_{Single}^{Visual}\) and \(G_{Single}^{Audio}\), respectively. Regarding them as subgraphs, \(E_{Multi}^{Visual+Audio}\) contains edges connecting them. Employing the gated graph convolutional network, \(E_{Single}^{Visual}\) and \(E_{Single}^{Audio}\) can be learned by the previous layers and be passed to the layer for \(G_{Global}\) by adding initialised edges connecting \(V_{Visual}\) and \(V_{Audio}\).