## 2 Proposed Methodology
### Multi-Branch Feature Extraction Module
#### 2.1.2 Audio Feature Extraction for Multigranularity  
Audio data are typically represented as a sequence of waveforms in the time domain. Our original idea to learn their representation and make them assist visual features for downstream tasks lies in the fact that the occurrence of a disaster event is usually accompa  
Figure 3: Structure of the proposed multi-branch feature extraction module for constructing node features from video frames.  
nied by a corresponding sound. For example, a fire is usually accompanied by the crackling sound of burning objects or a storm is usually accompanied by the sound of rain or wind. We designed an audio preprocessing algorithm to convert one-dimensional waveforms into two-dimensional audio features. LFCC and CQT as acoustic features are extracted after performing voice active detection (VAD) on the audio.  
A Voice Activity Detection (VAD) module was implemented [27] for data preprocessing. This module filters activated speech by calculating the short-time energy and short-time zero-crossing counter of the audio. Suppose there is a segment of audio \(w_{i}\) that is paired with \(x_{i}\), as mentioned before, thus:  
\[W=\{w_{1},w_{2},\dots,w_{T}\}. \tag{12}\]  
The energy and zero crossing counter of \(w_{i}\) can be calculated as follows:  
\[Energy_{i}=w_{i}^{2}, \tag{13}\]  
\[ZCC_{i}=|sign(w_{i})-sign(w_{i-1})| \tag{14}\]  
By averaging the \(T\)-length audio into \(F\) segments, the short-time energy and short-time zero-crossing counter can be expressed as follows:  
\[Energy_{short-timef}=\sum_{f\in F}Energy_{f} \tag{15}\]  
\[ZCC_{short-timef}=\sum_{f\in F}ZCC_{f} \tag{16}\]  
Suitable audio clips are filtered through a set threshold with the filtering rule:  
\[\left(Energy_{f}>Threshold_{Energy}\right)\ \wedge\left(ZCC_{f}<Threshold_{ZCC}\right) \tag{17}\]  
LFCC and CQT are extracted on the activated audio clips  
\[LFCC_{active}=DCT\big{(}\log_{10}LinearFeqFilterBank(STFT(w_{t=ActiveFrames}))\big{)}, \tag{18}\]  
\[CQT_{active}=abs(ConstantQFilterBank(STFT(w_{t=ActiveFrames}))), \tag{19}\]  
where \(LFCC_{active}\) and \(CQT_{active}\) are both matrices. They are concatenated to form an image-like 2-channel acoustic feature vector for the subsequent audio feature extraction  
\[AudioSharedFeat_{w_{i}}=Concat(LFCC_{active},CQT_{active}) \tag{20}\]  
Multiple audio events may be included in a single audio clip. Inspired by [17], we modified the originally shared single linear layer to linear layers containing only one neuron corresponding to the number of events, i.e., \(FC_{fi}\) and \(FC_{ci}\), to obtain a node representation of the corresponding events. Therefore, a single audio can provide feature vectors for all events, i.e., \(E_{f}\) and \(E_{c}\), and the process can be briefly expressed as follows:  
\[NodeFeat_{c_{i}}^{Audio}=\sigma(W_{ci}\cdot W_{Audio}\cdot(AudioSharedFeat_{wi })) \tag{21}\]  
\[NodeFeat_{fj}^{Audio}=\sigma\Big{(}W_{fj}\cdot W_{Audio}\cdot(AudioSharedFeat_{wi })\Big{)} \tag{22}\]  
where \(W_{ci}\) and \(W_{fj}\) is the learnable weights of \(FC_{ci}\) and \(FC_{fj}\) for the \(i^{th}\) and \(j^{th}\) event in \(E_{c}\) and \(E_{f}\). \(W_{Audio}\) denotes the learnable parameters of the audio feature extraction module. Figure 4 illustrates the structure of the audio feature extraction module.