## 3 The Proposed Framework
### Self-Supervised Audio-Visual Representation  
According to the above definition, the localization of an AVE requires a combination of information from both visual and audio signals. To extract useful information from the raw signal, we followed existing methods [6; 7; 8; 9; 10; 11; 12; 13] to extract spatial features in video frames and audio spectrograms using two-dimensional CNN. Considering that the video is sequential, Bi-LSTMs are then used for the modeling of temporal relations.  
Due to the scarcity of audio-visual video samples that can be used for training, the performance of training feature encoders from scratch is often unsatisfying, making it necessary to pretrain the CNN on large-scale datasets [14; 15]. However, this pretraining also poses a problem: there is a huge heterogeneity gap that exists between the features extracted by the pretrained CNN and it naturally hinders information fusion. To overcome this problem, our solution is to adapt the CNN features from different modalities to the same semantic space. In practice, we accomplish the feature adaptation in a self-supervised manner by exploiting the relationship between vision and hearing as supervision. Specifically, there are two general cross-modal relationships in AVEs, namely, audio-visual _co-occurrence_ and _correspondence_. On the one hand, _co-occurrence_ refers to the fact that when an AVE happens in a video segment, its visual and auditory stimuli are provided simultaneously. For example, a baby's cry is accompanied by a sad facial expression. Audio-visual _correspondence_, on the other hand, means that the appearance of a particular object is associated with its characteristic sound. For example, when a bell appears in a scene, the neural network should match it to the ringing sound in the audio stream at that moment.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Symbols** & **Definition** \\ \hline \(S_{a}/S_{v}\) & Audio/visual portion of the video \\ \hline \(\mathbf{y}^{t}\) & Annotation of the \(t\)-th video segment \\ \hline \(A/\mathbf{V}\) & Preprocessed audio/visual data \\ \hline \(f_{a}/f_{v}\) & Encoded audio/visual feature \\ \hline \(Z_{a}/Z_{v}\) & Projected audio/visual embedding \\ \hline \(\mathbf{\mathcal{C}}\) & Cross-correlation matrix of \(\mathbf{Z}_{a}/Z_{v}\) \\ \hline \(W_{a}^{1}/W_{v}^{1}\) & Parameters for computing audio/visual query and key \\ \hline \(M^{\text{tr}}/M^{\text{tr}}\) & Cross-modal attention matrix \\ \hline \(W^{\text{trtr}}/W^{\text{trtr}}\) & Weight matrix for \(M^{\text{tr}}/M^{\text{tr}}\) \\ \hline \(W^{\text{trtr}}/W^{\text{trtr}}\) & Weighted attention matrix \\ \hline \(W^{2}_{a}/W^{2}_{v}\) & Parameters for computing audio/visual value \\ \hline \(A^{/\text{trse}}/V^{\text{trse}}\) & Attended audio/visual feature \\ \hline \(W^{\text{trtr}}_{a}/W^{\text{trtr}}_{v}\) & Parameters for \(\mathbf{A}^{/\text{trse}}/V^{\text{trse}}\) \\ \hline \(f_{a+\text{tr}}/f_{v+a}\) & Fused audio/visual feature \\ \hline _pred_ & Prediction of the input video \\ \hline \hline \end{tabular}
\end{table}
Table 1: Main symbols used throughout the paper.  
Utilizing these two attributes of AVE means that the network should match the relevant semantic information carried by the synchronized audio and visual signal pairs.