## 3 The Proposed Framework
### Self-Supervised Audio-Visual Representation
#### 3.2.1 Audio-Visual Barlow Twins  
Based on the above consideration, we modified the pipeline in [23] and design an AV-BT that trains the audio-visual encoder in a self-supervised manner to narrow the heterogeneity gap between different modal data. Figure 2 illustrates the full AV-BT pipeline, with the three steps involved detailed below. **Preprocessing. ** For each video segment, we convert the raw audio into a log-Mel spectrogram and randomly sample one frame from the visual stream. The resulting spectrograms and extracted frames of the whole video are denoted as \(\mathbf{A}\in\mathbb{R}^{T\times H_{a}\times W_{a}}\) and \(\mathbf{V}\in\mathbb{R}^{T\times H_{a}\times W_{a}\times C}\), respectively. Since random data augmentations can boost the performance of self-supervised learning and help to fight against overfitting [23; 24; 25; 26; 27; 28; 51], we employ similar strategies in AV-BT. Specifically, cropping and Gaussian blurring are randomly applied to the extracted visual frames. To simplify notation, \(\mathbf{V}\) is assumed to be augmented. In addition, the random frame sampling technique also plays a role in data augmentation, as it can provide slight variations in visual content during each training epoch and helps the feature encoder capture more information along the timeline when the visual content remains stable. An in-depth analysis of the random sampling technique is provided in the next subsection. **Feature encoding. \(\mathbf{A}\) and \(\mathbf{V}\)** are then sent to a two-stream audio-visual encoder simultaneously to learn spatial-temporal features. The output features are denoted as \(\mathbf{f}_{a}\in\mathbb{R}^{T\times d_{l}}\) and \(\mathbf{f}_{v}\in\mathbb{R}^{T\times d_{l}}\), where \(d_{l}\) denotes the feature dimension. **Projecting. ** Eventually, two three-layer multi-layer perceptrons (MLP) are used as projectors to map the \(\mathbf{f}_{a}\) and \(\mathbf{f}_{v}\) into the same latent space, where we compute their cross-correlation matrix afterward to explore the audio-visual relationships. We assign \(d\) as the output dimension of these projectors, so the output embeddings are denoted as \(\mathbf{Z}_{a}\in\mathbb{R}^{T\times d}\) and \(\mathbf{Z}_{v}\in\mathbb{R}^{T\times d}\). To simplify notations, \(\mathbf{Z}_{a}\) and \(\mathbf{Z}_{v}\) are assumed to be normalized along the time dimension after projection. It is necessary to note that these two projectors share the same parameters so that they can force the encoders to learn joint representations carrying similar semantic information about the video. To calculate the objective function of AV-BT, we first compute the cross-correlation matrix \(\mathbf{\mathcal{C}}\in\mathbb{R}^{d\times d}\) of the normalized embeddings \(\mathbf{Z}_{a}\) and \(\mathbf{Z}_{v}\) along the time dimension. Specifically, the element in \(\mathbf{\mathcal{C}}\) is obtained by the following equation:  
\[\mathcal{C}_{ij}\triangleq\frac{\sum_{t}Z_{at_{j}}Z_{v_{t}j}}{\sqrt{\sum_{t} \left(Z_{at_{j}}\right)^{2}}\sqrt{\sum_{t}\left(Z_{v_{t}j}\right)^{2}}}, \tag{1}\]  
where \(t\) indexes the time dimension and \(i\), \(j\) indexes the feature dimension of \(\mathbf{Z}_{a}\) and \(\mathbf{Z}_{v}\) individually. Then we optimize the network by minimizing the difference between \(\mathbf{\mathcal{C}}\) and the identity matrix, and the objective function is written as:  
\[\mathcal{L}_{AVBT}\triangleq\sum_{i}\left(1-\mathcal{C}_{ii}\right)^{2}+ \lambda\sum_{i}\sum_{j\
eq i}\mathcal{C}_{ij}^{2}\quad, \tag{2}\]  
where \(\lambda\) is the same hyperparameter in the original method [23] and is used to trade off the importance between the first term (_invariance term_) and the second term (_redundancy reduction term_). Note that Equation (1) is defined for each video, and the total loss of a training step is obtained based on the average \(\mathbf{\mathcal{C}}\) over all videos in a batch. By minimizing \(\mathcal{L}_{AVBT}\), the representation network is trained to bridge the semantic information between synchronized audio and visual data.