## 3 The Proposed Framework
### Self-Supervised Audio-Visual Representation
#### 3.2.1 Audio-Visual Barlow Twins
Another key point in Equation (1) is that our approximative identity correlation matrix is computed on the video level rather than the segment level. This is because of the existence of the _background_ segment where audio and visual signals are not semantically associated with each other. In-depth analysis of how the objective function \(\mathcal{L}_{AVBT}\) guides the network to encode informative audio-visual representations is discussed in the next subsection. The pseudocode of our AV-BT is shown in Algorithm 1. ```
1:
```
1:  
one modality. Second, similarly to the different distorted versions of an image, vision and sound can be regarded as two heterogeneous expressions of the same event. Hence in our \(\mathcal{L}_{AVBT}\), the _invariance term_ makes the features extracted from audio and visual streams both carry semantic information about the underlying target. Note that such a \"target\" refers to an object that may potentially have both visual and aural attributes. For example, for a person who appears in the scene, the semantic representation of the appearance extracted by the visual encoder should be tightly bound to the representation of the speaking voice extracted by the audio encoder. This is like how humans naturally associate the sound of speech when they see a person talking. In contrast, a table does not make any characteristic sound, so it is not audibly perceptible despite its visual characteristics. Therefore, an object like a table is treated as visual background and the encoder naturally ignores it. Third, since ambient noise and visual background in the environment do not carry meaningful semantic information across modalities, the _redundancy reduction term_ forces the representations of both modalities to contain as little information about irrelevant interference as possible. This is analogous to the idea of the original method. **Examination of random frame sampling technique. ** It is worth noting that in Equation (1), the cross-correlation matrix is calculated along the time dimension for each sample in a batch (while it was computed along the batch dimension in [23]), which means that only the similarity of visual and audio embeddings from the same moment is measured. This is general to exploit audio-visual _co-occurrences_ but inadequate for AVE representation owing to the possible contextual relevance between asynchronous audio and visual signals. For example, if the \(t\)-th segment and the \(\left(t-1\right)^{th}\) segment contain the same AVE, obviously \(\mathbf{Z}_{a,t}\) and \(\mathbf{Z}_{v,t-1}\) are correlated. To fully describe our technique for compensating for this defect, an example is given as follows. We first denote the \(t\)-th audio-visual segment pair as \((S_{a}^{t},\,S_{v}^{t})\), and there are \(N\) frames in \(S_{v}^{t}\), denoted as \(f_{n}^{t}\) where \(n\in\left\{1,2,\,\ldots,N\right\}\). Suppose that we extract \(f_{1}^{t}\) at random during a training epoch, such that the network is taught to match the information shared between \(f_{1}^{t}\) and \(S_{a}^{t}\). Similarly, the network will learn to model pairs such as \(S_{a}^{t},f_{1}^{t}\) and \(S_{a}^{t-1},f_{N}^{t-1}\) by training with abundant epochs. Note that \(f_{1}^{t}\) and \(f_{N}^{t-1}\) are indeed two adjacent frames and only a minor difference exists between them (if visual content has not changed much in this period), such that the association within \(S_{a}^{t},f_{N}^{t-1}\) can be broadcast through \(f_{1}^{t}\). This technique allows the network to model the relation between adjacent segments to a certain extent. An intuitive example is demonstrated in Figure 3. Moreover, processing only one frame for each segment can help to fit the video memory of the GPU and this random sampling technique also helps to prevent overfitting. Figure 3: An illustration of the random sampling technique in AV-BT. Three continuous video segments segments that contain the “train horn” are depicted in this example.