SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Audio-visual downstream tasks: To adapt our approach to the four audio-visual downstream tasks, we replace the pre-trained audio and visual encoders with a frozen HTS-AT [3] and a frozen SwinTransformer [20], respectively. The trainable DG-SCT modules in Fig. 2 (4) are injected into the frozen layers to let audio and visual modalities guide each other. We then use this as our audio-visual feature extractor. For AVE task, our feature extractor is combined with CMBS [36]. The event category label of each video segments is required to be predicted in supervised manner. We adopt [33, 38, 37, 36] and exploit the overall accuracy of the predicted event category as the evaluation metrics; Combined with MGN [24], DG-SCT is able to tackle the weakly-supervised AVVP task. Following [32], we evaluate the parsing performance of all events (audio, visual, and audio-visual events) under segment-level and event-level metrics; For AVS task, We combine our audio-visual feature extractor with the original AVS model [43]. We use the Jaccard index  $\mathcal{J}$  [6] and F-score  $\mathcal{F}$
Table 1: Audio-visual event localization. Comparisons on the test set of AVE in supervised manner. The result of our re-implementation of LAVisH [17] is  $79.7\%$ .
Method	Visual Encoder	Audio Encoder	Acc
AVEL(Audio-Visual) [33]	VGG-19	VGG-like	71.4
AVEL(Audio-Visual+Att) [33]	VGG-19	VGG-like	72.7
AVSDN [16]	VGG-19	VGG-like	72.6
CMAN [38]	VGG-19	VGG-like	73.3
DAM [35]	VGG-19	VGG-like	74.5
CMRAN [37]	VGG-19	VGG-like	77.4
PSP [44]	VGG-19	VGG-like	77.8
CMBS [36]	VGG-19	VGG-like	79.3
LAVisH [17]	Swin-V2-L (shared)	81.1
LAVisH [17]	Swin-V2-L (shared)	79.7
LAVisH*4	Swin-V2-L	HTS-AT	78.6
Ours	Swin-V2-L	HTS-AT	82.2
Table 2: Audio-visual video parsing. Comparisons on the test set of LLP.
Methods	Segment-level	Event-level
A	V	AV	Type	Event	A	V	AV	Type	Event
AVE [33]	49.9	37.3	37.0	41.4	43.6	43.6	32.4	32.6	36.2	37.4
AVSDN [16]	47.8	52.0	37.1	45.7	50.8	34.1	46.3	26.5	35.6	37.7
HAN [32]	60.1	52.9	48.9	54.0	55.4	51.3	48.9	43.0	47.7	48.0
MGN [24]	60.7	55.5	50.6	55.6	57.2	51.0	52.4	44.4	49.3	49.2
Ours	59.0	59.4	52.8	57.1	57.0	49.2	56.1	46.1	50.5	49.1
as the evaluation metrics; For AVQA task, our audio-visual feature extractor is used in the original ST-AVQA [14]. Similarly, we use the answer prediction accuracy as the evaluation metrics.
Audio-visual few-shot/zero-shot tasks: We incorporate DG-SCT modules as adapters between the frozen CLIP image encoder ViT [26] and frozen CLAP audio encoder HTS-AT [5], generating audio and visual features for text-audio and text-image contrastive learning, as shown in Fig. 2 (e). For the zero-shot setting, the model is pre-trained on the VGG-Sound(40K) [2] dataset. More details will be discussed in Appendix.
4.3 Compared with state-of-the-arts on audio-visual downstream tasks
First, we challenge our method against current state-of-the-art methods on the four audio-visual tasks. As demonstrated in Table 1, our model outperforms CMBS [36] and LAVisH [17] by a significant margin (2.9% and 2.5%); In Table 2, our model attains either a competitive or even better performance. For instance, DG-SCT surpasses MGN by 3.9% points in the segment-level visual event parsing metric, demonstrating our method can utilize large-scale models to extract more useful features and further promote the fusion of these features in downstream tasks. We also achieve state-of-the-art results on S4 setting of AVS task (Table 3). Lastly, our model performs exceptionally well on AVQA task and outperforms previous leading methods on AQ, VQ, and AVQ question types, respectively. The experimental results reveal that our model has the capability of utilizing pre-trained audio and visual models to extract more comprehensive, higher-quality features tailored for downstream tasks than the cross-attention mechanism of LAVisH. Moreover, our model exhibits excellent generalization abilities, achieving impressive results across various audio-visual tasks.
4.4 Compared with state-of-the-arts on audio-visual few-shot/zero-shot tasks
Furthermore, as presented in Fig. 4, our DG-SCT model surpasses top-performing methods like CoCoOp, CLIP-Adapter, and MaPLe by a non-negligible margin on our newly proposed audio-visual few-shot/zero-shot scenarios. Our few-shot (shot=16) learning for the AVE task attains  $72.4\%$ ,
Our modified implementation of LAVisH for fair comparisons.
Table 3: Audio-visual segmentation. Comparisons on the S4 and MS3 settings of AVSBench.  $\mathcal{M}_{\mathcal{T}}$  and  $\mathcal{M}_{\mathcal{F}}$  denote the mean  $\mathcal{J}$  and  $\mathcal{F}$  metric values (section 4.2) over the whole dataset.
Method	Visual Encoder	Audio Encoder	Setting
S4	MS3
MJ	MF	MJ	MF
AVS [43]	PVT-v2	VGG-like	78.7	87.9	54.0	64.5
LAVisH [17]	Swin-V2-L (shared)	80.1	88.0	49.8	60.3
LAVisH*	Swin-V2-L	HTS-AT	78.0	87.0	49.1	59.9
Ours	Swin-V2-L	HTS-AT	80.9	89.2	53.5	64.2
Table 4: Audio-visual question answering. Comparisons on the test set of MUSIC-AVQA. We report accuracy on three types of questions, e.g., audio (AQ), visual (VQ), and audio-visual (AVQ).
Method	Visual Encoder	Audio Encoder	AQ	VQ	AVQ	Avg
AVSD [28]	VGG-19	VGG-like	68.5	70.8	65.5	67.4
Pano-AVQA [40]	Faster RCNN	VGG-like	70.7	72.6	66.6	68.9
ST-AVQA [14]	ResNet-18	VGG-like	74.1	74.0	69.5	71.5
LAVisH [17]	Swin-V2-L(shared)	75.7	80.4	70.4	74.0
LAVisH*	Swin-V2-L	HTS-AT	75.4	79.6	70.1	73.6
Ours	Swin-V2-L	HTS-AT	77.4	81.9	70.7	74.8
outperforming MaPLe for  $5.3\%$  points, and is even comparable to previous full-training baselines (Table 1). Note that experimental results for few-shot on the LLP classification task are worse than zero-shot scenario. We argue that this is due to a significant gap between downstream and pre-training data, which may have disrupted the original parameters of the CLIP model. However, we still outperform MaPLe by  $2.2\%$  and  $0.7\%$  points, for zero-shot and few-shot (shot=16) settings, respectively. The outcomes demonstrate that our dual-guided audio-visual prompts extract task-specific information better than text-image [26, 46, 45, 7, 12] and text-audio [5] pre-trained models and exhibit stronger adaptability on audio-visual tasks.
4.5 Ablation analysis
We verify the effectiveness of the three modules, namely, S (spatial), C (channel), and T (temporal), in DG-SCT. As shown in Table 5, compared to the baseline model, the individual incorporation of these attention modules leads to substantial performance enhancements of our model across a range of downstream tasks, demonstrating the effectiveness of these modules. Although cross-modal interactions in any dimension can enhance feature extraction, we argue that distinct modules can also mutually guide and enhance each other's performance. For instance, removing the T module individually does not result in a significant decrease in the performance of the model, which indicates that the combination of S and C can slightly substitute the function of T. The reason might be that if the features of a certain segment are more prominent in both spatial and channel dimensions, it is more likely to be important in the temporal dimension as well.
Table 5: Ablation study of the spatial-channel-temporal attention module. "S", "C", and "T" denote spatial, channel, and temporal attention, respectively.
Module	AVE	AVVP	AVS	AVQA
S	C	T	Acc	Segment-level AV	Event-level AV	S4	MS3	AQ	VA	AVQ	Avg
MJ	MF	MJ	MF
-	-	-	78.6	49.8	43.9	78.0	87.0	49.1	59.9	75.4	79.6	70.1	73.6
-	✓	-	81.8	51.3	45.6	79.9	88.4	51.0	62.1	76.0	80.9	70.8	74.4
✓	-	-	80.9	51.7	44.7	78.6	87.6	50.9	61.6	74.9	80.3	69.5	73.3
✓	✓	-	82.0	52.3	45.9	80.0	88.6	51.8	62.3	77.0	81.9	70.3	74.6
-	-	✓	81.5	50.9	45.7	79.9	88.9	52.2	61.5	76.0	81.3	70.2	74.1
✓	✓	✓	82.2	52.8	46.1	80.9	89.2	53.5	64.2	77.4	81.9	70.7	74.8
Figure 3: The qualitative results of DG-SCT on AVE and AVS tasks.
Figure 4: Results of our model and previous methods on few-shot/zero-shot tasks.
4.6 Qualitative analysis
Fig. 3 represents examples of the effectiveness of our DG-SCT. On the left, we observe that with the guidance of the audio prompts, the module can accurately focus on the critical visual regions for different AVE events. For instance, when multiple objects are present in video frames, our model is capable of accurately localizing the sounding objects (e.g., bark, cat, the second example of flute, the fifth example of speak). Moreover, our model can precisely pinpoint the sound source location at a fine-grained level, such as the strings of a violin and the hands of the performer, and the speaker's mouth. In addition, DG-SCT achieves excellent results on AVS task. The right part of Fig. 3 indicates that our model can accurately segment the pixels of sounding objects and outline their shapes perfectly. The excellent qualitative results of our model on various downstream tasks illustrate its strong potential for generalization.
As depicted in Fig. 5, we employ t-SNE [34] to visualize the learned audio and visual features and compare features generated by our model with features generated by baseline without DG-SCT, on various tasks. Each spot denotes the feature of one audio or visual event, and each color corresponds to a particular category, such as "cat" in orange as shown in Fig. 5 (AVE). As we can see, features extracted by the proposed DG-SCT are more intra-class compact and more inter-class separable. These meaningful visualizations further demonstrate that the DG-SCT model successfully learns compact and discriminative features for each modality across diverse downstream tasks.
4.7 Efficiency analysis
Our efficiency analysis on AVE task is presented in Table 6. Our approach utilizes more trainable parameters. However, our proposed DG-SCT attention mechanism requires a comparable number of parameters to the latent tokens utilized in LAVisH [17]. The increase trainable parameters primarily arises from the inclusion of a two-dimensional convolution kernel and a linear projection (section 3.2). These additions ensure consistency of dimensions between the audio and visual prompts. In other words, the increase in parameter count from our approach mainly results from addressing the inconsistency in dimensions between the audio and visual encoders.
We also conducted a comparison of computational cost [29] on AVE task. Our approach involves fine-tuning the pre-trained model, which inevitably leads to a reduction in speed compared to previous late-interaction baselines (CMBS [36]). However, we have achieved outstanding results, and our approach is applicable to multiple audio-visual tasks. Overall, the benefits are substantial.
This paper introduces DG-SCT, a method that leverages audio and visual modalities as prompts in the early layers of frozen pre-trained encoders. By doing so, our model can extract higher-quality
Figure 5: Qualitative visualizations of original visual (top row), our visual (second row), original audio (third row), and our audio (bottom row) features on AVE, AVVP, and AVS tasks.
Table 6: Efficiency analysis on AVE task. For trainable parameters of LAVisH* and our model, the first number (17.3) represents the trainable parameters of a two-dimensional convolution kernel and a linear projection.
Method	Trainable Params (%)	Total Params (M)	GFLOPs	Acc
CMBS [36]	14.4	216.7	40.9	79.3
LAVish [17]	10.1	238.8	406.7	79.7
LAVish*	17.3+13.3=30.6	374.9	416.1	78.6
Ours	17.3+26.3=43.6	461.3	460.8	82.2
and finer-grained audio-visual features, enhancing performance in subsequent tasks. We conduct comprehensive experiments on four datasets, including AVE, AVVP, AVS, and AVQA, as well as our newly proposed few-shot/zero-shot audio-visual tasks. Across 25 experimental settings, our approach achieves state-of-the-art results on 19 of them. Additionally, ablation studies conducted on these datasets validate the effectiveness of our proposed spatial, channel, and temporal attention modules. Furthermore, our approach demonstrates robust generalizability and holds potential for application in more audio-visual scenarios in the future.
[44] Jinxing Zhou et al. "Positive sample propagation along the audio-visual event line". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, pp. 8436-8444.
[45] Kaiyang Zhou et al. "Conditional prompt learning for vision-language models". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 16816-16825.
[46] Kaiyang Zhou et al. "Learning to prompt for vision-language models". In: International Journal of Computer Vision 130.9 (2022), pp. 2337-2348.
A Implementation details
For the AVE and the AVQA tasks, we set  $\alpha = 0.3$ ,  $\beta = 0.05$ , and  $\gamma = 0.1$ . We train the model with a batch size of 8 and a learning rate of  $5 \times 10^{-4}$  and  $1 \times 10^{-4}$ , respectively; For the AVVP task, we set  $\alpha = 0.3$ ,  $\beta = 0.05$ , and  $\gamma = 0.05$ . We train the model with a batch size of 8 and a learning rate of  $3 \times 10^{-4}$ ; For the S4 setting of the AVS task, we set  $\alpha = 0.3$ ,  $\beta = 0.05$ , and  $\gamma = 0.05$ . We train the model with a batch size of 8 and a learning rate of  $3 \times 10^{-4}$ ; For the MS3 setting of the AVS task, we set  $\alpha = 0.2$ ,  $\beta = 0.1$ , and  $\gamma = 0.1$ . We train the model with a batch size of 2 and a learning rate of  $1.5 \times 10^{-4}$ .
For few-shot/zero-shot tasks, we set the learning rate to  $3 \times 10^{-4}$  with a batch size of 2. For the AVE task, we set  $\alpha = 0.2$ ,  $\beta = 0.05$ , and  $\gamma = 0.01$ ; For the AVE classification and the LLP classification tasks, we set  $\alpha = 0.2$ ,  $\beta = 0.05$ , and  $\gamma = 0.05$ .
All of our experiments are trained on one NVIDIA A100 GPU.
B More details of datasets
Table 7: Dataset statistics. Each dataset is shown with the number of videos and the annotated frames. The "annotations" column indicates whether the frames are labeled by category, pixel-level masks, or answer.
Datasets	Videos	Frames	Classes/Answers	Types	Annotations
AVE [33]	4,143	41,430	28	video	category
LLP [32]	11,849	11,849	25	video	category
AVSBench [43]	5,356	12,972	23	video	pixel
VGG-Sound(40K) [2]	40,801	408,010	141	video	category
MUSIC-AVQA [14]	9,288	45,867	42	video	answer
AVE dataset  $^{5}$  We evaluate the AVE task on the AVE dataset [33] originating from the AudioSet [9]. The AVE dataset contains 4,143 videos covering 28 categories. Each video lasts for 10 seconds and contains an event category labeled for each video on a segment level.
LLP dataset<sup>6</sup> We evaluate the AVVP task on the LLP dataset [32], which has 11,849 video-level event annotations on the presence or absence of different video events and each video is 10s long and has at least 1s audio or visual events. There are 7,202 videos that contain events from more than one event category and per video has averaged 1.64 different event categories. For evaluation, there are 4,131 audio events, 2,495 visual events, and 2,488 audio-visual events for the 1,849 videos.
AVSBench dataset<sup>7</sup> We evaluate the AVS task on the AVSBench dataset [43], which contains two settings: 1) Single Sound Source Segmentation (S4), which contains 4,932 videos over 23 categories; 2) Multiple Sound Source Segmentation (MS3), which contains 424 videos over 23 categories. Each video is 5 seconds long.
MUSIC-AVQA dataset  $^{8}$  We conduct our experiments of the AVQA task on the MUSIC-AVQA dataset [14], which contains 9,288 videos, 45,867 question-answer pairs, 33 question templates, and 42 answers.
VGG-Sound(40K) $^9$  The pre-training data of our zero-shot task is VGG-Sound(40K), which is split from the VGG-Sound dataset [2]. VGG-Sound contains over 200k clips for 300 different sound classes, while our VGG-Sound(40K) has 40,801 clips for 141 sound classes.
<sup>5</sup>https://github.com/YapengTian/AVE-ECCV18
<sup>6</sup>https://github.com/YapengTian/AVVP-ECCV20
<sup>7</sup>http://www.avlbench.openplab.cn/download.
<sup>8</sup>https://gewu-lab.github.io/MUSIC-AVQA/.
<sup>9</sup>https://www Robots.ox.ac.uk/~vgg/data/vggsound/.
Figure 6: Details of our framework in few-shot/zero-shot scenarios. Right: two text branches are used to operate image-text and audio-text matching, respectively. The final result is obtained by adding the weighted sum of the two outputs from image-text matching and audio-text matching.
C More details of few-shot/zero-shot tasks
C.1 Framework and method
As shown in Fig. 6, our DG-SCT module is injected into every layer of the frozen CLIP [26] image encoder and the frozen CLAP [5] audio encoder. After that, the image encoder and the audio encoder are denoted as  $f_{v}(\cdot)$  and  $f_{a}(\cdot)$ , respectively. Two text branches  $f_{t}^{v}(\cdot)$ ,  $f_{t}^{a}(\cdot)$  are employed to perform image-text and audio-text matching, respectively, as the text encoders from different pre-trained models exhibit a substantial gap, using only one of them will result in a significant decrease in accuracy.
Let  $(\nu_{i},a_{i},t_{i})$  represents a piece of visual-audio-text pair indexed by  $i$ . The visual embedding  $E_{i}^{v}$ , the audio embedding  $E_{i}^{a}$ , and the corresponding text embeddings  $T_{i}^{v}$  and  $T_{i}^{a}$ , are obtained by encoders with projection layers, respectively:
$$
E _ {i} ^ {v} = \operatorname {M L P} _ {v} \left(f _ {v} \left(v _ {i}\right)\right), \quad E _ {i} ^ {a} = \operatorname {M L P} _ {a} \left(f _ {a} \left(a _ {i}\right)\right), \quad T _ {i} ^ {v} = \operatorname {M L P} _ {t} ^ {v} \left(f _ {t} ^ {v} \left(t _ {i}\right)\right), \quad T _ {i} ^ {a} = \operatorname {M L P} _ {t} ^ {a} \left(f _ {t} ^ {a} \left(t _ {i}\right)\right), \tag {8}
$$
where the visual/audio/text projection layer is a 2-layer multilayer perception (MLP) with ReLU as the activation function to map the encoder outputs into the same dimension  $D$  (i.e.,  $E_{i}^{v}, E_{i}^{a}, T_{i}^{v}, T_{i}^{a} \in \mathbb{R}^{D}$ ).
The model is trained with the contrastive learning paradigm between the paired visual and text embeddings, as well as the paired audio and text embeddings, following the same loss function in [26]:
$$
\mathcal {L} _ {v} = \frac {1}{2 N} \sum_ {i = 1} ^ {N} \left(\log \frac {\exp \left(E _ {i} ^ {v} \cdot T _ {i} ^ {v} / \tau_ {v}\right)}{\sum_ {j = 1} ^ {N} \exp \left(E _ {i} ^ {v} \cdot T _ {j} ^ {v} / \tau_ {v}\right)} + \log \frac {\exp \left(T _ {i} ^ {v} \cdot E _ {i} ^ {v} / \tau_ {v}\right)}{\sum_ {j = 1} ^ {N} \exp \left(T _ {i} ^ {v} \cdot E _ {j} ^ {v} / \tau_ {v}\right)}\right), \tag {9}
$$
$$
\mathcal {L} _ {a} = \frac {1}{2 N} \sum_ {i = 1} ^ {N} \left(\log \frac {\exp \left(E _ {i} ^ {a} \cdot T _ {i} ^ {a} / \tau_ {a}\right)}{\sum_ {j = 1} ^ {N} \exp \left(E _ {i} ^ {a} \cdot T _ {j} ^ {a} / \tau_ {a}\right)} + \log \frac {\exp \left(T _ {i} ^ {a} \cdot E _ {i} ^ {a} / \tau_ {a}\right)}{\sum_ {j = 1} ^ {N} \exp \left(T _ {i} ^ {a} \cdot E _ {j} ^ {a} / \tau_ {a}\right)}\right), \tag {10}
$$
Where  $\tau_v$  and  $\tau_a$  are learnable temperature parameters for scaling the loss.  $N$  is usually the number of data, but during the training phase,  $N$  is used as the batch size. Let  $y_v$  denotes the image-text
Figure 7: Results of our model and previous methods on few-shot/zero-shot tasks.
matching score, and  $y_{a}$  denotes the audio-text matching score, we set hyperparameters  $w_{1} = \frac{y_{v}}{y_{v} + y_{a}}$  and  $w_{2} = 1 - w_{1}$ , thus the overall objective function is:
$$
\mathcal {L} = w _ {1} \cdot \mathcal {L} _ {v} + w _ {2} \cdot \mathcal {L} _ {a}, \tag {11}
$$
When the image-text matching score is higher, it indicates that the visual modality is more certain in determining the category of the sample, therefore the model needs to increase the weight of visual modality in the loss function, and vice versa.
The same weight settings for  $w_{1}$  and  $w_{2}$  in Fig. 6 are also significant during the inference process, indicating that some samples require more guidance from the audio modality while others might require more guidance from the visual modality.
For the few-shot setting, we train the model by selecting shot samples of each category from the training dataset of the downstream tasks. For the zero-shot setting, we pre-train the model using the VGG-Sound(40K) dataset.
C.2 Evaluation metrics
For the AVE task, the category label of each video segment is required to be predicted in supervised manner. We adopt [33, 37, 36], and exploit the overall accuracy of the predicted event category as the evaluation metrics; For the AVE classification task, the category label of each video is required to be predicted. We leverage the overall accuracy of the video category as the evaluation metrics; For the LLP classification task, the category label of each video is required to be predicted. Note that some videos might have multiple categories, we randomly select one as ground truth for brevity. We leverage the overall accuracy of the video category as the evaluation metrics.
Table 8: Ablation study of A2V and V2A modules on AVE, AVS, and AVQA tasks.
Module	AVE	AVS	AVQA
A2V	V2A	Acc	S4	MS3	AQ	VQ	AVQ	Avg
MJ	MF	MJ	MF
-	-	78.6	78.0	87.0	49.1	59.9	75.4	79.6	70.1	73.6
✓	-	79.2	80.7	89.0	51.4	61.6	76.1	82.0	70.8	74.7
-	✓	81.3	79.1	88.0	50.7	62.4	75.9	80.8	70.8	74.3
✓	✓	82.2	80.9	89.2	53.5	64.2	77.4	81.9	70.7	74.8
C.3 Results
As shown in Fig. 7 (a), (b) and (c), for few-shot ( $\text{shot} = 16$ ) setting, our method outperforms MaPLe by a significant margin on AVE, AVE classification, and LLP classification tasks, achieving  $72.4\%$ ,  $84.2\%$ , and  $32.6\%$ , respectively. However, the performance of our model on both the AVE and LLP classification tasks may not be as effective as some previous methods when  $\text{shot} < 8$ . The classification tasks are relatively simple and are already closely related to the image-text matching task in the upstream pre-training of the CLIP model. Therefore, the CLIP model's pre-trained parameters can achieve good results. However, when  $\text{shot} > 8$ , our model gains significant advantages. We also observe that the accuracy curves of other few-shot learning methods are relatively flat, particularly for
(a)  $\alpha$
(b)  $\beta$
Figure 8: Hyperparameters. We explore the impact of the hyperparameters  $\alpha$ ,  $\beta$ , and  $\gamma$  on the experimental results of the AVE task. a) Fix  $\beta = 0.05$  and  $\gamma = 0.1$ , observe the impact of  $\alpha$ ; b) Fix  $\alpha = 0.3$  and  $\gamma = 0.1$ , observe the impact of  $\beta$ ; c) Fix  $\alpha = 0.3$  and  $\beta = 0.05$ , observe the impact of  $\gamma$ .
(c)  $\gamma$
the LLP classification task, where the accuracy sometimes decreases when shot increases. This issue might be the limited ability of the image-text models to extract rich information from the training data of audio-visual tasks. Thus, the potential for improving accuracy is limited. For zero-shot setting, our model also achieves state-of-the-art on all three tasks. The results demonstrate that our model can extract audio-visual task information more effectively and achieve better results.
Table 9: Ablation study of A2V and V2A modules on AVVP task.
Module	Segment-level	Event-level
A2V	V2A	A	V	AV	Type	Event	A	V	AV	Type	Event
-	-	57.8	56.3	49.8	55.2	54.9	48.2	51.7	43.9	48.8	47.6
✓	-	56.4	59.5	53.3	56.4	55.0	47.4	55.9	46.3	49.9	47.8
-	✓	59.4	57.3	50.8	55.8	56.8	49.2	54.1	44.4	49.2	48.6
✓	✓	59.0	59.4	52.8	57.1	57.0	49.2	56.1	46.1	50.5	49.1
D More ablation studies
Next, we conduct more ablation studies to investigate how different components of our model affect the performance on the downstream tasks.
A2V and V2A modules. We first investigate the effectiveness of the bidirectional attention mechanism. We compare our final bidirectional approach with the unidirectional variants that only use either audio-to-visual (A2V) or visual-to-audio (V2A) spatial-channel-temporal attention mechanism, and also a baseline that does not use any cross-modal connections.