SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As indicated in Table 8, integrating A2V or V2A individually leads to substantial performance enhancements across AVE, AVS, and AVQA tasks compared to the baseline model. Furthermore, the bidirectional DG-SCT outperforms the unidirectional A2V and V2A baselines (e.g.,  $3.0\%$  and  $0.9\%$  on the AVE task). In Table 9, we observe that using the A2V module alone does not significantly decrease the accuracy for visual and audio-visual events. However, without visual guidance (V2A), the performance of audio events suffers a considerable decline; Likewise, the performance of visual events drops without audio guidance (using the V2A module alone). These experimental findings demonstrate the necessity of visual guidance for audio events and the need for audio guidance for visual events. Our proposed DG-SCT model can bidirectionally guide the representation of each modality, thus enhancing the accuracy of downstream audio-visual tasks.
Hyperparameters. Now, we explore the impact of the hyperparameters  $\alpha$ ,  $\beta$ , and  $\gamma$  on the experimental results of the AVE task. As we can see in Fig. 8, our model achieves the best accuracy  $(82.2\%)$  when  $\alpha = 0.3$ ,  $\beta = 0.05$ , and  $\gamma = 0.1$ .
Figure 9: Qualitative examples of the baseline method (w/o DG-SCT) and our DG-SCT framework, under the S4 setting of the AVS task.
E Additional qualitative analyses
Qualitative examples of the AVS task. We provide some qualitative examples of the AVS task to test the effectiveness of the DG-SCT model. As shown in Fig 9, under the S4 setting, the DG-SCT model can not only outline the object shapes more perfectly (shown in the upper left, upper right, and bottom left figures) but also locate the sounding objects more accurately than the baseline model without the DG-SCT module. As we can see in the bottom right figure of Fig. 9, DG-SCT can locate the dog that is barking, simultaneously excluding the mower that hasn't produced sound. The baseline model, however, mistakenly locates the dog and the mower at the same time. In Fig 10, under the MS3 setting, the DG-SCT model can locate multiple sound sources and outline their shape nicely. In some difficult cases (the bottom row in Fig. 10), DG-SCT can still pinpoint the correct sounding source even if the sound has stopped or the scene has changed dramatically.
Qualitative examples of the AVVP task. In Fig. 11, we visualize the video parsing results of DG-SCT and baseline (w/o DG-SCT) on different examples. "GT" denotes the ground truth annotations. The results show that adding the DG-SCT module achieves more accurate parsing performance by acquiring mutual guidance from audio and visual modalities during the representation. For example, in Fig. 11 (b), our model can recognize the mixture of "banjo" and "singing" while the baseline model (w/o DG-SCT) falsely predicts the sound as "violin fiddle".
Figure 10: Qualitative examples of the baseline method (w/o DG-SCT) and our DG-SCT framework, under the MS3 setting of the AVS task.
(b)
Figure 11: Qualitative examples of the baseline method (w/o DG-SCT) and our DG-SCT framework, under the AVVP task.