SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation Details. We conduct the training and evaluation processes on a single NVIDIA GTX 2080 Ti GPU with 11 GB memory. Following the data preprocessing in previous works, we decode a 10-second video at 8 fps into 10 segments. Instead of using pre-trained ResNet-152 [45] and R(2+1)D [51], we also utilize CLIP's [7] image encoder (i.e., pre-trained ViT-B/16 [52]) to extract visual features with richer semantic information. Furthermore, we leverage CLAP [8] pretrained on Audioset [53] to extract the audio features instead of using VGGish [46]. Following HAN [4] we adopt the Adam optimizer and the learning rate 2e-4 drops by a factor of 0.25 for every 6 epochs. We train the model with a batch size of 32 for 20 epochs. We also add layers of HAN [4] to increase the complexity of the network for better performance.  $\alpha$  is set to 4 and  $\beta$  is 0.4.
To demonstrate the effectiveness of our method, we compare it to several baseline works including the audio-visual event localization approach AVE [1] and AVSDN [16] and the state-of-the-art methods for the weakly-AVVP task, such as HAN [4], MA [5], JoMoLD [6], etc.
Table 3: Analysis of different strategies to indicate that no event appears in the prompt.
Method	Segment-Level	Event-level
A	V	A-V	Type	Event	A	V	A-V	Type	Event
[Class, Not Class]	40.8	63.1	39.3	47.7	51.9	33.8	59.8	33.5	42.3	44.4
[Class, None]	66.3	71.4	61.2	66.3	67.1	57.9	67.8	53.5	59.7	58.7
[Class, Other]	68.7	71.3	63.4	67.8	68.2	61.5	67.4	55.9	61.6	60.6
Table 4: Algorithm analysis of dynamic re-weighting. All results are of visual metric.
(a) Analysis on different  $\alpha$  and  $\beta$  values.
\(\alpha\)β	0.2	0.4	0.6	0.8
3.5	71.1	71.1	71.3	71.0
4	71.2	71.3	70.9	70.6
4.5	70.3	70.9	70.3	70.2
5	69.0	70.3	70.8	69.9
(b) Analysis of different re-weighting strategies.
Method	Visual
Segment-Level	Event-Level
re-weight on w/ event	70.4	66.8
re-weight on w/o event	70.7	67.0
stable re-weight	70.5	66.8
dynamic re-weight	71.3	67.8
As shown in Table 1, we conduct our experiments on the LLP dataset with all state-of-the-art methods. We observe that our method has competitive performance on all evaluation metrics. Compared with our baseline method HAN [4], the proposed method can significantly improve the performances on visual metrics. Especially, our method gains 14.2 and 10.5 points on segment-level visual parsing and audio-visual parsing metrics with ResNet and VGGish features, respectively. Similar improvements are also observed on event-level metrics.
Compared with state-of-the-art methods, our method still yields better performance on all visual-related metrics. Especially, compared with JoMoLD, we achieved 3.3, 2.2, and 2.3 points of improvement in terms of segment-level visual, audio-visual, and type parsing. With the CLAP and CLIP features, we improve the segment-level audio metrics from  $62.7\%$  to  $68.7\%$ , and visual metrics from  $67.1\%$  to  $71.3\%$ .
Effectiveness of language guided segment-level denoising. The ablation study of denoising is shown in Table 2. We observe that the results on segment-level audio/visual parsing are improved by  $3.2\% /15.2\%$  when we construct prompts and remove segment-level noisy labels by language guidance. The significant improvement demonstrates that with the help of language description, our method does provide more accurate segment-level labels for both audio and visual modality, which enhances network learning.
Analysis of different types of prompts. As shown in Table 3, we investigate the influence of using different language descriptions to express that no event appears in the segment. We observe that when using prompts like [Class, Not Class], the performance is much lower than our method. We suspect the reason is that the text encoder is more sensitive to meaningful words like event class while neglecting the negative word not. When using none, the result in audio-visual metrics is about 2.2 points lower than ours. The reason is that though the pre-defined events do not appear, background scenes like sky and trees may exist. When using the description other, we obtain the best performance, where we neither throw away the possibility of the appearance of background classes nor introduce event categories that do not appear.
Effectiveness of dynamic re-weighting. The ablation study of dynamic re-weighting is shown in Table 2. We observe that the results on visual parsing are improved by  $1.2\%$  and  $1.1\%$  on segment-level and event-level, respectively. The improvement demonstrates that our method accurately finds the possible mislabeled segments to learn them softly.
Analysis of different  $\alpha$  and  $\beta$  values. According to Eq. 4, the hyper-parameter  $\alpha$  and  $\beta$  are used to be multiplied with similarity to serve as the weight for unreliable segments with/ without the event, respectively. In Table 4 (a), as  $\alpha$  increased from 3.5 to 4, the results are usually improved, as our denoising method identifies most of the segments into the correct category, so a larger weight should
(a) Speech.
(b) Chicken_rooster.
(c) Frying food.
Figure 2: Similarity percentage distribution of the visual modality. The similarity is calculated between the visual segments and the event description. We split the segments into with (blue)/ without (orange) the event. Examples of three events are shown.
Figure 3: Qualitative result on the LLP test set. The results of our method (LSLD) and the state-of-the-art method JoMoLD [6] are shown. "GT" denotes the ground truth annotations.
be given. However, when  $\alpha$  is too big, the performance drops, because most of the labels will remain at 1 without any re-weighting to unreliable segments. A similar observation is found in  $\beta$ .
Analysis of different re-weighting strategies. As shown in Table 4 (b), four different strategies are compared. The first two rows denote conducting dynamic re-weighting on only segments with or without the event, where the results are  $0.9\%$  and  $0.6\%$  lower than ours on segment-level visual metrics, respectively. "stable re-weight" denotes using stable value instead of weighted similarity as the soft label, where we find the results are  $0.8\%$  and  $1\%$  lower than ours in terms of segment-level and event-level. This demonstrates that since segments with higher similarity are more likely to belong to the category, re-weighting with dynamic similarity helps to better learn the unreliable segments.
In addition, we visualize three examples of similarity distribution with/ without an event in Fig. 2. We observe that in most cases, such as chicken rooster and frying food, the similarity distribution is well separated, while only a small area is taken as unreliable segments. However, there is a special case, speech, where we can hardly recognize whether it happens through the similarity with the language prompt of the class. We suspect the reason is that speech is difficult to recognize in vision. For example, the vision is about a basketball game while speech could only occur in the audio track.
Visualization of video parsing results. We compare our method with the SOTA method JoMoLD [6] by visualizing the audio and visual parsing results in Fig. 3. As for the visual parsing task, the visual track only contains speech on the segments, where JoMoLD recognizes it as the Violin_fiddle, and our method identifies the speech event and localizes the temporal boundaries correctly. The results prove
(a) Video with Speech and Car
(b) Video with Cello and Violin_fiddle
(c) Video with Singing
(d) Video with Baby laughter and Speech
Figure 4: Visualization of segment-level label denoising on visual track. The similarities between our prompts and segments are presented. Results on JoMoLD [6] and our method are shown. Each example presents three segments of a video.
that our approach is more effective in recognizing events in visual segments. As for the audio parsing task, our LSLD properly identifies the segments with speech, yet JoMoLD only recognizes speech on some of the segments. This demonstrates that LSLD not only works well on the visual track but is also capable of identifying the events that appear on the audio segment as well. However, when identifying the Violin_fiddle, both LSLD and JoMoLD only recognize the event in some segments due to the interference of the speech voice, which impels us to further refine the method on the audio modality.
Visualization of the denoising procedure. We visualize the segment-level label denoising results of our method and JoMoLD [6] in Fig. 4. We observe that, in all examples, our prompt of the highest similarity captures the appeared events accurately. As shown in Fig. 4 (a), in the last two segments there is no event exists, where JoMoLD still recognizes Car because Car appears in the first segment and JoMoLD can only denoise at video-level. In contrast, LSLD correctly assigns labels for all the segments. In Fig. 4 (b), LSLD recognizes different event appearances on the segment level, while JoMoLD fails to do so, which demonstrates the importance of segment-specific label assignment and the effectiveness of our method. In Fig. 4 (c), LSLD successfully removes Singing on the first segment, yet JoMoLD is unable to remove it. A similar observation could be found in Fig. 4 (d).
In this paper, we address the weakly-supervised audio-visual video parsing task. Instead of focusing on the overall video instance level denoising, we propose to assign specific segment-level labels for each video on the audio and visual track. Since each segment label is not predefined but could be any combination of events that occur in the video, we introduce the language modality into our task and construct language prompts to represent all cases of event appearance. The similarities between each segment and prompts are calculated, where the prompt with the highest similarity is considered the segment-level label. To further alleviate the effect of segment-level label noise, we apply dynamic weighting on visual segments that may be mislabeled. Extensive experiments and qualitative results show that our method outperforms the state-of-the-art methods on the visual metrics by a large margin.
Limitation. Since only the LLP dataset currently has fine-grained (segment-level and modality-level) annotations and labels for audio-visual understanding, we only perform LSLD on the LLP dataset. In future work, more effort will be made to explore the effectiveness of our method on other audio-visual datasets.
Broader impact. Audio-Visual Video Parsing (AVVP) has numerous potential real-life applications, such as professional media production, audiovisual archive management, entertainment, etc. Our
method provides a new perspective to address AVVP, i.e., through the guidance of natural language, which could benefit from large-scale vision-language models and will inspire future works.
Acknowledgment. This work was supported in part by the National Natural Science Foundation of China under Grant 62001331, in part by the Natural Science Foundation of Hubei Province under Grant 2021CFB475, and in part by the Special Fund of Hubei Luojia Laboratory under Grant 220100018.
[1] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 247–263, 2018.
[2] Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9248-9257, 2019.
[3] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12475–12486, 2021.
[4] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly-supervised audio-visual video parsing. In Proceedings of the European Conference on Computer Vision (ECCV), pages 436–454, 2020.
[5] Yu Wu and Yi Yang. Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1326–1335, 2021.
[6] Haoyue Cheng, Zhaoyang Liu, Hang Zhou, Chen Qian, Wayne Wu, and Limin Wang. Joint-modal label denoising for weakly-supervised audio-visual video parsing. In Proceedings of the European Conference on Computer Vision (ECCV), pages 431–448, 2022.
[7] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, 2021.
[8] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
[9] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Advances in neural information processing systems, 2016.
[10] Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2827-2836, 2016.
[11] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pages 609-617, 2017.
[12] Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. Advances in Neural Information Processing Systems, 2018.
[13] Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems, pages 9758-9770, 2020.
[46] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In 2017 iee international conference on acoustics, speech and signal processing (icassp), pages 131-135, 2017.
[47] Yan-Bo Lin, Hung-Yu Tseng, Hsin-Ying Lee, Yen-Yu Lin, and Ming-Hsuan Yang. Exploring cross-video and cross-modality signals for weakly-supervised audio-visual video parsing. Advances in Neural Information Processing Systems, pages 11449–11461, 2021.
[48] Xun Jiang, Xing Xu, Zhiguo Chen, Jingran Zhang, Jingkuan Song, Fumin Shen, Huimin Lu, and Heng Tao Shen. Dhn: Dual hierarchical hybrid network for weakly-supervised audio-visual video parsing. In Proceedings of the 30th ACM International Conference on Multimedia, pages 719–727, 2022.
[49] Jiashuo Yu, Ying Cheng, Rui-Wei Zhao, Rui Feng, and Yuejie Zhang. Mm-pyramid: Multimodal pyramid attentional network for audio-visual event localization and video parsing. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6241–6249, 2022.
[50] Junyu Gao, Mengyuan Chen, and Changsheng Xu. Collecting cross-modal presence-absence evidence for weakly-supervised audio-visual event perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18827-18836, 2023.
[51] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450-6459, 2018.
[52] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[53] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780, 2017.