SlowGuess/NeurIPS_2023_OCR huggingface dataset:
5.2 Unified Label Elaboration for State-of-the-Art Audio-Visual Video Parsing
To demonstrate the effectiveness of VALOR, we evaluate our method on the AVVP benchmark. Existing works include: 1) weakly-supervised audio-visual event localization methods AVE and AVSDN, 2) HAN and its network architecture advancements MM-Pyramid, MGN, CVCMS, and DHHN, and 3) different label refinement methods MA, JoMoLD, and VPLAN. We report the results on the test split of the LLP dataset in Table 1.
We achieve the new state-of-the-art (SOTA) on all metrics consistently with a large margin. Our method VALOR significantly improves the baseline (HAN) by 8.0 in segment-level Type@AV. Compared to previous published SOTA, JoMoLD, VALOR scores higher on all metrics, including the 5.4 F-score improvement for segment-level Type@AV, under a fair setting. With light hyperparameter tuning, VALOR+ further achieves a significant 2.4 improvement on Type@AV, with a deeper yet thinner HAN while keeping a similar number of trainable parameters. Our improvement on the audio side w.r.t. the concurrent preprint VPLAN is more significant than the visual side, which may be attributed to our effective audio teacher CLAP and label elaboration along the modality axis. We empirically conclude that VALOR has successfully unified label refinement along both modality and temporal dimensions. To push to the limits, we further proposed VALOR++ by replacing the feature extraction models with CLIP and CLAP, achieving another consistent boost, including 3.0 in segment Type@AV. We will release the VALOR++ pre-trained checkpoint, features, and harvested labels to boost future AVVP research.
Table 2: Selection of modality-independent labeler. Note that utilizing a cross-modal labeler HAN instead of CLIP and CLAP to generate segment-level labels hardly improves the baseline (HAN). On the other hand, modality-less segment-level labels deteriorates the performance. All results are reported on the validation split of LLP.
Dense Labeler	Modality Label	Segment-level	Event	Event-level
A	V	AV	Type	Event	A	V	AV	Type	Event
None	✓	62.0	54.5	50.2	55.6	57.1	53.5	50.5	43.6	49.2	50.3
HAN	✓	62.1	56.4	52.1	56.8	57.6	53.4	52.0	45.4	50.3	50.6
CLIP&CLAP	×	41.0	59.0	34.5	44.9	52.1	33.2	56.2	28.2	39.2	43.1
CLIP&CLAP	✓	62.7	66.3	61.0	63.4	61.8	55.5	62.0	54.1	57.2	53.8
Table 3: Fidelity of the elaborated labels. We conduct a comparison between the segment-level labels generated from VALOR and those from a naive approach where we assume video-level labels also serve as segment-level labels. We directly evaluate these pseudo labels on the validation split before training. The results clearly indicate that VALOR-generated labels are more accurate than the naive ones.
Label Generation Methods	Audio	Visual	Audio-Visual
Video Labels	80.08	67.21	59.45
VALOR	85.07 (+4.99)	82.14 (+14.93)	77.07 (+17.62)
The impressive results achieved in Table 1 are based on careful design. In this subsection, we elaborate on why we choose CLIP and CLAP to synthesize dense labels with modality annotations with empirical support. Furthermore, we break down the loss function and modeling components into orthogonal pieces and evaluate their individual effectiveness.
How to choose the labeler? In Table 2, we show the necessity of modality-independent pre-trained models (CLIP and CLAP) over the multi-modal model (HAN) as the labeler (2nd row) and that modality-aware labels beat modality-agnostic labels (3rd row). We aim to demonstrate the necessity and importance of using large-scale pre-trained uni-modal models to annotate modality-aware segment-level labels. To validate the former, we employ a baseline model (HAN) that has been trained on AVVP to individually annotate segment-level labels within the two modalities. Experimental results show that modality-aware temporal dense labels generated by a multi-modal model (HAN), learned from weak labels, are less effective than those generated by large-scale pre-trained uni-modal models (CLIP and CLAP), thereby underscoring the essentiality of using large-scale pre-trained uni-modal models. Subsequently, to validate the latter, we generate modality-agnostic segment-level labels from CLIP and CLAP, meaning that these labels only reveal the events occurring in each segment but do not disclose the modality of the event. As seen from the third row of Table 2, while such a labeling method increases the F-score for visual events, it dramatically decreases the F-score for audio events. The overall performance (Type F-score) is even worse than that of HAN (the first row), clearly indicating the importance of modality-aware labeling for the model to learn the AVVP task effectively.
How accurate are the elaborated labels? To measure the fidelity of the pseudo labels generated via VALOR in audio and visual modalities, we conduct a comparison between the segment-level labels generated from VALOR and those from a naive approach where we assume that video-level labels also serve as segment-level labels, i.e., we assume that an event occurs in both modalities and all segments if it occurs in the video. We directly evaluate these pseudo labels on the validation split before using them for training. The results, presented in Table 3, clearly demonstrate the superiority of our generated segment-level audio and visual pseudo labels compared to the naive counterparts. Notably, our segment-level visual F-score exceeds the naive approach by nearly 15 points while the audio-visual F-score significantly surpasses for more than 17 points. These results highlight the reliability of the VALOR-generated pseudo labels, which provide more faithful temporal and modal information to facilitate model training.
Table 4: Ablation study. "global" denotes only video-level labels observed, while "dense" indicates segment-level labels available as ground truth. "base" is the baseline method [73]. "New Feat." denotes the use of features from CLAP, CLIP, and  $\mathrm{R}\left( {2 + 1}\right) \mathrm{D}$  ,and "Deep HAN" is that of the 256-dim 4-layer HAN model. All results are reported on the validation split of LLP.
Audio Loss
global	dense	Visual Loss
global	dense	New
Feat.	Deep
HAN	Segment-level
A	V	AV	Type	Event
base	X	base	X	X	X	62.0	54.5	50.2	55.6	57.1
X	KD	X	KD	X	X	51.1	64.0	48.0	54.3	55.5
VALOR	X	VALOR	X	X	X	62.1	65.8	59.0	62.3	61.2
base	X	X	VALOR	X	X	60.5	66.7	60.8	62.7	59.8
X	VALOR	base	X	X	X	62.2	54.5	52.7	56.5	56.5
X	VALOR	X	VALOR	X	X	62.7	66.3	61.0	63.4	61.8
X	VALOR	X	VALOR	X	✓	64.5	67.1	63.1	64.9	63.2
X	VALOR	X	VALOR	✓	✓	71.4	69.4	64.9	68.6	69.7
Figure 3: Ablation study of whether using video-level labels as filters.
Figure 4: The extent to which the models address the modality non-alignment issue.
How to use the elaborated labels? We conduct an ablation study on utilizing CLIP and CLAP together. The results are presented in Table 4. The replacement of the smoothed video-level event labels  $\tilde{y}^a$  and  $\tilde{y}^v$  with their respective refined weak labels  $\hat{y}^a$  and  $\hat{y}^v$  derived from our method leads to a significant increase in the Type@AV F-score, from 54.0 to 60.8. This finding underscores the importance of incorporating labels that are proximal to ground truth, albeit weak. Furthermore, we leverage the CLIP and CLAP models to generate segment-level labels for each modality. This approach results in an improvement of 8.0 Type@AV F-score over the baseline, indicating that explicitly informing the model of the events occurring in each segment of the audio-visual video facilitates the learning of the Audio-Visual Video Parsing (AVVP) task. In addition, CLIP and CLAP are also used to obtain more representative features. Replacing the ResNet-152 and VGGish features with CLIP and CLAP features yields a Type@AV F-score improvement of 4.0.
Whether using video-level labels as filters? Video-level labels are pivotal for generating reliable pseudo labels in our method, where we employ them as filters to eliminate impossible events misclassified by CLIP or CLAP. In Figure 3, we conduct experiments to underscore the necessity of using video-level labels as filters. Notably, without utilizing video-level labels as filters, both audio and visual F-scores plummet, reaching 47.9 and 53.8, respectively.
How well can the modality non-aligned problem be solved? As we have pointed out that the modality independence of events is one of the crucial challenges in the AVVP task, we assess the extent of the modality non-aligned problem in the LLP dataset and the extent to which the models can solve the problem. First, we define the word "segment-level event" as the cumulative sum of the number of events that occur without modality differences across all segments. In other words, if an audio event and a visual event from the same category occur within a segment, they are counted as a single "segment-level event." In the LLP dataset's validation split, there are 9126 segment-level events. Among these, 4048 segment-level events are modality non-aligned, i.e., they occur in exactly one modality. To measure how well trained models address the modality non-aligned issue, we conduct experiments involving several models, including our own, to predict both the modality and event of these segments. A successful prediction entails correctly identifying the event and confirming its presence in both modalities. The results, as displayed in Figure 4, reveal that HAN exhibits the poorest performance in predicting modality non-aligned events. Conversely, our methods VALOR and VALOR++ outperform the prior SOTA, JoMoLD. This highlights the effectiveness of our approach in mitigating the modality non-alignment challenge within the AVVP task.
Figure 5: Qualitative Comparison with Previous AVVP Works. "GT" denotes the ground truth annotations. We compare with HAN [73] and JoMoLD [11].
5.4 Generalize VALOR to Audio-Visual Event Localization
In this section, we showcase the additional generalization ability of VALOR by applying it to the Audio-Visual Event Localization (AVE) task. We consider the weakly-supervised version of AVE, where segment-level ground truths are not available to the model during training, meaning that no timestamp is provided for the event, motivating us to apply VALOR to harvest event labels. Without task-specific modification, we directly apply HAN and VALOR to AVE. The only difference is that at inference, we combine the audio and visual prediction to obtain the audio-visual event required in this task. Please refer to the supplementary for more implementation details of the AVE task.
Quantitative Results From Table 5, we observe that our baseline method performs on par with the previous state-of-the-art method CMBS [80]. When our method is applied to the model, the accuracy leaps from 75.3 to 80.4, indicating the generalizability of our method. In addition, we surpass CMBS [80] and have become the new state-of-the-art on the weakly-supervised AVE task with an improvement of 4.4 in accuracy.
Table 5: Results on the AVE task.
Method	Accuracy(%)
VGG-like, VGG-19 features
AVEL [72]	66.7
AVSDN [46]	67.3
CMAN [85]	70.4
AVRB [58]	68.9
AVIN [57]	69.4
AVT [44]	70.2
CMRAN [82]	72.9
PSP [95]	73.5
CMBS [80]	74.2
VGG-like, Res-151 features
AVEL [72]	71.6
AVSDN [46]	74.2
CMRAN [82]	75.3
CMBS [80]	76.0
CLAP, CLIP, R(2+1)D features
HAN	75.3
VALOR	80.4
5.5 Additional Analyses
Qualitative Comparison Aside from quantitative comparison with previous AVVP works, we perform a qualitative evaluation as well. We qualitatively compare with the baseline method HAN [73] and the state-of-the-art method JoMoLD [11]. From Figure 5, it can be seen that only our model can correctly predict the "Baby Cry" visual event. HAN not only fails to predict "Baby Cry" correctly but also mistakenly identifies the woman in the video as speaking. In the audio modality, all models correctly predict the presence of "Baby Cry" in the sound, but they also simultaneously misinterprets that someone is talking. Among all models, our model makes the least severe misjudgments."
Class-wise F-score Comparison. We further evaluate the effectiveness of providing accurate uni-modal segment-level pseudo labels for the model training. We visualize class-wise improvements between our generated segment-level labels for each modality and the naive segment-level labels derived from the video-level labels. In Figure 6, we observe that when our audio pseudo labels are used, most of the audio events improve. In Figure 7, when our visual pseudo labels are used, nearly every event's F-score increases. These results indicate the effectiveness of our method in guiding the model to learn events in each modality. For the inferior performance on the "Speech" event, since CLIP is inept at extracting fine-grained visual information, it is not expected to recognize the "Speech" event well, which requires close attention on mouth movements.
Classwise Audio F-score Comparison
Figure 6: Class-wise improvement on audio events. Using the derived audio segment-level pseudo label is advantageous over the baseline using video-level labels as if they were audio segment-level.
Classwise Visual F-score Comparison
Figure 7: Class-wise improvement on visual events. VALOR's visual segment-level labels clearly outperforms the video-level labels. Note that CLIP is applicable to extract global but not fine-grained information from visual inputs. Thus, it is not expected to produce proper visual cues for the "Speech" event, which requires close attention to mouth movements.
We propose Visual-Audio Label Elaboration (VALOR) for weakly-supervised Audio-Visual Video Parsing. By harnessing large-scale pre-trained contrastive models CLIP and CLAP, we generate fine-grained temporal labels in audio and visual modalities, providing explicit supervision to guide the learning of AVVP models. We show that utilizing modality-independent pre-trained models (CLIP and CLAP) and generating modality-aware labels are essential for AVVP. VALOR outperforms all the previous works when comparing in a fair setting, demonstrating its effectiveness. In addition, we demonstrate the generalizability of our method in the Audio-Visual Event Localization task, where we improve the baseline greatly and achieve a state-of-the-art result.
Limitations While VALOR performs well on AVVP, it is uncertain whether it will maintain this efficacy when the number of events to classify expands. Moreover, because CLIP is far from perfect at capturing fine-grained visual details, it may fail to generate precise labels when the subject of the event is small or when the video quality is poor, potentially confounding the model.
Broader Impacts As an event recognition model, VALOR could be applied to future intelligent surveillance systems. While may reduce physical crime concerns, it could on the other hand infringe people's privacy and rights. Since the input consists of videos of people, data privacy issues are inevitable, and it is essential to prioritize data protection against unauthorized access.
We thank National Center for High-performance Computing (NCHC) for providing computational and storage resources. We appreciate the NTU VLL members: Chi-Pin Huang, Kai-Po Chang, Chia-Hsiang Kao, and Yu-Hsuan Chen, for helpful discussions.
[1] Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE TPAMI, 44(12):8717-8727, 2018. 1, 5
[2] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. The conversation: Deep audio-visual speech enhancement. In INTERSPEECH, 2018. 5
[3] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. My lips are concealed: Audio-visual speech enhancement through obstructions. In INTERSPEECH, 2019. 5
[4] Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, et al. Audio visual scene-aware dialog. In CVPR, 2019. 5
[5] Relja Arandjelovic and Andrew Zisserman. Objects that sound. In ECCV, 2018. 5
[6] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. 3
[7] Moitreya Chatterjee, Jonathan Le Roux, Narendra Ahuja, and Anoop Cherian. Visual scene graphs for audio source separation. In ICCV, 2021. 5
[8] Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In ECCV, 2020. 5
[9] Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In CVPR, 2021.
[10] Changan Chen, Sagnik Majumder, Ziad Al-Halah, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning to set waypoints for audio-visual navigation. In ICLR, 2021. 5
[11] Haoyue Cheng, Zhaoyang Liu, Hang Zhou, Chen Qian, Wayne Wu, and Limin Wang. Joint-modal label denoising for weakly-supervised audio-visual video parsing. In ECCV, 2022. 5, 6, 9, 17, 18
[12] Joon Son Chung, Bong-Jin Lee, and Icksang Han. Who said that?: Audio-visual speaker diarisation of real-world meetings. In INTERSPEECH, 2019. 5
[13] Soham Deshmukh, Benjamin Elizalde, and Huaming Wang. Audio retrieval with wavtext5k and clap training. In INTERSPEECH, 2023. 4
[14] Yifan Ding, Yong Xu, Shi-Xiong Zhang, Yahuan Cong, and Liqiang Wang. Self-supervised learning for audio-visual speaker diarization. In ICASSP, 2020. 5
[15] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap: Learning audio concepts from natural language supervision. In ICASSP, 2023. 4
[16] Yingying Fan, Yu Wu, Yutian Lin, and Bo Du. Revisit weakly-supervised audio-visual video parsing from the language perspective. arXiv preprint arXiv:2306.00595, 2023. 5
[17] Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. In ICRA. 5
[18] Chuang Gan, Deng Huang, Peihao Chen, Joshua B Tenenbaum, and Antonio Torralba. Foley music: Learning to generate music from videos. In ECCV, 2020. 1, 5
[19] Ruohan Gao and Kristen Grauman. 2.5 d visual sound. In CVPR, 2019. 5
[20] Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. In ICCV, 2019. 5
[21] Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In ECCV, 2018. 5
[90] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In ICCV, 2019. 5
[91] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In ECCV, 2018. 5
[92] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In ICCV, 2019. 5
[93] Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In ECCV, 2020. 5
[94] Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose-controllable talking face generation by implicitly modularized audio-visual representation. In CVPR, 2021. 5
[95] Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, and Meng Wang. Positive sample propagation along the audio-visual event line. In CVPR, 2021. 5, 9
[96] Jinxing Zhou, Dan Guo, Yiran Zhong, and Meng Wang. Improving audio-visual video parsing with pseudo visual labels. arXiv preprint arXiv:2303.02344, 2023. 5, 6, 17, 18
Table 6: The List of Input Captions and Thresholds for CLIP and CLAP. We add the prompt "A photo of" before each event name to make CLIP's input captions and the prompt "This is a sound of" to make CLAP's input captions.
Events	Input Captions	thresholds θ
CLIP	CLAP	θCLIP	θCLAP
Speech	A photo of people talking.	This is a sound of speech	20	0
Car	A photo of a car.	This is a sound of car	15	0
Cheering	A photo of people cheering.	This is a sound of cheering	18	1
Dog	A photo of a dog.	This is a sound of dog	14	4
Cat	A photo of a cat.	This is a sound of cat	15	6
Frying_(food)	A photo of frying food.	This is a sound of frying (food)	18	-2
Basketball_bounce	A photo of people playing basketball.	This is a sound of basketball bounce	18	4
Fire_alarm	A photo of a fire alarm.	This is a sound of fire alarm	15	4
Chainsaw	A photo of a chainsaw.	This is a sound of chainsaw	15	2
Cello	A photo of a cello.	This is a sound of cello	15	2
Banjo	A photo of a banjo.	This is a sound of banjo	15	2
Singing	A photo of people singing.	This is a sound of singing	18	1
Chicken_rooster	A photo of a chicken or a rooster.	This is a sound of chicken, rooster	15	2
Violin_fiddle	A photo of a violin.	This is a sound of violin fiddle	15	3
Vacuum Cleaner	A photo of a vacuum cleaner.	This is a sound of vacuum cleaner	15	0
Baby laughter	A photo of a laughing baby.	This is a sound of baby laughter	15	2
Accordion	A photo of an accordion.	This is a sound of accordion	15	2
Lawn_mower	A photo of a lawnmower.	This is a sound of lawn mower	15	2
Motorcycle	A photo of a motorcycle.	This is a sound of motorcycle	15	0
Helicopter	A photo of a helicopter.	This is a sound of helicopter	16	2
Acoustic_guitar	A photo of a acoustic guiter.	This is a sound of acoustic guitar	14	-1
Telephone_bell_ringing	A photo of a ringing telephone.	This is a sound of telephone bell ringing	15	2
Baby_cry_infant_cry	A photo of a crying baby.	This is a sound of baby cry, infant cry	15	3
Blender	A photo of a blender.	This is a sound of blender	15	3
Clapping	A photo of hands clapping.	This is a sound of clapping	18	0
A Caption Construction and Threshold Determination in VALOR
We provide detailed explanations on how we devise input captions for each event to be used with CLIP and CLAP. For the CLIP's input captions, we add the prompt "A photo of" before each event name and modify some of the captions to make them sound reasonable, e.g. changing "A photo of speech" to "A photo of people talking." As for CLAP, we add the prompt "This is a sound of" before each event name. All input captions devised for CLAP and CLIP are included in Table 6 for reference.
Furthermore, the determination of class-dependent threshold values,  $\theta^{\mathrm{CLIP}}$  for CLIP and  $\theta^{\mathrm{CLAP}}$  for CLAP, is based on the visual and audio segment-level F-scores of the validation split, respectively. These scores are achieved by comparing the segment-level pseudo labels generated by the respective models against the ground truth labels.
B More AVVP Implementation Details
In our experiments, we apply two different model architectures: 1) the standard model architecture, which is employed in VALOR, consists of a single HAN layer with a hidden dimension of 512; 2) the variant model architecture, which is used in VALOR+ and VALOR++, is a thinner yet deeper HAN model, comprising four HAN layers with a hidden dimension of 256. Both models contain approximately the same number of trainable parameters. The above details are summarized in Table 7. The models are trained using the AdamW optimizer, configured with  $\beta_{1} = 0.5$ ,  $\beta_{2} = 0.999$ , and weight decay set to 0.001. We employ a learning rate scheduling approach that initiates with a linear warm-up phase over 10 epochs, rises to the peak learning rate, and then decays according to a cosine annealing schedule to the minimum learning rate. We set the batch size to 64 and train for 60 epochs in total. We clip the gradient norm at 1.0 during training. We attach the code containing our model and loss functions to the supplementary files.
More Details of Using Video Labels as Filters In this section, we provide more details regarding video label filtering. First, when video labels are not used as filters, we need to adjust the event thresholds again on the validation split. To save time for the adjustment, we transition from class-dependent thresholds to unified (class-independent) thresholds. This means that after the adjustment, the threshold for each event is the same. For the sake of fairness, we also switched to using unified thresholds when using video labels as filters. The experimental results, as shown in Table 8, indicate that simply changing the thresholds from class-dependent