Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser
Authors: Yung-Hsuan Lai
Yen-Chun Chen
Yu-Chiang Frank Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Audio-Visual Video Parsing
Audio-Visual Learning

Abstract: Audio-visual learning has been a major pillar of multi-modal machine learning, where the community mostly focused on its $\textit{modality-aligned}$ setting, $\textit{i.e.}$, the audio and visual modality are $\textit{both}$ assumed to signal the prediction target.
With the Look, Listen, and Parse dataset (LLP), we investigate the under-explored $\textit{unaligned}$ setting, where the goal is to recognize audio and visual events in a video with only weak labels observed.
Such weak video-level labels only tell what events happen without knowing the modality they are perceived (audio, visual, or both).
To enhance learning in this challenging setting, we incorporate large-scale contrastively pre-trained models as the modality teachers. A simple, effective, and generic method, termed $\textbf{V}$isual-$\textbf{A}$udio $\textbf{L}$abel Elab$\textbf{or}$ation (VALOR), is innovated to harvest modality labels for the training events.
Empirical studies show that the harvested labels significantly improve an attentional baseline by $\textbf{8.0}$ in average F-score (Type@AV).
Surprisingly, we found that modality-independent teachers outperform their modality-fused counterparts since they are noise-proof from the other potentially unaligned modality.
Moreover, our best model achieves the new state-of-the-art on all metrics of LLP by a substantial margin ($\textbf{+5.4}$ F-score for Type@AV). VALOR is further generalized to Audio-Visual Event Localization and achieves the new state-of-the-art as well.

Modality-Independent Teachers Meet Weakly-Supervised Audio Audio-Visual Event Parser
Yung-Hsuan Lai,' Yen-Chun Chen, Yu-Chiang Frank Wang
National Taiwan University "Microsoft 'NVIDIA
r109420970mtu, edu.. tw, chen. yenchun. twwgmail com, frankvangbavidia.com
Abstract
Audio-visual learning has been major pillar of multi-modal machine learning, where the community mostly focused on its modality-aligned setting, i.e., the audio and visual modality are both assumed signal the prediction target. With the Look, Listen, and Parse dataset (LLP), we investigate the under-explored unaligned setting, where the goal is to recognize audio and visual events in a video with only weak labels observed. Such weak video-level labels only tell what events happen without knowing the modality they are perceived (audio, visual, or both). To enhance learning in this challenging setting, we incorporate large-scale contrastively pre-trained models as the modality teachers. simple, effective, and generic method, termed Visual-Audio Label Elaboration (VALOOR), is innovated harvest modality labels for the training events. Empirical studies show that the harvested labels significantly improve an attentional baseline by 8.0 in average F-score (Type @ AV). Surprisingly, we found that outperform their modality-fusee other potentially unaligned modality. Moreover, our best model achieves the new state-of- the-art art on all metries of LLP by substantial margin (+5.4 F-score for Type@AV AV). VALOR is further generalized to Audio Vistol Event Localization and achieves the new offtt--f-hhe-tt as well.'
teachers counterparts since they are noise-proof from the
Introduction
Multi-modal learning has become a pivotal topic in modern machine learning research. Audio-visual learning undoubtedly one of the primary focuses, human frequently uses both hearing and vision to perceive the surrounding environment. Countless researchers have devoted to its modality-aligned setting with strong assumption that the audio and visual modality both contain learnable clues the desired prediction target. Numerous audio-visual tasks and algorithms have then been proposed, such as audio-visur speech recognition [1. 66, 68]. audio-visual action recognition [22, 53, 81], sound generation from visual data [18, 69, 84], audio-visual question answering [40, 89], and many more. However, almost all real-world events can be audible while invisible, and vice versa, depending on how they are perceived. For example, a mother doing dishes in the kitchen might hear baby crying from the living room, but be unable to directly see what is happening to the baby.
Having observed this potential modality mismatch in generic videos, Tian et al. [73] proposed the Audio-Visual Video Parsing (AVVP) task, which aims to recognize events in videos independently of the audio and visual modalities and also temporally localize these events. AVVP presents an unaligned setting of audio-visual learning since all 25 event types considered can be audio-only, visual-only, or audio-visual. Unfortunately, due to the laborious labeling process, Tian et al. [73] created this dataset (Look, Listen, and Parse: LLP) in a weakly- supervisee setting." More specifically,
"Code available https: //ttthub.. com//ranklin905/ /VALOR.
2 AVVP, LLP are used interchangeably in the literature. We use AVVP for the task, and LLP for the dataset.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Ground Truth Events
6og
Violi
speech
speech
HAN's Predictions
speech
speech
audio stvert
vincall averd
Figure 1: Modality-unaligeee samples from LLP. Note that recent AV approaches like HAN [73] are vulnerable to unaligned data modality and produce incorrect predictions.
only video-level event annotations are available at training. In other words, the modality (audio, visual, or both) and the timestamp which an event occurs are not given to the learning model.
The AVVP task poses significant challenges from three different perspectives. First, event is typically modality independent, i.e., knowing an event occurs one modality says nothing about the other modality. As illustrated in Fig. sleeping dog is seen but may not be heard; conversely, a violin being played could sometimes go out the camera view. Second, existing works heavily rely on the Multi-modal Multiple Instance Learning (MMIL) loss [73] to soft-select the modality (and timestamp), given only weak modality-less labels. This would be challenging for models to learn the correct event modality without observing large amount of data. The uni-modal guided loss via label smoothing is also used introduce uncertainty to the weak labels and thus regularize modality recognition. However, we hypothesize this improvement could be sub-optimal because explicit modality information is introduced. Finally, AVVP requires models to predict events for all I-second segments in given video. Learning from weak video-level labels without timestamps makes challenging for models to predict on a per-segment basis.
To address the above challenges in AVVP, propose to incorporate large-scale pre-trained open- vocabulary models, namely CLIP [56] and CLAP [79], to enhance learning with weak labels. Pre-trained pixels and waveforms (and contrastively pre-trained with natural language), these models are inherently isolated from potential spurious noise from the other modality. Another benefit the applicability of their prediction in open-vocabulary fashion. Therefore, to benefit from CLIP and CLAP, we aim to harvest explicit modality learning signals from them. Moreover, we aim to inference these models per video segment, yielding fine-gradd fine-grained temporal annotations.
While it might be tempting naively treat these pre-trained models as teachers and then applies knowledge distillation (KD) [28]. this could be sub-optimal as some events are difficult to distinguish from a single modality, even for humans. For example, cars, motorcycles, and lawn mowers all produce similar sounds. To better utilize CLIP and CLAP, introduce Visual-Audio Label Elaboration (VALOR), to harvest modality and timestamp labels in LLP. We prompt CLIP/CLAP with natural language description of all visual/audio event types for each video segment-by-segneen and then extract labels when a threshold is met. Additionally, implausible events are filtered out using the original weak labels accompanied with the video to mitigate the above indistinguishable problem. VALOR constructs fine- aaee-graindd temporal labels in both modalities so that models have access explicit training signals.
In addition to achieving the promising performance of AVVP, we observe that modality-indepeedeet teachers, CLIP and CLAP, generate more reliable labels than modality-fused one, a cross-modal transformer. We also showcase the generalization capability of VALOR via the Audio Visual Event Localization (AVE) task, in which our method also achieves the new state-of-the-art. Our contributions are summarized as follows:
A simple and effective AVVP framework, VALOR, proposed to harvest modality and temporal labels directly from video-level annotations, with an absolute improvement of +8.0 F-score.
We are the first to point out that modality independence could be crucial for audio-visual learning in the unaligned and -uuklysuupeeiseed setup.
Our VALOR achieves new state-of-ther oft-----ttttttttt- results with significant improvements on AVVP (+5.4 F-score), with generalization to AVE (+4.4 accuracy) jointly verified.

---

2 Preliminaries
Audio-Vo Audio-Visual Visual Video Parsing (AVVP) The AVVP [73] task is to recognize events of interest in a video in both visual and audio modalities and temporally identify the associated frames. For the benchmark dataset of Look, Listen, and Parse (LLP), a a ''seecond video is split into I non-overlapping segments. Each video segment is paired with set of multi-class event labels (yi. yt) ) E {0, 1 (y:: visual events, yf: audio events, C: number of event types). However, in the training split, the dense "segment-level" labels (y)" y(. are not available. Instead, only the global modality-less 'video-level' labels y == max; vy A yt Jt=1 are provided (A: element-wise 'logical and"). In other words, AVVP models need to be learned in a weukly-supervisee setting.
Baseline Model We now briefly review the model of Hybrid Attention Network (HAN) [73], which is a common baseline for AVVP. In HAN, ResNet-152 [26] and R(2+1)D [75] are employed extract 2D and 3D visual features. Subsequently, they are concatenated and projected into segment-level features F = (fi E R (d: hidden dimension). Segment-level audio features Fa = { E xd are extracted using VGGish [27] and projected to the same dimension. HAN takes these features and aggregates the intra-modal and cross-modal information through self-attention and cross-attention:
fr
=
=- f + Au(Se F", F") + Au(Se, F", FF) ff= F", FF) + Au(ff", F", F*),
(2)
where Att K, V) denotes multi-head attention [76]. Following Transformer's practice, the outputs are further fed through LayerNorms [6] and a 2-layer FFN to yield fr: With another linear layer, the hidden features are transformed into categorical logits 2,, 2 for visual and audio events, respectively. Finally, the segment-level audio and visual event categorical probabilities, P and PI (E [0, 1 are obtained by applying Sigmoid activation.
As key module in Tian al. [73], Multi-modal Multiple Instance Learning pooling (MMIL) is applied to address the above weakly-
learning task, which predicts the audio and visual event probabilities (pt m E {a, v). audio and visual modalities) as:
A {a, 1h }. softmax, (F""" (
p" = E, at p...
where trainable parameters n E dx are implemented as linear layers (o: element-wise product). For video-level event probability p:
B ((
Jemm softmax, (XW),
p P- PーΣ E,
о p."
where X = {('m) Jm E 22x7xd and W as trainable linear layer. Moreover, modality training targets are obtained via label smoothing (LS) [71]: ÿ"M = LSy).. Finally, the model is trained with binary cross entropy (BCE) as the loss function:
(5) In summary, by the attention mechanisms introduced HAN, MMIL pooling assigns event labels for each modality across time segments with only video-level event labels observed during training.
Crase Lvitee + Luidded + Ligiided Lyiden BCE y). Guided = BCE(P" ".").
3 Method
With only vidco-level event labels observed during training, we address three major challenges of AVVP: 1) modality independence of events' occurrence, 2) reliance on MMIL pooling for event label assignment under insufficient data, and 3) demand for dense temporal predictions. To address these challenges, we propose to leverage large-scal scale pre-trained contrastive models, CLIP and CLAP, to extract modality-aware, temporally dense training signals to guide model learning.
3.1 Zero-Shot Transfer of Contrastive Pre-trained Models
Radford et al. [56] proposed Contrastive Language-Image Pre-training (CLIP) to utilize web-scale image-text pairs to train a strong image encoder. As result, CLIP overthrows the limitation of predicting predefined categories. Due to its large training data size (400M), CLIP has demonstrated remarkable zero-shot performance
wide range of visual recognition tasks. All the above motivates us to incorporate CLIP to improve visual event recognition in AVVP.
In our work, CLIP's visual understanding of AVVP is extracted as the following. We extract T evenly spaced video frames and pass them into CLIP's image encoder to obtain the visual features 1=z1 e _ (dg: the dimension of CLIP's feature). For simplicity and readability, we will
(L T
3

---

Visual Label Elaboration (ef the i-th sagrrent)
Cross-moda Learning
CLIP
CLIP
logical
M/
collect over segmeets
M
VALOR
CLIP
CAr
thresholds
photoo cl
collo
Pi
pil
video level labels
banjo
fxed model
Luides
HAN
MMIL Pooling
y
Audio Label Elaboration segment)
VP VR
1/
CLAP
CLAP
logical
CLAP
A
collect over segments
This soundd [ collo
GAr
thresholds
//
99
VALOR
banjo
video-ievel
labele
leamiing model
Figure 2:
VALOR framework. With modality-independee label elaboration via CLIP and CLAP, the harvested temporally dense labels serve as additional modality- and time-aware aware cues.
omit the time subscript & for the remainder of this paper when there is ambiguity. Next, convert the AVVP event categories to concepts that CLIP understands. A caption for each event is constructed by adding "A photo of prefix to the event's natural language form. These captions are processed by the CLIP's text encoder, resulting in event features GCLIP = {gr
CLIP C E RCxd, where e indexes the events, and Je represents the text feature of the e-th event. Frame-level event logits CLIP E RC can be obtained by calculating the inner products:
(6) In light of the notable success of CLIP [56], several studies have sprouted research on learning representative audio embeddings and text embeddings through Contrastive Language-Audio Pre- training (CLAP) [13, 15, 25, 49, 79]. In the same way as images and text are encoded in CLIP, web-scale audios and text are pre-trained with a contrastive objective in CLAP. Symmetrically, we obtain CLAP's understanding of AVVP audios as the following. From the audio channel, the raw waveform is extracted and split into T segments with the same lengths and then fed into CLAP, yielding segment-level audio features (fCT.AP), }. Txxx (d3: the dimension of CLAP's feature). On the other hand, an audio event caption is constructed by adding the prefix "This is sound of" to each AVVP event's name. Processed by the CLAP text encoder, we obtain CLLAP = {g. Je E Cx
CLIP fCLLPG GCLPP"
CLAP)
Segment-level audio event logits 2 cLAP E TRC are obtained by the inner products:
CLAP
CLAP GCLAP"
(7)
We note that Eqn. (6) and (7) can be viewed as CLIP's and CLAP's understanding of the associate video frame in the event space of AVVP.
3.2
Harvesting Training Signals
Given the logits CLP and CLAP aim to convert them to useful training signals for the AVVP task. An intuitive idea is to teach our model via knowledge distillation (KD) [28]. To deploy KD in training, segment-level normalized probabilities
first computed: qº softmax.
TTh = softmax..22" where (m, P) E {(v, CLIP), (a, CLAP)] denotes data modality (audio/visual) and pre-trained model (CLIP/CLAP) pair. Next, KL-divergence for all segments is calculated: CKD Σ, KL(g ). Finally, KD training is done by optimizing the loss function:
(8) However, as we find out empirically (shown in Table 4), this not the optimal usage of CLIP and CLAP. We hypothesize that some events are hard to distinguish from single modality, e.g. cars, motorcycles, and lawn mowers produce the sound of an engine. Therefore, we design VALOR, utilizing video-level labels to filter out the impossible events, hence mitigating the confusion.
CKD = Cvidso + CRD + LID.
Visual-Audio Label Elaboration (VALOR) To better exploit CLIP and CLAP, we design simple yet effective method, VALOR, harvest dense labels in both modalities. In particular, first define class- desendentt thresholds 0º E I for each modality to obtain segment-level labels from logits. Next. the impossible events are excluded using the given video-level labels, done via logical AND. Formally. this process can be written as: D = [x] > 0²) } A with the overall loss function:
LVLLOR = Lviden + L + L
C = Σ, BCE(p," .
(9)

---

To summarize, we design simple yet effective method, VALOR, to utilize large-scale pre-trained contrastive models, CLIP and CLAP, to generate segment-level labels in both modalities. Due the nature of immunity spurious noise from the other modality, the contrastive pre-training methods, and the large pre-training dataset size, CLIP and CLAP are able to provide reliable labels visual and audio modality, respectively. In addition, they are able to provide temporally dense labels to explicitly guide the model in learning events in each segment.
Related Work
4.1
Audio- Video Parsing with Look, Listen, and Parse
For AVVP, research flourishes along two orthogonal directions: enhancing the model architecture and label refinement. Architectural improvements include cross- moos- co-oceurrence module [45], class- aware uni-modal features and cross-modal grouping [51], and Multi-Modal Pyramid attention [87]. On the other hand, label refinement shares similar spirit with ours. MA [77] corrupted the data by swapping the audio channel of two videos with disjoint video-level event sets. The model's likelihood of the corrupted data was then used to determine the modality label. More recently, JoMoLD [11] utilized a two-stage approach. First, an AV model was trained as usual. Next, another model was trained while denoising the weak labels with prior belief from the first model. Both MA and JoMoLD produced global modality labels without timestamps. Concurrent to ours, VPLAN [96] and LSLD [16] generate dense temporal visual annotations with CLIP; however, the audio labels remain absent. Our VALOR represents a unified framework to elaborate the weak labels, along modality and temporal dimension, via zero-shot transfer of pre-trained models. We further emphasize the importance of modality independence when synthesizing modality supervision.
4.2 More Audio- Visual Learning
Audio-Visual Event Localization (AVE) Tian al. [72] proposed AVE to recognize the audio- visual event in a video while localizing its temporal boundaries. Numerous studies have been conducted, including Lin et al. [46] with seq2seq models, Lin and Wang [44] using intra&inter frame Transformers, Wu et al. [78] via dual attention matching, audio-spatial channel-attentin by Xu et al. [82], positive sample propagation from Zhou et al. [95], and Xia and Zhao [80] employing background suppression. We generalize VALOR to AVE's weakly supervised setting.
Audio Audio-Visual Visual Assistance While significant advancements have been made in speech recognition, speech enhancement, and action recognition, noise bias residing in the uni-modal data is still problematic. An effective solution could involve integrating data from an additional modality. This research direction encompasses various areas including speech recognition [1, 30, 66, 68], speaker recognition [12, 14, 55, 61, 63, 67], action recognition [22, 35, 36, 53, 81], speech enhancement or separation [2, 3, 34, 39, 50, 60], and object sound separation [7, 20, 21, 59, 74, 83, 91, 92].
Audio- Visual Correspondence and Understanding Humans possess an impressive capacity to deduce occurrences in one sensory modality using information solely from another. This fascinating human ability to perceive across modalities has inspired researchers to delve into sound generation from visual data [18, 19, 37, 45, 54, 69, 84, 93], video generation from audio [38, 41, 43, 90, 94], and audio-visual retrieval [42, 70]. In the pursuit of understanding how humans process audio- visual events, numerous studies have been undertaken on audio-visual understanding tasks such as sound localization in videos [5, 31, 32, 52, 64], audio-visual navigation [8-10, 17, 48, 86, 88], and audio-visual question answering [4, 24, 29, 40, 62, 65, 89].
Experiments
5.1 Experimental Setup
The LLP dataset is composed of 11849 10-second Youtube video clips covering 25 event categories, such as human activities, musical instruments, vehicles, and animals. The dataset is divided into training, validation, and testing splits, containing 10, 000, 649, and 1200 clips, respectively. The official evaluation uses F-score to evaluate audio (A), visual (V), and
Dataset and Metrics
5

---

Table AVVP benchmark. Note that pseudo label denoising is not applied for VPLANI. VALOR+ is trained on a thinner yet deeper HAN of similar size. VALOR+ further uses CLIP and CLAP as feature extractors and significantly boosts all metrics. The best numbers are bold and the second best numbers are underlined.
Segment-level
Event-level AV
Methods
A
AV
Type
Event
A
Type
Event
AVE [72]
47.2
37.1
35.4
39.9
41.6
40.4
34.7
31.6
35.5
36.5
AVSDN [46]
47.8
52.0
37.1
45.7
50.8
34.1
46.3
26.5
35.6
37.7
HAN [73]
60.1
52.9
48.9
54.0
55.4
51.3
48.9
43.0
47.7
48.0
MM-Pyr [87]
60.9
54.4
50.0
55.1
57.6
52.7
51.8
44.4
49.9
50.5
MGN [51]
60.8
55.4
50.4
55.5
57.2
51.1
52.4
44.4
49.3
49.1
CVCMS [47]
59.2
59.9
53.4
57.5
58.1
51.3
53.5
46.2
51.0
49.7
DHHN [33]
61.3
58.3
52.9
57.5
58.1
54.0
55.1
47.3
51.5
51.5
MA [77]
60.3
60.0
55.1
58.9
57.9
53.6
56.4
49.0
53.0
50.6
JoMoLD [11]
61.3
63.8
57.2
60.8
59.9
53.9
59.9
49.6
54.5
52.5
VPLAN [96]
60.5
64.8
58.3
61.2
59.4
51.4
61.5
51.2
54.7
50.8
VALOR
61.8
65.9
58.4
62.0
61.5
55.4
62.6
52.2
56.7
54.2
VALOR+
62.8
66.7
60.0
63.2
62.3
57.1
63.9
54.4
58.5
55.9
VALOR++ 68.1
68.4
61.9
66.2
66.8
61.2
64.7
55.5
60.4
59.0
audio-visual (AV) events separately. Type@AV (Type) is the averaged F-scores A, V, and AV. Event AV (Event) measures the ability to detect events in both modalities by combining audio and visual event detection results. Different from segment-leve metrics, the event-level metrics treat consecutive positive segments whole, and mloU of 0.5 is applied to calculate F-scores.
Implementation Details Unless otherwise specified, VALOR uses HAN under fair setting w.r.t. previous works with same data pre-processing. For the visual feature extraction, video frames are sampled at 8 frames per second. Additionally, we conduct experiments using CLIP and CLAP as feature extractors. The pre-trained ViT-L CLIP and HTSSAT-ROBERTT-hssso CLAP are used to generate labels and extract features. Note that for all experiments with CLAP, we use the implementation from Wu et [79] pre-trained on LAION-Audio--300, We do not use the version pre-trained on AudioSet (a larger pre-training corpus) since it overlaps with the AVVP validation and testing videos.
5.2 Unified Label Elaboration for State-of-the-Art Audio-Visual Video Parsing
To demonstrate the effectiveness of VALOR, we evaluate our method on the AVVP benchmark. Existing works include: 1) wcakly-supervised audio-visual event localization methods AVE and AVSDN, 2) HAN and its network architecture advancements MM-Pyramid, MGN, CVCMS, and DHHN, and 3) different label refinement methods MA, JoMoLD, and VPLAN. We report the results on the test split of the LLP dataset Table 1.
We achieve the new state-of-the-art (SOTA) on all metrics consistently with a large margin. Our method VALOR significantly improves the baseline (HAN) by 8.0 in segment-level Type AV. Compared to previous published SOTA, JoMoLD, JOTA,, VALOR scores higher on all metrics, including the 5.4 F-score improvement for segment-level Type@ AV. under a fair setting. With light hyper- parameter tuning. VALOR+ further achieves significant 2.4 improvement on Type AV, with deeper yet thinner HAN while keeping a similar number of trainable parameters. Our improvement on the audio side w.r.t. the concurrent preprint VPLAN is more significant than the visual side, which may be attributed to our effective audio teacher CLAP and label elaboration along the modality axis. We empirically conclude that VALOR has successfully unified label refinement along both modality and temporal dimensions. feature extraction models with CLIP and CLAP, achieving another consistent boost, including 3.0 in segment Type @ AV. We will release the VALOR++ pre-trained checkpoint, features, and harvested labels to boost future AVVP research.
push the limits, we further proposed VALOR+ by replacing the

---

Table 2: Selection of modality-independent labeler. Note that utilizing cross-modal labeler HAN instead of CLIP and CLAP to generate segment-level labels hardly improves the baseline (HAN). On the other hand, modality-less segment-level labels deteriorates the performance. All results reported on the validation split of LLP.
Dense Labeler
Modality Label
Segment-level AV
Event-level AV
A
V
Type
Event
A
Type
Event
None
62.0
54.5
50.2
55.6
57.1
53.5
50.5
43.6
49.2
50.3
HAN
V
62.1
56.4
52.1
56.8
57.6
53.4
52.0
45.4
50.3
50.6
CLIP&CLAP
X
41.0
59.0
34.5
44.9
52.1
33.2
56.2
28.2
39.2
43.1
CLIP&CLAP
V
62.7
66.3
61.0
63.4
61.8
55.5
62.0
54.1
57.2
53.8
Table 3: Fidelity of the elaborated labels. We conduct a comparison between the segment-level labels generated from VALOR and those from naive approach where we assume video-level labels also serve segment-level labels. We directly evaluate these pseudo labels on the validation split
before training. The results clearly indicate that VALOR-generated labels are more accurate than the naive ones.
Label Generation Methods
Audio
Visual
Audio-Visual
Video Labels
80.08
67.21
59.45
VALOR
85.07 (+4.99)
82.14 (+14.93)
77.07 (+17.62)
5.3
Ablation Studies
The impressive results achieved in Table 1 are based on careful design. In this subsection, we elaborate on why choose CLIP and CLAP synthesize dense labels with modality annotations with empirical support. Furthermore, we break down the loss function and modeling components into orthogonal pieces and evaluate their individual effectiveness.
How to choose the labeler? In Table 2, we show the necessity of models (CLIP and CLAP) over the multi-modal model (HAN) the labeler (2nd row) and that modality- labels beat modality-agnostii and importance of using large-scale pre-trained uni-modal models annotate modality segment-level labels. To validate the former, we employ baseline model (HAN) that has been trained AVVP individually annotate segment-level labels within the two modalities. Experimental results show that modality-aware temporal dense labels generated by multi-modal model (HAN), learned from weak labels, are less effective than those generated by large-scale pre-trained uni-modal models (CLIP and CLAP), thereby underscoring the essentiality of using large-scale pre-trained uni-modal models. Subsequently, to validate the latter, we generate modality-annostic segment-level labels from CLIP and CLAP, meaning that these labels only reveal the events occurring in each segment but do not disclose the modality of the event. As seen from the third row of Table 2, while such a labeling method increases the F-score for visual events, it dramatically decreases the F-score for audio events. The overall performance (Type F-score) is even worse than that HAN (the first row), clearly indicating the importance of modality-aware labeling for the model to learn the AVVP task effectively.
pre-traue pre-trained
labels (3rd row). We aim to demonstrate the necessity
How accurate are the elaborated labels? To measure the fidelity of the pseudo labels generated via VALOR audio and visual modalities, we conduct a comparison between the segment-level labels generated from VALOR and those from a naive approach where we assume that vidco-level labels also serve segment-level labels, i.e., we assume that an event occurs in both modalities and all segments if it occurs in the video. We directly evaluate these pseudo labels on the validation split before using them for training. The results, presented in Table 3, clearly demonstrate the superiority of our generated segment-level audio and visual pseudo labels compared to the naive counterparts. Notably, our segment-level visual F-score exceeds the naive approach by nearly points while the audio-visual F-score significantly surpasses for more than 17 points. These results highlight the reliability of the VAL.OR- -egerrtge pseudo labels, which provide more faithful temporal and modal information to facilitate model training.
7

---

Table 4: Ablation study. "global" denotes only video-level labels observed, while "dense" indicates segment-level labels available as ground truth. "base" is the baseline method [73]. "New Feat." denotes the use of features from CLAP, CLIP, and R(2+1)D, and "Deep HAN" is that of the 256-dim 4-layer HAN model. All results are reported on the validation split of LLP.
Audio Loss
Visual Loss
New
Deep HAN
Segment-leve AV
global
dense
global
dense
Feat.
A
Type
Event
base
base
X
x
x
62.0
54.5
50.2
55.6
57.1
x
KD
X
KD
x
51.1
64.0
48.0
54.3
55.5
VALOR
VALOR
X
X
x
62.1
65.8
59.0
62.3
61.2
base
X
VALOR
x
60.5
66.7
60.8
62.7
59.8
x
VALOR
base
X
X
x
62.2
54.5
52.7
56.5
56.5
x
VALOR
X
VALOR
X
x
62.7
66.3
61.0
63.4
61.8
x
VALOR
X
VALOR
64.5
67.1
63.1
64.9
63.2
x
VALOR
X
VALOR
71.4
69.4
64.9
68.6
69.7
The Effect of Utilizing Video Labels as Filters
The Accuracy of Models Solving Modality Non-alignment Problem
70.0
wib farling
62.7
40
36.3
00.0
Acouraoyys
5.5000
$3.8
50.0
47.9
4.6
40.0
Audio
Visual
HAN
JoNALD
VALEA
WALO+++
Modelity
Methods
Figure 3: Ablation study of whether using
Figure 4: The extent to which the models ad- dress the modality non-alignment issue.
video-level labels as filters.
How to use the elaborated labels? We conduct ablation study on utilizing CLIP and CLAP together. The results are presented Table 4. The replacement of the smoothed video-level event labels j and f"" with their respective refined weak labels j and j derived from our method leads to significant increase the Type AV F-score, from 54.0 to 60.8. This finding underscores the importance of incorporating labels that are proximal to ground truth, albeit weak. Furthermore, we leverage the CLIP and CLAP models to generate segment-level labels for each modality. This approach results in improvement of 8.0 Type@ AV F-score explicitly informing the model of the events occurring in each segment of the audio-visual video facilitates the learning of the Audio- Visual Video Parsing (AVVP) task. In addition, CLIP and CLAP are also used to obtain more representative features. Replacing the ResNet- 152 and VGGish features with CLIP and CLAP features yields Type@ @ AV F-score improvement of 4.0.
the baseline, indicating that
Whether using video-level labels as filters? Video-level labels are pivotal for generating reliable pseudo labels in our method, where we employ them as filters to eliminate impossible events misclassified by CLIP or CLAP. In Figure 3, we conduct experiments underscore the necessity of using video-level labels as filters. Notably, without utilizing video-level labels filters, both audio and visual F-scores plummet, reaching 47.9 and 53.8, respectively.
How well can the modality non-aligned problem modality independence of events is one of the crucial challenges in the AVVP task, we assess the extent of the modality non-aligned problem in the LLP dataset and the extent to which the models solve the problem. First, define the word "segment-level event" the cumulative number events that occur without modality differences across all segments. other words, an audio event and visual event from the same category occur within segment, they are counted as a single "segment-level event." In the LLP dataset's validation split, there are 9126 segment-level events. Among these, 4048 segment-level events are modality non-aligned, i.e., they occur in exactly one modality. To measure how well trained models address the modality non-aligned issue, we conduct experiments involving several models, including our own, to predict both the modality and event of these segments. A successful prediction entails correctly identifying the event and confirming its presence in both modalities. The results, as displayed in Figure 4, reveal that HAN exhibits the poorest performance in predicting modality non-aligned events. Conversely, our methods VALOR and VALOR++ outperform the prior SOTA, JoMoLD. This highlights the effectiveness our approach in mitigating the modality non-alignment challenge within the AVVP task.
solved? As we have pointed out that the
of the

---

Visual Events: Baby Cry
Speech
HAN
JoMcLD
Ours
Audio Events:
Speech
GT
HAN
JoMeLL
JAMALD Ours
Figure 5: Qualitative Comparison with Previous AVVP Works. "GT" denotes the ground truth annotations. We compare with HAN [73] and JoMoLD [11].
5,4 Generalize VALOR to Audio Visual Event Localization
In this section, we showcase the additional generalization ability of VALOR by applying it to the Audio Visual Event Localiza- version
Table 5: Results on the AVE task.
tion (AVE) task. We consider the of AVE, where segment-level ground truths are not available to the model during training, meaning that no timestamp is pro- vided for the event, motivating us apply VALOR to harvest event labels. Without task-specific modification, we directly apply HAN and VALOR to AVE. The only difference that at inference, we combine the audio and visual prediction to obtain the audio-visual event required in this task. Please refer to the supplementary for more implementation details of the
Method
Accuracy(5))
VGG-ake, VGG-19 features
AVEL [72]
66.7
AVSDN [46]
67.3
CMAN [85]
70.4
AVRB [58]
68.9
AVIN [57]
69.4
AVT [44]
70.2
CMRAN [82]
72.9
AVE task.
PSP [95]
73.5
CMBS [80]
74.2
Quantitative Results From Table 5, observe that baseline method performs on par with the previous state-of- the-art method CMBS [80]. When our method is applied to the model, the accuracy leaps from 75.3 to 80.4, indicating the generalizability of method. In addition, surpass CMBS [80] and have become the new state-of-the-arr on the weakly-supervised AVE task with an improvement of 4.4 in accuracy.
VOG-lite, Res-151 festunes
AVEL [72]
71.6
AVSDN [46]
74.2
CMRAN [82]
75.3
CMBS [80]
76.0
CLAP CLIP. R(2-1JD featunes
HAN
75.3
VALOR
80.4
5.5 Additional Analyses
Qualitative Comparison Aside from quantitative comparison with previous AVVP works, we perform a qualitative evaluation as well. We qualitatively compare with the baseline method HAN [73] and the ofate-of-the-ar method JoMoLD I]. From Figure 5, it can be seen that only our model can correctly predict the "Baby Cry" visual event. HAN not only fails predict "Baby Cry" correctly but also mistakenly identifies the woman in the vidco as speaking. In the audio modality, all models correctly predict the presence of "Baby Cry" in the sound, but they also simultaneously misinterprets that someone is talking. Among all models, our model makes the least severe misjudgments.
Class-wise F-score Comparison. We further evaluate the effectiveness of providing accurate uni-modal segment-level pseudo labels for the model training. We visualize class-wise improvements between our generated segment-level labels for each modality and the naive segment-level labels derived from the video-level labels. In Figure we observe that when our audio pseudo labels are used, most of the audio events improve. In Figure 7, when our visual pseudo labels are used, nearly every event's F-score increases. These results indicate the effectiveness of our method in guiding the model to learn events in each modality. For the inferior performance on the "Speech" event, since CLIP is inept at extracting fine-grained visual information, not expected to recognize the "Speech" event well, which requires close attention mouth movements.

---

Classwise F-score Comparison
F-score
Blay
oy Chearin
Consen
Arocsc
6a3
Helcopler
Brring
Coltr
Ascontion
Boot Boorccaa
Frect
Hotoryyda
Chomses
alamm
Eskay
Laee
Speach
Ninsher
Cheroe
enging
Event Categories
Figure 6: Class-wise improvement audio events. Using the derived audio segment-level pseudo label is advantageous over the baseline using video-level labels as if they were audio segment-level.
Classwise Visual F-score Comparison
F-cco19
Strech
Car
Paskented
mnottriin Proog
adeat ideet
Mexte
helccple
stem
Drll
Chapang
Melt
(Manse
pittal
Crannaa
Cale
enterr
Stago
Actrdder
Gia
Raiey
Categaries
Figure 7: Class-wise improvement on visual events. VALOR'S visual segment-level labels clearly outperforms the video-level labels. Note that CLIP applicable extract global but not fine- ane-grained information from visual inputs. Thus, it is not expected to produce proper visual cues for the "Speech" event, which requires close attention mouth movements.
Conclusion
We propose Visual-Audio Label Elaboration (VALOR) for weakly-supervised Audio-Visual Video Parsing. By harnessing large-scale pre-trained contrastive models CLIP and CLAP, we generate fine- mme-ggriied temporal labels audio and visual modalities, providing explicit supervision to guide the learning of AVVP models. We show that utilizing and CLAP) and generating modality-aware labels are essential for AVVP. VALOR outperforms all the previous works when comparing in fair setting, demonstrating its effectiveness. In addition, we demonstrate the generalizability of our method in the Audio- Visual Event Localization task, where we improve the baseline greatly and achieve state-of-the-art result.
pre-trained models (CLIP
Limitations While VALOR performs well on AVVP, is uncertain whether it will maintain this efficacy when the number of events to classify expands. Moreover, because CLIP is far from perfect capturing fine-grained visual details, may fail to generate precise labels when the subject of the event is small or when the video quality is poor, potentially confounding the model.
Broader Impacts As an event recognition model, VALOR could be applied to future intelligent surveillance systems. While may reduce physical crime concerns, it could on the other hand infringe people's privacy and rights. Since the input consists videos of people, data privacy issues are inevitable, and it is essential to prioritize data protection against unauthorized access.
10

---

Acknowledgments and Disclosure of Funding
We thank National Center for High-performance Computing (NCHC) for providing computational and storage resources. We appreciate the NTU VLL members: Chi-Pin Huang, Kai-Po Chang, Chia- Chia-Hsiang Kao, and Yu- Hsuan Chen, for helpful discussions.
References
[1] Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE TPAMI, 44(12):8717-872 2018. 1,5
[2] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. The conversation: Deep audio-visual speech enhancement, In INTERSPEECH, 2018. 5
[3] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman, My lips concealed: Audio-visual speech enhancement through obstructions. INTERSPEECH, 2019. 5
[4] Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim Marks, Chiori Hori, Peter Anderson, et al. Audio visual scene-aware dialog. CVPR, 2019.
[5] Relja Arandjelovic and Andrew Zisserman. Objects that sound. FCCVV, 2018. 5
[6] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. arXiv:1607.064500 2016. 3
Layer normalization. arXiv preprint
[7] Moitreya Chatterjee, Jonathan Le Roux, Narendra Ahuja, and Anoop Cherian. Visual scene graphs for audio source separation. In ICCV, 2021.
[8] Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicene Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visur navigation in 3d environments. In ECCV, 2020. 5
Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In CVPR, 2021.
[10] Changan Chen, Sagnik Majumder, Ziad Al-Halah, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning to set waypoints for audio-visual navigation. ICLR, 2021.
[11] Haoyue Cheng, Zhaoyang Liu, Hang Zhou Chen Qian, Wayne Wu, and Limin Wang. Joint-modal label denoising for weakly-superviae audio-visual video parsing. In ECCV, 2022. 5, 9, 17, 18
[12] Joon Son Chung, Bong-Jiu Lee, and leksang Han. Who said that?: Audio-visual speaker diarisation of real-world meetings. In INTERSPEECH, 2019.
[13] Soham Deshmukh, Benjamin Elizalde, and Huaming Wang. Audio retrieval with wavtext5k and clap training. In INTERSPEECH, 2023. 4
[14] Yifan Ding, Yong Xu, Shi-Xiong Zhang, Yahuan Cong, and Liqiang Wang. Self-supervised learning for audio-visual speaker diarization. In ICASSP, 2020. 5
[15] Benjamin Elizalde. Soham Deshmukh. Mahmoud Al Ismail, and Huaming Wang. Clap: Learning audio concepts from natural language supervision. ICASSP, 2023.
[16] Yingying Fan, Wu, Yutian Lin, and Du. Revisit supervisd audio-visual video parsing from the language perspective. arXiv preprint arXiv:2306.0055 2023.
[17] Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua Tenenhaum. Look, listen, and act: Towards audio-visual embodied navigation. In ICRA.
[18] Chuang Gan, Deng Huang, Peihao Chen, Joshua Tenenbaum, and Antonio Torralba. Foley music: Learning to generate music from videos. In ECCV, 2020. 1,5
[19] Ruohan Gao and Kristen Grauman. 2.5 d visual sound. In CVPR, 2019. 5
[20] Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. In ICCV, 2019. 5
[21] Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In ECCV, 2018. 5

---

[22] Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. In CVPR, 2020. 1,5
[23] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-lahcled dataset for audio events. In ICASSP, 2017. 19
[24] Shijie Geng, Peng Gao, Moitreya Chatterjee, Chion Hori, Jonathan Le Roux, Yongfeng Zhang, Hongsheng Li, and Anoop Cherian. Dynamic graph representation learning for video dialog via multi-modal shuffled transformers. In AAAI, 2021. 5
[25] Andrey Guzhov, Federico Rauc, Jör Hees, and Andreas Dengel. Audioclip: Extending clip to image, text
and audio. In ICASSP, 2022. 4
[26] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 3
[27] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Can architectures for large-scale audio classification. In ICASSP, 2017.
[28] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 103300333, 2015. 2,4
[29] Chiori Hori, Anoop Cherian, Tim K Marks, and Takaaki Hori. Joint student-teacher learning for audio- visual scene-aware dialog. In INTERSPEECH, 2019. 5
[30] Di Hu, Xuelong Li, et al. Temporal multimodal learning in audiovisual speech recognition. In CVPR, 2016. 5
[31] Di Hu, Yake Wei, Rui Qian, Weiyao Lin, Ruihua Song, and Ji-Rong Wen. Class-aware sounding objects localization via audiovisual correspondence. IEEE TPAMI, 44(12):9844-9859, 2021.
[32] Xixi Hu, Ziyang Chen, and Andrew Owens. Mix and localize: Localizing sound sources in mixtures. In CVPR, 2022.
[33] Xun Jiang. Xing Xu, Zhiguo Chen, Jingran Zhang, Jingkuan Song, Fumin Shen, Huimin Lu, and Heng Tao Shen. Dhhn: Dual hierarchical hybrid network for reakly-supervisee audio-visual video parsing. ACM MM, 2022.
[34] Zhiqi Kang, Mostafa Sadeghi, Radu Horaud, Xavier Alameda- Pineda, Jacob Donley, and Anurag Kumar. The impact of removing head movements on audio-visual speech enhancement. In ICASSP, 2022. 5
[35] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In ICCV, 2019.
[36] Bruno Korhar, Du Tran, and Lorenzo Torresani. Sesampler: Sampling salient clips from video for efficient action recognition. In ICCV, 2019. 5
[37] Vinod K Kurmi, Vipul Bajaj, Badri Patro, KS Venkatesh, Vinay P Namboodiri, and Precthi Jyothi. Collaborative learning to generate audio-video jointly. In /CASSP, 2021. 5
[38] Hsin- Ying Lee, Xiaodong Yang. Ming- Yu Liu, King-Chun Wang, Yu-Ding Lu, Ming- Ming-Hsuan Yang, and Jan Kautz, Dancing to music. In NeurlPS, 2019. 5
[39] Jiyoung Lee, Whan Chung, Sunok Kim, Hong-Goo Kang, and Kwanghoon Sohn, Looking into your speech: Learning cross- modsl affinity for audio-visual speech separation. In CVPR, 2021. 5
[40] Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In CVPR, 2022. 1,5
[41] Ruilong Li, Shan Yang. David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In ICCV, 2021. 5
[42] Xuelong Li, Di Hu, and Xiaoqiang Lu. Image2song: Song retrieval via bridging image content and lyric words. In ICCV, 2017. 5
[43] Yuanzhi Liang, Qianyu Feng, Linchao Zhu, Li Hu, Pan Pan, and Yi Yang. Seeg: Semantic energized co-speech gesture generation. In CVPR, 2022. 5
12

---

[44] Yan-Bo Lin and Yu-Chiang Frank Wang. Audiovisual transformer with instance attention for audio-visual event localization. In ACCV, 2020. 5.9
[45] Yan-Bo Bo Lin and Yu-Chiang Frank Wang. Exploiting audio-visual consistency with partial supervision for spatial audio generation. In AAAI, 2021.
[46] Yan- Bo Lin, Yu-Jhe Li, and Yu- Chiang Frank Wang. Dual-modality seq2seq network for audio-visual event localization. In ICASSP, 2019. 5, 6,
[47] Yan-Bo Lin, Hung- Yu Tseng, Hsin-Ying Lee, Yen-Yu Lin, and Ming-Hsuan Yang. Exploring cross-vidco and mross- signals for weakly-supervised audio-visual video parsing, In NeurIPS, 2021. 6
[48] Sagnik Majumder, Ziad Al-Halah, and Kristen Grauman. MoveZhear: Active audio visual source separation. In ICCV, 2021. 5
[49] Xinhao Mci, Xubo Liu, Jianyuan Sun, Mark D Plumbley, and Wenwu Wang. On metric learning for audio-text cross-modal retrieval. In INTERSPEECH, 2022. 4
[50] Daniel Michelsanti, Zheng-Hun Hua Tan, Sigurdur Sigurdsson, and Jesper Jensen, On training targets and objective functions for deep-lcarning-basee audio-visual speech enhancement. In ICASSP, 2019. 5
[51] Shentong Mo and Yapeng Tian. Multi-modal grouping network for weakly-supervised audio-visual video parsing. In NeurlPS, 2022. 5,6
[52] Andrew Owens and Alexei A Efros, Audio-visual scene analysis with self-supervised multisensory features. In ECCV, 2018. 5
[53] Rameswar Panda, Chun-Fu Richard Chen, Quanfu Fan, Ximeng Sun, Kate Sacnko, Aude Oliva, and Rogerio Feris. Adamml: Adaptive multi-modal learning for efficient video recognition. In ICCV, 2021. 1,
[54] Kranti Kumar Parida, Siddharth Srivastava, and Gaurav Sharma. Beyond mono to binaural: Generating binaural audio from mono audio with depth and cross modal attention. In WACV, 2022. 5
[55] Yanmin Qian, Zhengyang Chen, and Shuai Wang. Audio-visune deep neural network for robust person verification. JEEF/ACM Transactions on Audio, Speech, and Language Processing, 29:1079-100 1092, 2021. 5
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In /CML, 2021. 2, 3,
[57] Janani Ramaswamy. What makes the sound?: dual-modality interacting network for audio-visual event localization. In ICASSP, 2020. 9
[S8] Janani Ramaswamy and Sukhendu Das. See the sound, hear the pixels. WACV, 2020. 9
[59] Andrew Rouditchenko, Hang Zhao. Chuang Gan, Josh McDermott, and Antonio Toralla. Self-supervised audio-visual co-segmentation. In ICASSP, 2019. 5
[60] Mostafa Sadeghi and Xavier Alameda-Pineda. Robust unsupervised audio-visual speech enhancement using mixture of variational autoencoders. In ICASSP, 2020. 5
[61] Leda San,, Kritika Singh, Jiatong Zhou, Lorenzo Torresani, Nayan Singhal, and Yatharth Saraf. A multi-view approach audio-visual speaker verification. In ICASSP, 2021. 5
[62] Idan Schwartz, Alexander G Schwing. and Tamir Hazan. A simple baseline for audio-visual scene-aware dialog. In CVPR, 2019. 5
[63] Gregory Sell, Kevin Dub, David Snyder, Dave Etter, and Daniel Garcia-Romerg Audio-visso visual person recognition in multimedia data from the iarpa janus program. In ICASSP, 2018. 5
[64] Arda Senocak, Tac- Oh, Junsik Kim, Ming- Hsuan Yang. and In So Kweon. Learning to localize sound sources in visual scenes: Analysis and applications. IEEE TPAMI, 1365:1160- 2019. 5
[65] Ankit Shah, Shijie Geng. Peng Gao, Anoop Cherian, Takaaki Hori, Tim K Marks, Jonathan Le Roux, and Chiori Hori. Audio-visual scene-aware dialog and reasoning using audio-visual transformers with joint student-teacher learning. In ICASSP. 2022. 5
[66] Bowen Shi, Wei-Ning Hsu, and AbdeIrahman Mohamed. Robust self self-supervised audio visual speech recognition. In INTERSPEECH. 2022. 1,5
13

---

[67] Suwon Shon, Tae-Hyun Oh, and James Glass. Noise-tolerant audio-visual online person verification using attention-basee neural network fusion. In ICASSP, 2019. 5
[68] Qiya Song, Bin Sun, and Shutao Li. Multimodal sparse transformer network for audio-visual speech recognition. IEEE Transactions on Neural Networks and Learning Systems, 2022. 1,5
[69] Kun Su, Xiulong Liu, and Eli Shlizerman. Audco: Audio generation for silent performance video. In NeurIPS, 2020. 1,5
[70] Didac Surís, Amanda Duarte, Amaia Salvador, Jordi Torres, and Xavier Giró-i Nicto. embeddings for video and audio retrieval. In ECCV workshops, 2018. 5
Cross-modal
[71] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
[72] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In ECCV, 2018. 5, 6, 9, 19
[73] Yapeng Tian, Dingzeyu Li, and Chenliang Xu, Unified multisensory perception: Weakly-supervised audio-visual vidco parsing. In ECCV, 2020. 2, 3, 6, 8, 9, 17, 18
[74] Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. In CVPR, 2021. 5
[75] Du Tran, Heng Wang. Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look spatiotemporal convolutions for action recognition. In CVPR, 2018, 3
[76] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurlPS, 2017. 3
[77] Yu Wu and Yi Yang. Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In CVPR, 2021. 5,
[78] Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang. Dual attention matching for audio-visual event localization. In ICCV, 2019.
[79] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Kirkkaarta. and Shlomo Dubnov. Large- scale contrastive language-audin pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP, 2023. 2, 4,
[80] Yan Xia and Zhou Zhao. Cross-modal background suppression for audio-visual event localization. In CVPR, 2022. 5,9
[81] Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.0877 2020. 1,5
[82] Haoming Xu, Runhao Zeng. Qingyao Wu, Mingkui Tan, and Chuang Gan. Cross-modal relation-aware networks for audio-visual event localization. In ACM MM, 2020. 5,9
[83] Xudong Xu, Bo Dai, and Dahua Lin. Recursive visual sound separation using minus-plus net. In ICCV, 2019. 5
[84] Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural audio generation without binaural audios. In CVPR, 2021. 1,5
[85] Hanyu Xuan, Zhenyu Zhang, Shuo Chen. Jian Yang, and Yan Yan. Cross-modal attention network for temporal inconsistent audio-visual event localization. In AAAI, 2020. 9
[86] AbdeIrahman Younes, Daniel Honerkamp. Tim Welschehold, and Abhinav Valada. Catch me if you hear me: Audio-visual navigation in complex unmapped environments with moving sounds. IEFF Robotics and Automation Letters, 2023. 5
[87] Jiashuo Yu, Ying Cheng, Rui-Wei Zhao, Rui Feng, and Yuejie Zhang. Mm-pyramid: Multimodal pyramid attentional network for audio-visual event localization and video parsing. In ACM MM, 2022. 5,6 6
[88] Yinseng Yu. Wenbing Huang. Fuchun Sun, Changan Chen, Yikai Wang, and Xiaohong Liu. Sound adversarial audio-visual navigation. In ICLR, 2022. 5
[89] Heeseung Yun, Youngjae Yu. Wonsuk Yang. Kangil Lee, and Gunhee Kim. Pano-avqa: Grounded audio-visual question answering 360deg videos. In ICCV, 2021. 1,5
14

---

[90] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In JCCV, 2019.
[91] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In ECCV, 2018.
[92] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In ICCV, 2019. 5
[93] Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-sterco: Visually guided stereophonic audio generation by associating source separation. In ECCV, 2020. 5
[94] Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Lay, Xiaogang Wang, and Ziwei Liu. Pose-controllable talking face generation by implicitly modularized audio-visual representation. In CVPR, 2021. 5
[95] Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijic Hao, and Meng Wang. Positive sample propagation along the audio-visual event line. In CVPR, 2021. 5, 9
[96] Jinxing Zhou, Dan Guo, Yiran Zhong, and Meng Wang. Improving audio-visual video parsing with pseudo visual labels. arXiv preprint ar
2023. 5, 6, 17, 18
15

---

Table 6: The List of Input Captions and Thresholds for CLIP and CLAP. We add the prompt "A photo of before each event name to make CLIP's input captions and the prompt "This is a sound of" to make CLAP's input captions.
Input Captions
thresholds 0
Events
CLIP
CLAP
GCLAP
Speech
A photo of people talking.
This is sound of speech
20
0
Car
A photo of a car.
This is a sound of car
15
0
Cheering
A photo people cheering.
This sound of cheering
Dog
A photo of a dog.
This is sound of dog
Cal
A photo of cat.
This is sound of cat
15
6
Ffyong((food)
A photo of frying food.
This sound of frying (food)
18
-2
Baskethall bouunce
A photo of people playing baskethall.
This is a sound baskethall bounce
_ae__alarm
A photo of a fire alarm.
This is a sound of fire alarm
15
Chainsaw
A photo of a chuinsaw.
This is sound of chainsaw
15
2
Cello
photo of cello.
This sound of cello
Banjo
A photo of banjo.
This is sound of bunjo
15
Singing
A photo of people singing.
This is a sound of singing
18
Chicken rooster
A photo of a chicken or rooster.
This is a sound chicken, rooster
Violin fiddle
A photo of a violin.
This sound violin fiddle
Vacuum_cleese cleaner
A photo of vaceum cleaner.
This is sound of vacuum cleuner
15
laughte
A photo of a luughing baby.
This is a sound of baby laughter
15
2
photo of an accordion.
This is u sound accordion
I Lawn_ mower
A photo of lawnmower.
This sound of lawn mower
Motorcycle
A photo of a motorcycle.
This is a sound of motorcycle
15
0
Helicopter
A photo of a helicopter.
This is a sound helicopter
2
Acoustic guitar
acoustic guiter. A photo of a rrigging telephone.
A photo
This sound of acoustic guitar
Telephone_ bell bell_ringing
This is a sound of telephone bell ringing
15
Bahy_ery _erll__r_______eeeer jijjant__rrrry _er
A photo of a crying haby.
This is a sound of haby cry, infant cry
15
Blender
photo of a blender.
This is a sound blender This sound of clapping
Clapping
photo of hands clapping.
18
A Caption Construction and Threshold Determination in VALOR
We provide detailed explanations on how we devise input captions for each event to used with CLIP and CLAP, For the CLIP's input captions, we add the prompt "A photo of"" before each event name and modify some of the captions to make them sound reasonable, e.g. changing "A photo of speech" to "A photo of people talking." " As for CLAP, we add the prompt "This is sound of"" before each event name. All input captions devised for CLAPP and CLIP included in Table 6 for reference.
Furthermore, the determination of class-dependent threshold values, CLLP for CLIP and (LA for CLAP,, based on the visual and audio segment-level F-scores the validation split, respectively. These scores are achieved by comparing the segment-level pseudo labels generated by the respective models against the ground truth labels.
B More AVVP Implementation Details
In our experiments, we apply two different model architectures: 1) the standard model architecture, which is employed in VALOR, consists of single HAN layer with a hidden dimension of 512: 2) the variant model architecture, which is used in VALOR+ and VALOR++, is a thinner yet deeper HAN model, comprising four HAN layers with hidden dimension of 256. Both models contain approximately the same number of trainable parametens. The above details are summarized in Table 7. The models are trained using the AdamW optimizer, configured with B 0.5. 822 = 0.999, and weight decay set to 0.001. We employ learning rate scheduling approach that initiates with linear warm-up phase over 10 epochs, rises to the peak learning rate, and then decays according to a cosine annealing schedule to the minimum learning rate. We set the hatch size to 64 and train for 60 epochs in total. We clip the gradient norm at 1.0 during training. We attach the code containing our model and loss functions to the supplementary files.
C Additional Analysis
More Details of Using Video Labels as Filters In this section, we provide more details regarding video label filtering. First, when video labels not used as filters, we need adjust the event thresholds again on the validation split. save time for the adjustment, we transition from class dass-dependent thresholds to unified itlass-independent thresholds. This means that after the adjustment, the threshold for each event is the same. For the sake of faimess, we also switched to using unified thresholds when using video labels as filters. The experimental results, as shown in Table 8, indicate that simply changing the thresholds from class dependeen
16

---

Table 7: Two Different HAN Model Architectures. The "standard" model architecture is used in VALOR. The "variant" model architecture is used in VALOR+ and VALOR++.
HAN model
standard
variant
Model Arch. Hyper-parameters
hidden dim
512
256
hidden layers
I
4
trainable params
5.1M
5.05M
Training Hyper-pann parameters
peak learning rate
le-4
3e-4
min learning rate
1c-6
3c-6
Table 8: Ablation study of whether using video-level labels as filters. Left: the ablation study of using video-level labels in audio label elaboration. Right: the ablation study of using video-level labels in visual label elaboration. To save the time required for tuning event thresholds, we have transformed class-dependent event thresholds into unified event thresholds, which means that the thresholds for each event are the same.
labels
Event Taresholds
Segment-leve
labels as fitters
Segment-level
as filier
A
AV
Typo
Event
Thresholds
A
AV
Type
Event
unified unified
17.9
61.5
49.2
53.9
53.8
unified
62.8
53,8
50.9
55.9
58.6
60.9
class-depent depeneennd
62.7
66.3
61.0
63.4
61.8
class des-ndepndeet
62.7
66.3
61.0
63.4
61.8
unified does not significantly degrade the model's performance, whether the audio or visual modality. However, if vidco label filtering is not applied, the resulting audio and visual pscudo labels become highly inaccurate, leading to a model with an audio F-score of only 17.9 and visual F-score of only 53.8.
D VALOR with Pseudo Label Denoising
In this section, we explore the application of Pscudo Label Denoising (PLD), as proposed VPLAN [96], to refine the segment-leval labels generated by our method. The hyper-parameters for the PLD, specifically K = 4 and 6 for the visual modality. and K 10 and 10 for the audio modality, are chosen based on the visual and audio F-scores on the validation split. From Table 9, can that PLD is less effective in refining our pseudo labels compared to VPLAN's pseudo labels 1+1.5 v.s. +-222 in segment- metrics and + +2228 v.s. +3.41 wenttlleve metrics). However, it's worth noting the visual segment- labels derived from our method before PLD are nearly as accurate as those from VPLAN after PLD (72.34 v.s. 72.51). Although we do implement PLD in the audio modality, noticeable improvement is recorded for any audio pseudo labels. Referring to Table 10, model trained with our denoised segment- labels improves marginally. Nevertheless, we outperform VPLAN on Type@ AV and Event AV F-scores in segment-level and event-level metries.
E Qualitative Comparison with Previous AVVP Works
Aside from quantitative comparison with previous AVVP works, we perform qualitative evaluation as well. In Figure 8, we qualitatively compare with the baseline method HAN [73] and the state-of-the-art method JoMol.D [1]]. In the top video example, JoMoLD erroneously predicts the "Speech" audio event, while all other methods accurately identify the audio events. In the bottom example, HAN produces identical temporal annotations the "Speech" event in both modalities, despite event only occurring audibly. Additionally, our method provides annotations that more closely align with the ground truth than either HAN or JoMoLD does when the events occur intermittentl, which is challenging for models to generate accurate predictions.
F More Visual Event Localization Details
Baseline Method We adopt the baseline model HAN to aggregate uni-modal and cross-modal temporal information as we have done in the AVVP task. For brevity, we introduce our baseline method from the procedure after feature aggregation. The segment-level audio features and visual features, fo and for (E Rd), output from HAN are processed through a 2-layer borw-rr network (FFN) to yield the uni-modal segment-level predictions (logits), zद and 2 (E R(C+))) respectively:
FFN(f.") (a
(10)
17

---

Table 9: PLD refinement. We evaluate the fidelity (F-score) of the segment-level pseudo labels before and after pseudo label denoising (PLD). PLD is less effective in refining our pseudo labels compared to VPLAN's pseudo labels. However, the visual segment- labels generated from method before PLD are nearly as accurate as those generated from VPLAN after PLD (72.34 v.s. 72.51). Results reported the validation split.
Audio
Visual
Methods
PLD
Seg
Event
Seg
Event
VALOR VALOR
80.78
71.69
72.34
66,36
80.78
71.69
73.84 (+1.5)
68.64 (+2.28)
VPLAN [96]
70.29
64.68
:
VPLAN [96]
72.51 (+2.22)
68.09 (+3.41)
Table 10: Results of Training with Denoised Labels. We outperform VPLAN Type AV and Event@AV F-scores segment-level and event-level metrics with and without PLD. Results are reported the testing split.
Segment-level Type
Event-level Тype
Methods
PLD
Event
Event
VALOR
62.0
61.5
$4.2
VALOR
62.2
61.9
56.6
53.7
VPLAN [96]
61.2
59.4
54.7
50.8
VPLAN [96]
62.0
60.1
55.6
51.3
Visuel Eventa: Car
J&MLD
Ours
GT
HAN
JoMoLD
Visual
GT
HAN
JoMoLD
Audio
Speech
GT
Ours
GT
HAN
JoMoLD
Figure 8: Qualitative Comparison with Previous AVVP Works. In general, the predictions
generated by our method VALOR are more accurate than those produced by the other methods. "GT"" denotes the ground truth annotations. We compare with HAN [73] and JoMoLD [1]].
where + 1 denotes the number of event classes and the "background" event. Since segment-level labels are not available in the weakly-supervised setting. simply infer video-level logits z E g by averaging all logits over time dimension i and modality dimension Finally, the binary cross-entropy loss applied to train
18

---

the model:
video
BCE((igmoid 2)),))
1 2T
ΣΣ m
(I)
Harvesting Training Signals The main idea of our method is to leverage large-scale open-vocahulary pre- pre-trained models to provide specific segment-level pseudo labels. We elaborate how these pseudo labels are generated. Initially, segment-level audio logits and visual logits, 21 CLAP and 2 2 (E R2), are generated a manner identical to the AVVP task. Then, we use two sets of class-dependent thresholds, o CLAP and o CLP (E ), to construct the uni-modal segment-level labels of and Dă (E RR), respectively:
(12) In addition, we append an additional event "background" to the end of the segment labels y:" to expand the dimension to RC+1 If otTh consists solely of zeros, we assign the last dimension ("hackground") a value of assign it a value of zero. In other words, if an event could possibly occur in video and the
y = y {z!" > ""), (m, P) E {1v, CLIP), (a, CLAP))
one; otherwise, pre-trained model has certain confidence that the event is present in specifie video segment, that segment will segment will be labeled as "background", Having prepared
be labeled as containing the event; otherwise, the segment-level pseudo labels di and gf. we compute binary cross-entropy loss in individual modality and combine them to optimize the whole model instead of using the video-level loss C
VALOR BCF(Sigmoid(z 0.) BCE(Sigmoid( ".")"
(13)
The Audio- Event (AVE) Dataser [72] is composed of 4143 10- second video clips from AudioSet [23] that cover 28 real-world event categories, such human activities, musical instruments, vehicles, and animals. Each clip contains an event and uniformly split into len segments. Each segment is annotated with an event category if the event can be detected through both visual and auditory cues; otherwise, the segment is labeled as background. The AVE task is divided into supervised setting and weakly- -upervssd setting. In the former, we obtain ground truth labels for each segment during training; in the latter, similar to the AVVP task setting, we can only obtain video-Icvel labels, As with the AVVP task, we address the AVE task under the weakly-supervise, setting. We follow [72] to split the AVE dataset into training, validation, and testing split and report the results on the testing split. Following the previous work [72], we use the accuracy of segment-level event category predictions as the evaluation metric.
Dataset & Evaluation Metries
The pre-trained VIT-L CLIP and R(2+1)D are used to extract 2D and 3D visual features, respectively, which then concatenated to represent low-level visual features. The pre-trained ROSSAT-RBBERTafusiin CLAP is used to extract audio features. We adopt the standard HAN model (1 (1-layer and 512-dim) in this task and train the model with Adam optimizer, configured with 31 = 0.5, 31 = 0.999, and weight decay set to le - 3. A learning rate scheduling of linear warm-up for 10 epochs to the peak learning rate of 3e 1 and cosine annealing decay to the minimum learning rate of 3e 6 is adopted. The batch size and the number of total training epochs are 16 and 120, respectively. We clip the gradient training.
Implementation Details
1.0 during
19


Reviews:
[{"review_id": "Ht3bDmGQbA", "reviewer": "Reviewer_veqt", "summary": "In this work, the authors propose a unified weakly supervised audio-visual scene understanding framework for audio-visual video parsing and audio-visual event localization. Different from previous works, the proposed Visual-Audio Label Elaboration (VALOR) method is simple and effective. It leverages large-scale audio-text and visual-text contrastively pre-trained models as the modality teachers to predict individual labels for audio and visual modalities to tackle the modality and temporal uncertainty issue and boost event parsing performance.  Extensive experiments and ablation studies on the LLP and AVE datasets can validate the effectiveness of the proposed approach. ", "questions": "Please address questions in Weaknesses. ", "limitations": "The authors discussed method limitations in the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed method is new and technically sound.\nTo alleviate drawbacks in past approaches, the proposed VALOR leverages large-scale audio-text and visual-text contrastively pre-trained models as the modality teachers to predict individual labels for audio and visual modalities for weakly-supervised audio-visual event parsing tasks.\n\n+ Extensive experiments and ablation studies on two datasets are provided, and the strong results can demonstrate the effectiveness of the proposed method.\n\n+ The core code is provided in the supplementary material, and the authors promised to release the source code.\n\n+ The paper is easy to follow. ", "weaknesses": "+ The process for selecting class-dependent thresholds in VALOR is not clearly outlined in the paper. These thresholds are crucial hyperparameters for generating audio and visual labels for training, and it appears that different classes may require different thresholds. Further explanation is needed on how these thresholds are chosen and how changes in these thresholds could potentially impact the performance of event parsing. \n\n+ The authors did not provide a clear explanation as to why directly using Knowledge Distillation (KD) with audio and visual semantic class embeddings results in even worse parsing performance (especially for audio event parsing) on the LLP dataset, as shown in Table 3. While the proposed VALOR method does improve performance, a more detailed analysis on the KD models would be beneficial for further understanding.\n\n+ The CLIP model uses a text prompt that is created by adding a \"A photo of\" prefix to the event's natural language form. However, the visual data involved in audio-visual event parsing are videos, not single images. This raises the question of whether there is a better prompt that could be used to improve model performance by taking into account the video nature of the data.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer veqt**\nWe thank Reviewer veqt for the positive comments and suggestive remarks. Please see our responses below for each raised issue. \n\n\n**Q1. Selection of class-dependent thresholds in VALOR. What if other threshold choices?**\n\nA1: For simplicity and fairness, we select such thresholds for each class based on its best segment-level F-score on the validation split. If using a class-independent threshold, which is determined by the best overall segment-level F-score on the validation split, we observe a comparable performance in Table 1 of our global rebuttal pdf.  Thus, whether the thresholds are class-dependent or class-independent does not affect the results significantly.\n\n\n| CLAP Event Thresholds| Segment-level Audio F-score |\n|----------------------|:--------:|\n| class-dependent      |    62.7    | \n| class-independent  |   63.4   |\n\n| CLAP Event Thresholds| Segment-level Audio F-score |\n|-|:-:|\n| class-dependent |    66.3    | \n| class-independent |   65.9   |\n\n\n**Q2. Why Knowledge Distillation (KD) with audio and visual semantic class embeddings results in worse parsing performance?**\n\nA2: In order to perform knowledge distillation (KD), the Softmax function is applied to the logits predicted by pre-trained models (CLAP/CLIP) and those produced by the HAN model, followed by calculating their KL-divergence. Since the Softmax function predicts a single dominant label, which does not align with the multi-label setting of the AVVP task, the use of the KD-trained model is expected to obtain degraded performances than VALOR.\n\n\n**Q3. Possible to exploit visual prompts to describe visual information in videos?**\n\nA3: We note that visual/video data in the task of AVVP are presented as a collection of consecutive one-second video clips, with very few visual changes in each clip. Thus, it is sufficient to apply an image-text pre-trained model of CLIP with the standard prompt of “A photo of” to extract the visual information.\nAlthough we did not design prompts specifically for video data, we have created a variety of prompts tailored for images as suggested by Reviewer JWv6. Through this, we aimed to explore whether generating more prompts for each event could lead to more accurate pseudo labels. The experimental results are presented in Table 4 of the rebuttal PDF. We observe that generating more prompts for each event slightly improves the accuracy of the generated visual pseudo labels, resulting in a 2.1 increase in segment-level visual F-score. However, this approach does not improve the accuracy of the generated audio pseudo labels; instead, there is a slight decrease of 1.8 in segment-level audio F-score."}, {"review_id": "pUSmxi3bB5", "reviewer": "Reviewer_82wA", "summary": "This paper proposes methods for weakly-supervised audio-visual event learning. The method relies primarily on pre-trained audio-language (CLIP) and visual-language (CLAP) models to guide the learning process. These pre-trained models serve as teachers and provide pseudo-labels which are then used to compute loss which guides the overall learning process. Experiments are done on an LLP dataset and the proposed method improves the state of the art by a good margin. ", "questions": "Please follow-up on the questions/concerns in the weakness section. ", "limitations": "The authors describe limitations of their work in terms of whether the approach will generalize to larger settings. Societal impact and other such limitations are not discussed. Given the nature and scope of this paper that might be okay. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "– The proposed approach is simple – it relies on pre-trained CLIP and CLAP models and achieves good performance in both segment level and event level. \n\n– One key claim the authors make is that modality independent learning can be crucial for audio-visual learning in some conditions. While most multimodal learning works focus on using multiple modalities to improve performance on a given task, the significance of learning independently from each modality is a good point to highlight. \n\n– The paper is mostly clear and easy to follow except for some mathematical descriptions which I believe can be improved at a few places. \n", "weaknesses": "— The proposed approach relies heavily on the pre-trained models audio-language and image-language. Considering that it might be important to understand the impact of these models themselves. I believe that the paper is missing some crucial analysis in those areas. Some which might help pain a better picture. \n\n--- What kind of information (pseudo-labels) these pre-trained models are providing for the audio and video modalities ? Can we get some insight into the distribution of \\hat{y}^m_t and how that is related (if any) to the actual video level ? Do the audio and visual model provide complimentary or similar information ? This can be analyzed before any training. \n\n--- It’s not fully clear (Table 3) what happens if the ground truth event labels are not used at all and just the pre-trained models are used to train. \n\n--- What is the impact of using different pre-trained models -- different CLIP/CLAP type models? \n\n— I think experiments on some other weakly supervised data might be helpful. \n\n— While the “non alignment” of the modalities in the videos does sound like a valid problem to address – it is not clear how much of that is present in the LLP dataset. How often does the video and the audio do align with the event label ? There is no analysis of to what extent that is present in the current dataset and to what extent it gets addressed. \n\nSome other comments/questions\n\n-- What is KD in in Table 3 ? Is it the one where ground truth labels are not used ? Just knowledge distillation from teachers ? \n-- Why is label smoothing applied to modality training targets ( line 106 ) but it’s not applied anywhere else ?\n-- For event localization, I am not sure accuracy is the right metric. Some metric as well as visualization which factors in the beginning and of the event will be more informative. \n-- The notations (line 145 - 147) q^P and q^m are confusing. Simplifying and clearly explaining what precisely they represent will improve clarity. \n-- How are the threshold parameters \\theta^P obtained for each class ?\n-- In the current VALOR formulation only CLIP/CLAP outputs for only classes marked to be present are used (the logical and with y). What if you \n\n\n--- Updated review after rebuttal----", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer 82wA**\nWe thank Reviewer 82wA for the positive comments and suggestive remarks. Please see our responses below for each raised issue.\n\n\n**Q1. Missing analysis on pre-trained audio/image-language models.**\n\n**(1a) What info is extracted from the pre-trained single-modality models?**\n\nA1a: The information produced by the pre-trained models is their confidence (i.e., logit) of the occurrence of each event in a video segment. The hard pseudo labels derived from these logits and video-label filtering indicate the presence of the events in each video segment.\n\n**(1b) Pseudo labels and their relation to the actual video labels.**\n\nA1b: In the supplementary material Table 7, we first validate the accuracy of the pseudo labels generated by VALOR before using them to train the model. The audio pseudo labels achieve an audio F-score of 84.92, and the visual pseudo labels achieve a visual F-score of 82.8. In contrast, if we use video labels directly as segment labels, the audio F-score is only 79.33, and the visual F-score is only 69.3. This demonstrates that the pseudo labels we generated are much more reliable.\n\n**(1c) Pseudo labels and their relation to the actual video labels.**\n\nA1c: In the supplementary material Table 7, we first validate the accuracy of the pseudo labels generated by VALOR before using them to train the model. The audio pseudo labels achieve an audio F-score of 84.92, and the visual pseudo labels achieve a visual F-score of 82.8. In contrast, if we use video labels directly as segment labels, the audio F-score is only 79.33, and the visual F-score is only 69.3. This demonstrates that the pseudo labels we generated are much more reliable.\n\n**(1d) What if no ground truth labels are used during training?**\n\nA1d: If the video-label filtering is not performed (i.e., discard the intersection operation with the video-level labels), significantly degraded performances would be expected because this hurts the pseudo label quality. We now provide such results as an additional ablation in the initial rows of the two subtables found in the rebuttal PDF Table 1. For example, the resulting audio/visual F-score is merely 47.9/53.8, compared to 63.4/65.9 produced by our VALOR.\n\nIf the reviewer's intention is to suggest not only avoiding the use of video labels for filtering but also not using video labels as the ground truth for calculating the loss, then the task becomes purely unsupervised. This deviates too much from the original setting (weakly-supervised) and goes beyond the scope of our work.\n\n**(1e) What’s the impact of pre-trained models?**\n\nA1e: To further assess the flexibility of using pre-trained models in VALOR, we now replace CLAP and CLIP with AudioCLIP [24] and OpenCLIP [A], respectively. The experimental results of substituting CLAP with AudioCLIP are presented in the left subtable of Rebuttal PDF Table 2, while the results of replacing CLIP with OpenCLIP are shown in the right. The segment-level audio F-scores before and after the substitution are 63.4 and 61.0, respectively.The visual F-scores before and after the substitution are 62.3 and 61.6, respectively. We can conclude that our VALOR is not limited to the use of the specific instance of CLIP/CLAP.\n\n[A] Cherti et al., “Reproducible scaling laws for contrastive language-image learning.” In CVPR, 2023.\n\n**Q2. Missing analysis on how often visual-audio misalignment occurs**\n\nA2: To address this issue, we consider the validation split of LLP. We found that there are a total of 9126 event segments with labels assigned for at least one modality, and 4048 of them appear to be non-aligned, i.e., the label is assigned to exactly one modality. The baseline method, HAN [72], only correctly predicts 188 segments (accuracy of 4.6%). While our VALOR and VALOR++ were able to predict correct per-modality labels for 1518 and 1850 segments (accuracy of 37.5% and 45.7%), respectively. The SOTA method of JoMoLD only predicted 1471 of them and thus resulted in a poorer accuracy of 36.3%. Thus, it shows that our method is better at handling the “modality misalignment” problem.\n\n**Q3. Missing analysis on how often visual-audio misalignment occurs**\n\nA3: KD in Table 3 means knowledge distillation, indicating that the model learns directly from the logits output from CLIP and CLAP. The details of KD are mentioned in Sect. 3.2 in the main paper. Yes, the ground-truth video labels are not used in KD.\n\n**Q4. Why is label smoothing applied to modality training targets (line 106) but not anywhere else?**\n\nA4: Label smoothing is a heuristic method introduced by Tian et al. [72], which multiplies the ground-truth video labels by modality-dependent coefficients to assign labels to audio and visual data. However, [72] did not explicitly explain how such coefficients are determined. On the other hand, our method directly predicts pseudo labels in each modality by utilizing pre-trained audio/visual-language models, under the guidance of video-level labels. \n\n\n**Q5. Proper metric for event localization**\n\nA5: In the AVE task determined by [71], each video is divided into segments of equal and fixed length, with each segment containing only one event label. With such a setting, only per-segment prediction accuracy is required for evaluation. Note that accuracy is the official metrics [71] used in the AVE task.\n\n**Q6. Confusing notations (line 145 - 147) of $q^P$ and $q^m$**\n\nA6: $q$ represents the probabilities/logits of event labels. The superscript $P$ denotes the output from pre-trained models (CLIP or CLAP) and $m$ denotes the data modality (audio or visual). For example, $q^{CLIP}$ is the probabilities output from CLIP and $q^v$ denotes the visual probabilities output from the HAN model.\n\n**Q7. Determination of the threshold parameters $\\theta^{P}$**\n\nA7: For simplicity and fairness, we select such thresholds for each class based on the performance on the validation split, i.e., the best segment-level F-score."}, {"review_id": "xGxTx0CqdM", "reviewer": "Reviewer_JWv6", "summary": "The paper tackles the task of audio-visual event parsing, where the goal is to independently recognize the localize the events occurring in the visual and audio modality. The paper argues that modality independent processing can be crucial for this task compared to joint modeling of the two modalities. Consequently, the paper makes use of pre-trained vision and audio models like CLIP and CLAP to harvest temporal labels independently for each modality. The proposed method achieves clear boost in scores across the board. The experiments are clear and sufficient.\n\nRebuttal: I have read the rebuttal. The rebuttal answers all my questions. So I have raised my rating.", "questions": "See weaknesses.", "limitations": "Yes, the authors have discussed the limitations.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "I think the paper is clearly written (in most places), has a clear motivation and proposes simple intuitive methods to solve the discussed issues. The improvement in performance is significant and is useful for future works in this space.", "weaknesses": "The proposed automatic label harvesting using CLIP and CLAP have not been evaluated directly. Since a test set is available with segment labels, is it possible to evaluate the automatic annotation technique using the ground-truth labels on these test sets to directly evaluate how well the automatic annotation procedure works?\n\nThe automatic training signal extraction technique could have been explored in more detail, as it is the crux of the paper. To start with line 156-157 are quite unclear. What does “contrastive models understanding logits” mean? The intersection operation with y is not described in the text at all. Also, how are the threshold values decided for each class? Is it helpful to keep a slightly lower confidence label (z_t < theta) that is present in “y”? If both CLIP and CLAP models predict the same label with low probability (because something else is dominating the scene), can we keep that? Also some captions that are used to query CLIP or CLAP could be too fine-grained. E.g. the exact musical instrument or vehicle (car or motorcycle) might be hard to identify, so during the automatic training label harvesting, multiple variations of the captions could have been tried, such as using synonyms for example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer JWv6**\nWe thank Reviewer JWv6 for the positive comments and suggestive remarks. Please see our responses below for each raised issue.\n\n**Q1. How accurate are our pseudo labels (derived via CLIP/CLAP)?**\n\nA1: In Table 7 of the supplementary material, we have evaluated the quality of the pseudo labels generated by VALOR on the test split. As seen in Table 7, the audio segment-level F-score reached 84.92, and the visual segment-level F-score reached 82.8. These scores are significantly better than the approach directly leveraging video-level labels as segment labels, whose audio and visual segment-level F-scores are 79.33 and 69.30, respectively. \n\nIn addition, we also evaluate the pseudo labels simply predicted by CLIP/CLAP (i.e., without our video label filtering), resulting in 17.74 and 31.89 F-scores in audio and visual modalities, respectively. This indicates the effectiveness of our VALOR which exploits video-level labels to guide the learning in each modality.\n\n**Q2. More details about the automatic training signal extraction technique. What does “contrastive models understanding logits” mean? The intersection operation with y is not described in the text at all.**\n\nA2: We thank the reviewer for giving us the opportunity to improve our paper. Regarding “contrastive models understanding logits”, they are the logits output from the pre-trained contrastive models (CLIP or CLAP), denoted as where superscript could represent CLIP or CLAP. This term comes from our use of pre-trained contrastive models for describing the associate segments, however, we will simply rephrase this term as modality-aware logits to avoid confusion.\n\nAs for the intersection operation, it is the logical AND between the hard labels obtained from modality-aware logits (with thresholding) and the video-level labels. This operation eliminates the events that are erroneously predicted by the pre-trained models but not presented in the video. We will add this to the main paper for completeness.\n\n\n**Q3. How are the threshold values decided for each class? Possible to exploit labels with lower confidence (i.e., )? What if both CLIP and CLAP predict the same label but with low probability?**\n\nA3: For simplicity and fairness, we select such thresholds for each class based on its best segment-level F-score on the validation split. If using a class-independent threshold, which is determined by the best overall segment-level F-score on the validation split, we observe a comparable performance in Table 1 of our global rebuttal pdf. Thus, the performance is not sensitive to the thresholds selected from the validation set. \n\nRegarding the second issue, reducing the thresholds for events occurring in the video may not be helpful. This is because such an approach is equivalent to reducing all event thresholds in our method.  In our work, since such thresholds are selected via validation, any further adjustments to the thresholds would only result in poorer pseudo labels.\n\n\n**Q4. Possible to exploit captions (e.g., variants or synonyms) for CLIP or CLAP to improve pseudo label quality?**\n\nA4: Following the suggestion of the reviewer, we now utilize caption variants using different prefixes for CLIP/CLAP. For CLIP, the prefixes are 'A photo of', 'An image of', and 'This photo contains'; as for CLAP, the prefixes are 'This is the sound of' and 'This audio contains'.\n\nIn addition, we also follow the suggestion and consider synonyms when describing event labels. For instance, synonyms for 'car' are 'automobile' and 'motorcar', synonyms for 'cat' are 'feline' and 'kitty', and synonyms for 'laughing baby' are 'chuckling infant' and 'giggling baby'. \n\nWith the above caption manipulation/augmentation, we observe a 2.1 improvement in pseudo labels' visual F-score when employing three captions as opposed to a single caption per event. However, in terms of audio, this approach does not yield superior results and actually results in a 1.8 decrease in F-score. The complete results are presented in Table 4 of our rebuttal PDF. How to properly exploit captions for improved pseudo label prediction will be among our future research directions."}, {"review_id": "qL92YvDkBD", "reviewer": "Reviewer_Qd1s", "summary": "The paper proposes visual-audio label elaboration (VALOR) for weakly supervised audio-visual video parsing. It generates fine-grained temporal labels in audio and visual modalities by harnessing large-scale pretrained contrastive models CLIP and CLAP and providing explicit supervision to guide the learning of AVVP models. The paper shows that utilizing modality-independent pretrained models and generating modality-aware labels are essential for AVVP.", "questions": "Please check [*Weaknesses]", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 The paper proposes a simple and effective AVVP framework, VALOR, to harvest modality and temporal labels directly from video-label annotations, with an absolute improvement of +8.0 F-score.\n\n2 The paper is the first to point out that modality independence could be crucial for audio-visual learning in the unaligned and weakly-supervised setup.\n\n3 VALOR achieves new state-of-the-art results with significant improvements on AVVP (+5.4 F-score) with generalization to AVE (+4.4 accuracy) jointly verified.", "weaknesses": "1 The paper is really well-written and clear. It would be great if the paper can have a wider use scenerio. The current paper and contribution, experiments are limited to LLP dataset. It would be great if the paper could extend the method to other datasets. Adopting other widely used datasets to validate the proposed method can be a big plus for the paper.\n\n2 The paper achieves big improvements than previous methods. It would be great if the paper could add FLOPs, throughput and number of parameters comparisons with previous methods. Model complexity comparison is a common practice for video related methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer Qd1s**\nWe thank Reviewer Qd1s for the positive comments and suggestive remarks. Please see our responses below for each raised issue.\n\n**Q1. Extension to datasets other than LLP?**\n\nA1: In addition to audio-visual video parsing (AVVP), we also tackled the task of audio-visual event localization using the AVE dataset [71], which is a dataset different from the LLP dataset for AVVP. The AVE experiment results are in Table 4 in the main paper, where we observed a 5.1 improvement in accuracy compared to the baseline method of using the HAN model. It is worth noting that both AVE and AVVP tasks perform learning from audio-visual data in the challenging “modality unaligned” setting.\n\n**Q2. Details on FLOPs, throughput, and number of parameters when compared with previous methods.**\n\nA2: Please see table below for the implementation details and comparisons. We also add such information in Table 3 of the rebuttal PDF.\n\n|                        | VALOR    |VALOR+|  HAN |MM Pyramid| MGN | JoMoLD | CVCMS|\n|------------------------|:--------:|:----:|:----:|:--------:|----:|--------:|-----:|\n| FLOPs (K)↓             |   17.3   | 27.6 | 17.3 | 84273    |901.5|  17.3   | 98.9 |\n| throughput (K)↑        |   45.1   | 40.8 | 48.3 | 9.0      |16.2 |  46.8   | 14.7 |\n| trainable params (M)↓  |   5.1    | 5.0  | 4.6  | 44.0     | 4.4 |  4.6    | 11.4 | "}, {"review_id": "MlqgNoDxvP", "reviewer": "Reviewer_t3gt", "summary": "The paper aims to improve performance of audio-visual video parsing (AVVP) on the LLP dataset via a novel pseudo-labelling technique. This is because LLP contains only coarse video-level labels, and expects the model to match fine-grained, temporally dense labels during testing. This mismatch (due to the intense labelling effort that would be required to have temporally dense labels in the training set) is obviously an issue, and has been approached by prev. works using the Multi-modal Multiple Instance Learning (MMIL) loss  for soft-selection and label smoothing, without much success. This paper proposes to use CLIP and CLAP, two pre-trained image/audio encoders, to extract pseudo-labels for each video/audio in an attempt to create temporally dense labels for the training set. This work also uses CLIP and CLAP as feature encoders for better results. This method considerably outperforms previous works, and its design choices are motivated by an ablation study. The method also performs well on audio-visual event localization.", "questions": "When do you plan to make the code available?\n\nTo the best of the authors' knowledge, are there any other works that use CLIP or CLAP for pseudo-labelling, perhaps in audio-only or image-only literature? If so, it would be very relevant to discuss such approaches in the related work.", "limitations": "Some limitations are mentioned, which is good. However, there should also be a broader impact discussion (can be brief) about the potential misuse of this technology (e.g., aiding automated audio-visual surveillance, what happens if this model is deployed and gives a wrong output, etc.). ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and does a good job at introducing the reader to the LLP task, the challenges it entails and why previous works have struggled with it. The figures are great, in particular figure 2, and the tables are all clear and have a clear purpose. The narrative of the paper is consistent and has good momentum, and the initial questions and issues it poses are adequately solved/answered by the end of the paper, with good justifications and conclusions.\n\nThe method is rather simple but quite elegant, and seems to be the a clear low-hanging fruit to improve the LLP task. The preliminaries are well-explained and the choice not to perform distillation (which, again, would seem like a low-hanging fruit) is well-justified.\n\nRelated work seems to be adequately, comprehensively, and fairly referenced and discussed.\n\nWhen compared with other works, the results are very convincing. Ablations are also welcome and have clear motivations and conclusions. Table 3 is especially good in my opinion - it answers a lot of questions.\n\n\n\n", "weaknesses": "I don't think the method has any particularly noticeable weaknesses, apart from the limitations that are brought up after the conclusion, which are understandable and are largely outside of the scope of this work.\n\nHowever, I think the paper would benefit heavily from more experiments. CLIP and CLAP are great, and they are compared with HAN (which is effectively a completely different model), but some comparisons with other audio/image encoders would help justify the choice of using CLIP and CLAP specifically. Also it would be great if the authors could experiment with different datasets or even new tasks, if this is possible, as it would give us a wider breadth of results to draw conclusions from.\n\nWould be good to have the best numbers in bold for Table 2. Would be good to highlight second and third best to highlight valor and valor+ rather than just valor++.\n\nSome extra comments on the poor performing classes in Figure 3 would be nice.\n\nI would suggest that the authors focus more on the term \"pseudo-labelling\" rather than \"teacher\" since student-teacher modelling is a broad topic with multiple techniques, but clearly here the authors are performing pseudo-labelling, which is a much more specific technqiue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer t3gt**\nWe thank Reviewer t3gt for the positive comments and suggestive remarks. Please see our responses below for each raised issue.\n\n\n**Q1. Possible to consider visual/audio-language models other than CLIP or CLAP? What about other datasets?**\n\nA1: In Table 2 of the rebuttal PDF, CLAP and CLIP are replaced by AudioCLIP and OpenCLIP, respectively. In the left subtable of Table 2, the audio performance of the model slightly drops when it is trained with the AudioCLIP’s generated labels. On the other hand, in the right subtable of Table 2, the visual performance of the model trained with OpenCLIP’s generated labels is nearly the same as that of the model trained with CLIP’s.\n\nAlthough most existing papers addressing the AVVP task only consider one dataset, we also tackled the task of audio-visual event localization using the AVE dataset [71], which is a dataset different from the LLP dataset for AVVP. The AVE experiment results are in Table 4 in the main paper, where we observed a 5.1 improvement in accuracy compared to the baseline method of using the HAN model. It is worth noting that, both AVE and AVVP tasks perform learning from audio-visual data in the challenging “modality unaligned” setting.\n\n**Q2. Better visualization for top cases/numbers in Table 2**\n\nA2: In Table 1 in the main paper, as suggested, the best numbers are highlighted in bold and the second best numbers are already underlined. We will take the suggestion and also highlight the third highest numbers and update the table caption accordingly.\n\n**Q3. Comments on poor performing classes in Figure 3**\n\nA3: When comparing to the model simply using videl-level labels for audio segment pseudo-labels, we found that “cat”, “baby cry” and “cheering” were the events with slightly degraded audio events, as shown in Fig. 3. For these three audio events, a drop up to 3 in F-score were observed. When performing error analysis, we found that this was mainly due to the decrease of true positives instead of the increase of false positive ones. This suggests that these three audio events exhibit large intra-class variations. Nevertheless, as also shown in Fig. 3, most audio events reported improved F-scores (up to 13).\n\n**Q4. Suggest focusing more on the term \"pseudo-labeling\" rather than \"teacher\"**\n\nA4: We agree that using the terms “teacher” and “student” might lead readers to think that we are employing the knowledge distillation techniques for teacher-student model learning. Since the focus of our work is to provide modality-specific segment-level labels by leveraging large-scale pre-trained open-vocabulary models (CLIP and CLAP), we will just use “pseudo labeling” in our paper to avoid confusion.\n\n**Q5. Will the code be publicly available?**\n\nA5: Definitely. We have enclosed the code regarding model architecture and loss calculation in the supplementary materials, and we will release the code within a week after the author notification.\n\n**Q6. Any other works that use CLIP or CLAP for pseudo-labeling, even for audio-only or image-only literature?**\n\nA6: As discussed in Sect. 4.1, VPLAN [95] addresses the AVVP task by utilizing CLIP for pseudo labeling, which leverages CLIP to generate visual segment-level labels. Recently,  LSLD [A] is proposed (after the NeurIPS submission deadline) to address the AVVP task with a similar method. Note that they only focus on visual pseudo labeling only, while we generate pseudo labels in both modalities. We will update Sect. 4 accordingly.\n\n[A] Fan et al. \"Revisit weakly-supervised audio-visual video parsing from the language perspective.\" arXiv:2306.00595.\n\n**Q7. Discussion on potential misuse of this technology**\n\nA7: Since our VALOR requires pre-trained audio/visual-language models to predict pseudo-labels, the lack of ability in describing fine-grained or unseen semantic concepts would be the major limitation. Misclassification of audio or visual events due to the above constraint might endanger video analysis tasks like surveillance or autonomous driving. On the other hand, since our VALOR is not a generative model, we do not expect that its misuse would result in DeepFake-like deceptive content."}, {"review_id": "5i5hsCuXsH", "reviewer": "Reviewer_uN7g", "summary": "This paper proposes a method called VALOR to assign segment level audio-visual (AV) event/object labels given weak labels at the video-level. It makes use of pseudo-labels obtained from unimodal pretrained models CLIP and CLAP to derive additional guidance for the AV model. The final model is trained using a combination of AV, audio-only and video-only losses, and the AV component is based on the hybrid attention network (HAN) structure. \nExperiments are performed on two tasks, AV video parsing and AV event localization. In both cases, the VALOR approach outperformed previous studies on the respective tasks by a clear margin. Some additional ablation experiments further validate the design choices that have been made in the system.   \n\n* After rebuttal\n\nGiven the authors' response, and them clarifying my concerns, I am increasing my score, especially for presentation.", "questions": "1. Do we know the order of the weak labels? \n2. In AV event localization evaluations, did the authors allowed a margin to evaluate the F-scores? \n3. The paper describes Type@AV as the Average of A-only, V-only and AV F-scores. It might make it clearer if the name reflects this fact. \n4. Event@ AV score is also not very self-explanatory. How does it differ from the AV F-score? \n5. Do the authors consider how the number of occurrences of an event in the training dataset correlate with the final performance? Is there any bias do to class imbalance? \n6. Paper mentions class-dependent threshold to binarize the CLIP/CLAP outputs. Are they manually or automatically tuned? If automatic, which subsets are used to determine these numbers? \n7. Instead of binarizing the labels from CLIP/CLAP, have the authors tried to use the soft labels directly in the loss computation? Soft-labels have sometimes been useful in other semi/self-supervised applications.\n8. What is the chosen segment length (duration)? Have the authors tested various granularity levels? \n9. The notation is Section 2 can be improved. Even though we can understand their meaning from the context, there are many f's in various forms: $f$, $F$, $\\mathrm{F}$\n10. The level of detail in the case of MMIL looks a little long given that the rest of the paper does not mention it that frequently in the rest of the paper.", "limitations": "Some limitations have been mentioned in the paper.  For example, whether the model will be effective in the case of large vocabulary labels. Also how the inherent limitations of CLIP can harm the proposed VALOR approach. \n1. It is not clear whether there are any issues due to class imbalance in the dataset. Or if the LLP data has a similar label distribution in train/dev/test splits? \n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength is that the experimental results clearly outperforms older approaches on the two tasks described in the paper. \n\n* Originality: Even though the individual sub-components such as CLIp, CLAP, HAN are not novel, the paper brings them together in a certain way (unimodal guidance with dense labeling of the video segments) to solve the AV video parsing problem on a weakly labeled dataset.\n* Quality: Experimental evaluations successfully support the use of the newly proposed VALOR technique. \n* Clarity: Language is mostly clear. \n* Significance: The paper probably establishes the new state-of-the-art the for the AV video parsing task on the LLP dataset.", "weaknesses": "* The main weakness of the paper is probably the flow of the paper. \n\n1. There is a section called Preliminaries and then in Section 3.1, CLIP and CLAP are introduced. They can go to preliminaries.\n\n2. The motivation behind adding Section 4.2 is not clear. Similar applications have been already included in the Introduction. \n* Another weakness is lacking clarity about some of the experimental settings: Instead of Section 4.2, it could be better to save some space and describe the dataset a little clearer. For example, \n\n1. how many labels are there, are they coming from a closed vocab? (We later get the answers to these questions by looking at Figs. 3 and 4) \n\n2. Even though the timestamps are not given, do we know the order of the labels, or is it more like a bag of labels at the video level? \nb. What does CLIP+CLAP without modality labels work in Table 2? Do we take the union of CLIP and CLAP labels and then use this as the ground truth for both audio and video losses?\n\n* The improvements in F-scores are mostly stated between HAN and the VALOR approach, but as mentioned at times, there were other stronger baselines to compare (JoMoLD and CMBS in Tables 1 and 4, respectively). It could be better to mention improvements w.r.t. those ones although the gains will look slightly smaller in that case.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer uN7g**\nWe thank Reviewer uN7g for the constructive comments and suggestive remarks. Please see our responses below for each raised issue.\n\n**Q1. Improvement of presentation flow. Sect. 2 and Sect. 3.1. The motivation behind adding Section 4.2.**\n\nSect. 2 (Prelim) defines the task of AVVP, including the baseline HAN [72]. While Sect. 3 introduces our proposed model, Sect. 3.1 is to explain the idea of exploiting pre-trained cross-modality language models for zero-shot label transfer, in which CLIP and CLAP are utilized. As suggested by Reviewers 82wA and t3gt, we have verified that our method is not limited to the use of CLIP/CLAP.\n\nOur Section 1 explains audio-visual learning tasks and highlights the challenge of “modality unaligned” settings, which is also the main focus of our paper. To maintain the fluency and clarity of the Introduction, we only provide a few examples of related studies. As for Section 4 (Related Work), we provide thorough literature reviews covering relevant research on audio-visual learning in various settings.\n\n**Q2. Clarity in some experimental settings. How many labels? Are they from a closed vocab? Is the label order available? What’s CLIP+CLAP w/o modality labels in Table 2?**\n\nAs noted in Sect. 5.1, there are 25 events (labels) in total, which are from a closed/predefined vocabulary. With the dataset and settings defined in [72], the order of the video labels is not available. Table 2 is for ablation study purposes. CLIP+CLAP without modality labels means the pseudo labels derived for each segment are the union of audio and visual labels produced by VALOR, followed by training the HAN model for AVVP. Since degraded performance was observed, it verifies the effectiveness of our VALOR in predicting modality-aware pseudo labels for cross-modal learning.\n\n**Q3. Possible improvements on strong baselines (JoMoLD and CMBS)?**\n\nIn the second paragraph of Section 5.2, we have presented the improvement over JoMoLD. We note that VALOR scored higher across all metrics, including a 5.4 F-score improvement for segment-level Type@AV under a fair setting. In the quantitative results of Section 5.4, we also noted that we surpassed CMBS on the weakly-supervised AVE task with an improvement of 4.4% in accuracy.\n\n\n**Q4. Margin to evaluate the F-scores?**\n\nNo, and we are not able to do so. This is because, according to [71], F-score is not the evaluation metric defined for the AVE task. Since the AVE dataset consists of video segments with equal and pre-determined length, we adopt accuracy as the evaluation metric Moreover, since each video contains only one event, each segment is labeled with only one event label, which is why accuracy is qualified as an evaluation metric.\n\n**Q5. Type@AV naming confusion**\n\nWe thank the reviewer for the advice. We agree that Type@AV might not be sufficiently intuitive to represent the average of the A-only, V-only, and AV F-scores. However, since the term Type@AV was introduced in the first paper of the AVVP task [72]. Existing literatures (including ours) addressing AVVP follow this notation for presentation consistency.\n\n**Q6. How Event@AV differs from the AV F-score?**\n\nDefined by [72], Event@AV calculates the F-score of all audio/visual events (i.e., across all segments, modality-aligned or not); as for AV F-score, it is the F-score calculated for the “audio-visual” events only (i.e., across modality-aligned segments). \n\n\n**Q7. Does the number of event occurrences correlate with performance? Is there class imbalance bias?**\n\nTo address this issue, we show Figure 2 in the rebuttal PDF, showing that the number of event occurrences does not correlate with the final performance on the validation split. Note that the leftmost sub-figure in Fig. 2 shows the number of videos containing each event, the middle sub-figure shows the event-wise audio F-scores, and the rightmost figure shows the event-wise visual F-scores. Moreover, since ㄍthe F-score is calculated by (2 * TP) / (2 * TP + FP + FN) for each class, its value would not be biased even with the presence of class imbalance.\n\n**Q8. Determination of class-dependent thresholds to binarize the CLIP/CLAP outputs**\n\nFor simplicity and fairness, we select such thresholds for each class based on its best segment-level F-score on the validation split. As suggested by Reviewer veqt, if using a class-independent threshold, which is determined by the best overall segment-level F-score on the validation split, we observe a comparable performance in Table 1 of our global rebuttal pdf. Thus, the performance is not sensitive to the thresholds selected from the validation set.\n\n**Q9. Possible use of soft labels?**\n\nYes, we have tried to let our model directly learn from the logits output from the pre-trained models (without thresholding into hard labels), which is the knowledge distillation (KD) experiment in the second row of Table 3. From this table, we observe that such labeling strategy only resulted in F-scores of 51.1/64.0 for audio/visual events, while our VALOR achieved 62.7/66.3. Thus, use of our proposed VALOR to leverage video-level labels for pseudo label prediction is desirable.\n\n**Q10. What is the chosen segment length? Have the authors tested various granularity levels?**\n\nEach segment length (duration) is 1 second. Since the length of each video segment is pre-determined and fixed by [72], we are not able to consider video segments at different granularity levels.\n\n\n**Q11. The notation in Section 2 can be improved (e.g., many f's in various forms: $f$, $F$, $\\rm F$)**\n\nThank you for pointing this out. $f\\in R^d$ represents a segment audio/visual feature, $F\\in R^{T \\times d}$ represents the collection of all segment audio/visual features, and $\\rm F\\in R^{2 \\times T \\times d}$ represents the collection of all segment features in both modalities. To avoid confusion, we will replace $\\rm F\\in R^{2 \\times T \\times d}$ with $X$ instead.\n"}]

Author rebuttal:
## **General Response** 

We sincerely appreciate the valuable time and insightful feedback provided by the reviewers. We are grateful for the opportunity to address the concerns raised by each reviewer, which fundamentally strengthens our work. The strengths pointed out by the reviewers include:
	
**Method**: Motivation is clear. [Reviewer JWv6, t3gt]. Our VALOR is simple, technically sound, and novel. [Reviewer veqt, 82wA, JWv6, Qd1s, t3gt, uN7g]. 

**Experiment**: Extensive experiments (on two types of audio-visual tasks) and ablation studies are provided. [Reviewer veqt]

**Performance**: Strong performance was shown to prove the effectiveness of the proposed method. [Reviewer veqt, 82wA(good performance), JWv6, Qd1s, t3gt, uN7g]

**Presentation**: The paper is mostly clear and easy to follow. [Reviewer veqt, 82wA, JWv6, t3gt]

**Importance of modality-independent learning**: The claim that modality-independent learning being crucial for audio-visual learning under noisy audio-visual conditions is a good point to highlight. [Reviewer 82wA, Qd1s]

We would like to point out that particular concerns are raised, as listed below. Please refer to the responses to each reviewer for further details.

1. Selection of class-dependent thresholds [Reviewer veqt, JWv6, uN7g]
2. Variations of text prompt [Reviewer veqt, JWv6]
3. Ablation studies on video-label filtering [Reviewer 82wA]
4. Flexibility of re-trained model selection [Reviewer 82wA, t3gt]
5. Performance on runtime/speed [Reviewer Qd1s]
6. Label distribution analysis and its effect [Reviewer uN7g]


We thank the reviewers again for the suggestions and the raised issues. Should the reviewers have follow-up questions, we will be more than happy to answer them in the next phase. Given the recognized strengths in the initial reviews, together with additional experiments and clarification provided during rebuttal, we hope this work would be of great value to the audio-visual learning community.

Meta-review: The paper has been well received with all reviewers being convinced by the rebuttal and being positive about the paper. The evaluation and additional comparisons provided (for instance, additional models other than CLIP and CLAP), the evaluation on the different settings were all appreciated. The consensus of the reviewers is to accept the work. AC agrees with this recommendation with the suggestion that the final version should include all the additional responses provided in the rebuttal.
Decision: Accept (poster)