Revisit Weakly-Supervised Audio-Visual Video Parsing from the Language Perspective
Authors: Yingying Fan
Yu Wu
Bo Du
Yutian Lin
Venue: NeurIPS 2023 poster (2023)
Keywords: Weakly-Supervised Audio-Visual Video Parsing
Language Guided Segment-Level Label Denoising
Dynamic Re-weighting

Abstract: We focus on the weakly-supervised audio-visual video parsing task (AVVP), which aims to identify and locate all the events in audio/visual modalities. Previous works only concentrate on video-level overall label denoising across modalities, but overlook the segment-level label noise, where adjacent video segments (i.e., 1-second video clips) may contain different events. However, recognizing events on the segment is challenging because its label could be any combination of events that occur in the video. To address this issue, we consider tackling AVVP from the language perspective, since language could freely describe how various events appear in each segment beyond fixed labels. Specifically, we design language prompts to describe all cases of event appearance for each video. Then, the similarity between language prompts and segments is calculated, where the event of the most similar prompt is regarded as the segment-level label. In addition, to deal with the mislabeled segments, we propose to perform dynamic re-weighting on the unreliable segments to adjust their labels. Experiments show that our simple yet effective approach outperforms state-of-the-art methods by a large margin.

Revisit Weakly-Supervised Audio-Visual Video Parsing from the Language Perspective
Yingying Fan, Yu Wu, Bo Du, Yutian Lin" School of Computer Science, Hubei Luojia Laboratory, Wuhan University (fanyingying.cs, wuyues, dubo, yutian. 1in] edu. cn
Abstract
We focus on the weakly-supervisee audio-visual video parsing task (AVVP), which aims to identify and locate all the events in audio and visual modalities. Previous works only concentrate on video-level overall label denoising across modalities, but overlook the segment-level label noise, where adjacent video segments (i.e., I-second video clips) may contain different events. However, recognizing events the segment is challenging because its label could be any combination of events that occur in the video. To address this issue, we consider tackling AVVP from the language perspective, since language could freely describe how various events appear in each segment beyond fixed labels. Specifically, we design language prompts to describe all cases of event appearance for each video. Then, the similarity between language prompts and segments is calculated, where the event of the most similar prompt is regarded as the segment-level label. In addition, deal with the mislabeled segments, we propose to perform dynamic re-we weighittnng the unreliable segments adjust their labels. Experiments show that simple yet effective approach outperforms state-of-the-
methods by a large margin. Code and data are available https: :/ gi thub. com/ CSySSLLL
Introduction
Classic audio-visual tasks such as audio-visual event localization and audio-visual representation learning have been well explored previous works [1, 2, 3] and have achieved promising performance. However, most existing research studies on these tasks assume that audio and visual are correlated and temporally aligned, which is not always the case. For example, a person can hear dog barking but not see it in the video. To this end, we focus on the more general audio-visual video parsing (AVVP) task, which aims to recognize all events each modality and localize their temporal boundaries.
Since the full annotation process time-consuming and expensive, Tian et al. [4] address AVVP in weakly-supervised manner, which only requires video-level labels, without specifying the modality and temporal boundaries. They develop a hybrid attention network to leverage unimodal and cross-modal information and propose simple Multimodal Multiple Instance Learning (MMIL) framework to aggregate segment-level predictions into video-level ones. However, weakly-supervised labels cannot determine which modality the event comes from. Wu et al. [5] then introduce video-level label refinement method and adopt contrastive learning for audio and visual temporal alignment. Furthermore, Chen et al. [6] propose another label denoising method based on alleviating modality-specific noise through cross-modal loss patterns for both modalities.
However, previous works [5, 6] only considered the noisy label the overall video instance level, while ignoring the segment-level label noise. Segment-level label noise indicates the fact that not every segment of a video contains all the events of the video. For example, person may be speeching
"Coreeponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

in the video for a few seconds, and then he disappears in the following seconds. Therefore, it is inappropriate to consider speech as a supervised signal for all segments, where speech can be regarded as noisy label for some segments of this video. It could harm the network in event classification on segments when training with such segment-level noisy labels.
The challenge of learning reliable segment-level labels comes from that the label of each segment is not predefined, but could be any combination of events that occur in the video, e.g., segment may contain both speech and music events or none of them. To address this flexible label assignment challenge, we propose to leverage a third modality, the language, which enables better scene understanding beyond the labels. Language can describe what is happening in the scene, and who is involved, and identify segments not covered by the existing labels. Therefore, we propose to address AVVP from the language perspective, where natural language descriptions serve as a bridge connecting video and flexible weak labels.
In this paper, we propose I. Language guided Segment-level Label Denoising (LSLD) strategy. We leverage the generalization capability of language construct prompts that indicate all cases of event appearance for a segment. To connect vision and language, we introduce a vision-language pre-trained model, CLIP [7], into our work. Specifically, CLIP used to extract the visual features for all segments, and the similarity between the segment and all prompts is calculated for label denoising, In addition, to further mitigate the effect of noisy segment labels, we perform a dynamic re- weightitgg strategy on the segments that may be mislabeled. Specifically, we calculate the similarity between the language description of each event and all the segments across the dataset. We do this for both the segments with or without the event label (i.e., potentially positive and negative clips) separately to compare their similarity distribution. a segment's similarity falls within the overlap of the positive and negative groups, we consider it an unreliable segment and adjust its label dynamically according to its similarity to the event.
For the audio track, we introduce a large-scale audio-language pre-trained model LAION-CLAP [8] to denoise the audio label. The main contributions of our work are summarized follows:
We introduce the language modality to the AVVP task and construct language prompts that indicate all cases of event appearance for a segment, where the prompt with the highest similarity is regarded as the segment label to avoid segment-level label noise.
We propose simple yet effective Language-guided Segment-level Label Denoising (LSLD) mechanism based on CLAP and CLIP to align the modality of audio/visual and language for segment-level label denoising. In addition, we mine the visual segments with unreliable labels and learn from them with dynamic weight.
Experiments show our method achieves comparable performance with the state-of-the methods. Especially, we improve visual metric significantly from 66.4% to 71.3% on the segment level over the state-of-the-art.
2 Related work
Audio-Visual Learning. Audio-visual learning includes various interesting tasks such as audio- visual representation learning [9, 10, 11, 12, 13, 14, 3, 15), audio-visual event localization [1, 16, 17, 18, 19, 20, 21]. audio-visua sound separation [22, 23, 24, 25, 26, 27], audio-visual sound source localization [28, 29, 30, 31], and audio-visual action recognition [32, 33, 34, 35). Most previous work on these tasks assumes that audio and visual are temporally aligned and always correlated. However, in many videos, audio and visual do not share the same information at a certain time, for instance, something that can be seen is not audible and the sound source is not visible. Therefore, we focus on the andio-visual video parsing task and aim to separately identify the events that happen in each modality with only weakly-supervise video-Ievel labels for training.
Audio-Visue Video Parsing. Audio-visual video parsing (AVVP) task [4] intends to recognize events in each modality and localize their temporal boundaries. Since labeling each modality and segment of a video is time-consuming and -abor-intensive, recent work has been conducted in a weakly-supervisee manner. Tian el al. [4] utilize a hybrid attention network and multiple instance learning mechanism to aggregate segment features into video-level ones. Wu et al. [5] adopt the exchange of unrelated video and audio tracks to denoise veakly-supervvied labels at the video level. More recently, to explicitly group semantic-awar aware multi-modal contexts, Mo et al. [36] propose
2

---

Clrs A
Crestentt
Chor
Hightt
Closk
Sealerity
Label Beeoninn
Clansh
Other
Languag Fronpt
Claws
19 Dlugands
Ofher
repoeees
Langrage Goided Segnanbl Noley Label Hernoring
Sealurity
FELLBTT Aggripate
dittieake Cf
Label
Cdlle
MNI
Lou
Calla
Dewdeed
Skiging
seightg
Dynonke Re-neighring Cnrelatle Segnests
Traising
Segneet-Javel
Figure I: Algorithm Overview. (a) Language prompts construction provides prompts for video to indicate all cases of event appearance based on the weak label of the video. (b) Language guided segment-Level noisy-label removing calculates the similarity between prompts and segments, where the prompt with the highest similarity is taken as the segment label. (c) Dynamic re-weighting for unreliable segments gathers the segments with and without the event using the denoised labels and calculates their similarity to the event name respectively, then re-weight the segments' label on the overlap region according their similarity. (d) Training with Denoised Label on Segment-Level.
Finally, with denoised segment-level labels, we train the network following [4]. CLIP and CLAP are frozen during the process. In the figure, "-> 0 " denotes label removal, & is the cross product, and M denotes the maximum function.
Multi-modal Grouping Network to learn more discriminative audio and visual representations. Chen et al. [6] propose a dynamic training mechanism, which lowers the modality-speciit noise at the video level. Different from the above approaches, we propose to investigate segment-lew label noise. Specifically, we introduce the language modality into the AVVP task and remove noisy labels on the segment level by constructing prompts and calculating the text-image similarity.
Language-Superrvised Vision Representation Learning. Language-supp supervised vision represen- tation learning aims learning visual representations from language data. Early works for visual representation learning include predicting the bag-of-words representation [37] or training n-gram models [38] the YFCC100M dataset [39]. While some work like ICMLM [40] and VirTex [41] produce effective visual representations of COCO Captions [42]. Soon after, CLIP [7] quickly draws attention due to its simplicity, scale, and generalization capabilities. CLIP pre-trained on 400 million image-text pairs using the contrastive learning approach and can be transferred to multi-modal downstream tasks like image caption [43], and video caption [44] due to its zero-shot transfer ability. Thus we employ CLIP as our image encoder to bridge the gap between the image and language descriptions for segment-level label noising.
3 Method
In this paper, we propose a novel Language guided Segment-level Label Denoising (LSLD) method for AVVP, which aims to remove the segment-level noise with the help of natural language. The overview of proposed method shown Fig. I, where language prompts are generated to calculate the similarity with the segments for denoising. Then dynamic re-weight is adopted to adjust unreliable segment labels. With the newly assigned segment-level label, the network is trained end-to-end to better classify events segments.
3

---

Following, we introduce our method in detail. We begin with the preliminary and our baseline method (Sec. 3.1). Then we illustrate the language prompt construction (Sec. 3.2), segment-level label denoising (Sec. 3.3), and the dynamic re-weighting strategy (Sec. 3.4).
3.1 Preliminaries
Problem Setup and Notations. Given a T-segments audio-visual video sequence S [V,, A, A. A and V are the audio and visual track, respectively, and each segment of one second long. Our goal is to predict the event label of each segment in audio and visual, which may contain multiple or no events at all. During evaluation, for the 1-th video segment (V,,A)), the target is denoted as (yf, y(") R RIXC where yi. y", and y; are labels for audio, visual, and audio-visual events, respectively, and C' is the number of event classes. During training, we can only obtain the weak label of the video as whole, which is not specific modality and segment.
Baseline Framework. To obtain cross-modal and uni-modal information, we adopt HAN [4] as our baseline. HAN [4] extracts visual feature F = {fr')== E R xd and audio feature F [ E IR 7xd by Resnet [45] and VGGish [46], where d is the feature dimension. Then it adopts self-attention and Pross-attention layers to aggregate cross-modal and uni-modal information each segment. After gathering the audio and visual features, the segment-level event prediction for audio and visual (pi; P E IR. Ix can be obtained through an FC layer and a sigmoid function. Then the video-level prediction is aggregated by attentive MMIL pooling, which is formulated as:
T W.P..
T
M
pd = W WPP p-
aav = =Σ 1=1
W [m]] o [m|m].
(1)
where = (w2. w denote the temporal attention weights, P. P.= {pf, p includes audio and visual predictions, is the number of modalities. Finally, binary cross-entropy (BCE) loss adopted to optimize the model in weakly-superviser way:
Disse BCE(y"", .p"") + BCE(Y", .p") BCE .p").
(2)
where y", F" are video-level audio and visual labels by performing label smooth on y to mitigate the impact noisy labels. As the baseline method only utilizes video-level labels during training, which is affected by the segment-level noise, we propose Language guided Segment-level Label Denoising (LSLD) method to address segment-level noise.
3.2 Language Prompt Construction
As discussed above, during training, we can only access video-level labels, but for each segment, the label could be any combination of events, which is flexible. To address the flexible label assignment challenge, we introduce the language modality into our task, which describes how the events occur. For each video, to consider all cases of event appearance, we create prompts for each case according to its video-level labels. As shown in Fig. (a), taking a video labeled with two classes named A and as example, we construct 4 prompts describing A appears, appears, both classes appear and none of them appear, respectively. Finally, the language prompts for the video could be:
[Class A, Class B, Class A Aand B, other), where other means no pre-defined event appears.
3.3 Language Guided Segment-Level Label Denoising
With language prompts generated in Sec. 3.2, we intend to remove segment-level noisy labels by language guidance. Specifically, we calculate the similarity between each segment and prompts, where the prompt with the highest similarity is more likely to describe the segment accurately. As illustrated in Fig. (b), take the visual modality for example, we feed the constructed language prompts into CLIP's text encoder obtain the text features, and each frame of the video is fed into CLIP's image encoder to extract the image feature. We denote the text feature as f, E N xd and the visual feature as f, E Txd where N is the number of prompts, T is the number of segments, and d is feature dimension. Then we calculate the text-image similarity matrix s E R Nxx on the normalized text and image feature:
f G
s = Softmax
)T)
(3)
|fc.1

---

where @ denotes the matrix multiplication. For the audio modality, we feed the audio into CLAP's audio encoder and the language prompts into CLAP's text encoder.
With the similarity matrix, the language prompt with the highest similarity for each segment is obtained, which infers the segments' corresponding event label. For instance, as shown in Fig.1 (b), the video is labeled with class A and class B, while the most similar prompt of its second segment is Class A, which means that this segment most likely contains only with class A. In this case, we treat class B as the noisy class of this segment for label denoising. Likewise, if the most similar prompt is other, both class A and class B will be treated as noisy labels.
With segment-level label denoising, each audio/visual segment is assigned its own event label y? y?; instead of being regarded as having the same label the whole video. In this way, the labels depict the aadio/visual segments more accurately which boosts network learning.
3.4 Dynamic Re-weighting for Unreliable Segments
In Sec. 3.3, each segment is re-labeled to avoid segment-level noise. However, such noise is inevitable, because some segments may be mislabeled. To further alleviate the impact of mislabeled segments, in this section, we propose to mine the unreliable segments and learn from them with dynamic weight.
To mine the unreliable segments, for each event class, we aggregate the segments with and without the event into two groups (i.e., potentially positive and negative segments) by the denoised segment-level labels. Then, we calculate the similarity between the visual segments and the language prompts of the class for the two groups separately to compare their similarity distribution. The whole process is shown in Fig. (c), where the similarity distributions of with/ without Car are presented. We observe that though some visual segments in orange are labeled as without a Car, their similarity is still high, while some segments in blue that are labeled as with a Car are not that similar to the class description. Therefore, we consider the overlap region of the two similarity groups as mislabeled segments.
Then, we propose a dynamic re-weighting strategy to learn unreliable segments softly. Specifically, for an event and a corresponding unreliable segment, the weighted similarity between the segment and event description is adopted as the soft label. Then, for the t-th segment of video, its label for event class e can be calculated as:
Min(1, a x Se),
Y [c] = 1 and Se S MaTw/or
si
Min(1, 3 x se),
y [c] = 0 and Se 2 Minw,
(4)
yilc),
otherwise,
where Min() is function to return the item with the lowest value, and 3 are parameters for the segment with/ without the event c. Se is the similarity between segments and class description using Eq. 3. Min., is the lowest similarity of the segments with the event, Marw/o is the highest similarity of segments w/o the event. If the similarity of a segment w/o the event is higher than Minw, then its similarity is set to Min 8 x s,). If the similarity of segment with the event is lower than Maxw/o then its similarity is set to Min(1 a x Sz). Note that, during the application, we set a > B to ensure the segments with f7 [c] = 1 still have more possibility of the event appearance. We only adopt dynamic re- weighting on the label of unreliable visual segments.
After dynamic weiigtghn, we use the segment-level label to optimize the model on the audio and visual modality. The pipeline is shown in Fig. (d). Specifically, we replace the video-level label y with our segment-level label yr, and y with the segment- label yf in Eq. Notably, we also use CLAP//LIIP to extract the audio/visual features during training since CLAP//LIP trains from large-scale datasets and learns richer audio/visual information.
4 Experiments
4.1 Experimental Setup
Dataset. In the AVVP task, we only evaluate our method on the Look, Listen and Parse (LLP) Dataset [4] following previous AVVP work. The dataset contains 11,849 video clips 10 seconds long from 25 different event categories, including human activities, animals, musical instruments, etc. For the training process, we use 10,000 video clips with only vidco-level event labels. The remaining
5

---

Table Comparisons with the state-of-the-art methods on LLP dataset. "CLAP+CLIP" means we use the audio and visual features extracted by CLAP [8] and CLIP [7] respectively, while represents our baseline framework. The best results are marked in bold.
Segment-Leve
Event-level
Method
A
A-V
Type
Event
A
A-V
Type
Event
AVE [1]
47.2
37.1
35.4
39.9
41.6
40.4
34.7
31.6
35.5
36.5
AVSDN [16]
47.8
52.0
37.1
45.7
50.8
34.1
46.3
26.5
35.6
37.7
HAN [4]*
60.1
52.9
48.9
54.0
55.4
51.3
48.9
43.0
47.7
48.0
MA [5]
58.9
57.9
53.6
49.0
53.0
50.6
CVCMS [47]
60.8
63.5
57.0
60.5
59.5
53.8
58.9
49.5
54.0
52.1
DHHN [48]
61.4
63.4
56.8
60.5
59.5
54.6
60.8
51.1
55.5
53.3
MM-Pyramid [49]
61.1
60.3
55.8
59.7
59.1
53.8
56.7
49.4
54.1
51.2
MGN [36]
60.8
55.4
50.4
55.5
57.2
51.1
52.4
44.4
49.3
49.1
JoMoLD [6]
61.3
57.2
60.8
59.9
53,9
59.9
49.6
54.5
52.5
CMPAE [50]
64.2
66.4
59.2
63.3
62.8
56.6
63.7
51.8
57.4
55.7
LSLD (ours)
62.7
67.1
59.4
63.1
62.2
55.7
64.3
52.6
57.6
55.2
LSLD+CLAP+CLIP
68.7
71.3
63.4
67.8
68.2
61.5
67.4
55.9
61.6
60.6
Table 2: Ablation study of each component of LSLD. Here, "DeN" denotes language-guider segment- level label denoising. "ReW" denotes the dynamic weighting strategy on unreliable segments. All experiments are based CLAP and CLIP features.
Segment-Level
Event-level
DeN
Rew
A
A-V
Type
Event
V
A-V
Type
Event
x
x
64.1
54.9
49.0
56.0
60.0
53.3
51.0
41.8
48.7
51.6
X
67.3
70.1
61.2
66.2
67.2
60.8
66.3
54.3
60.5
60,2
V
68.7
71.3
63.4
67.8
68.2
61.5
67.4
55.9
61.6
60.6
1849 validation and test videos possess modality and segment-specific labels (i.e., start and end time of each event on audio and visual track). We conduct experiments following the official data splits from the LLP dataset.
Evaluation Metrics. We evaluate the parsing performance of audio, visual, and audio-visual events under both segment-level and event-level metrics by adopting F-scores. The segment-leva metrics can evaluate segment-wiss event prediction performance. For the event-level metrics, we extract events by concatenating consecutive positive segments in the same event and compute event-level F-score with mloU=0.55 as the threshold. In addition, we evaluate the parsing performance of audio-visual events using Type @ AV and Event AV metries. Type @ AV calculated by averaging audio, visual, and audio-visual event evaluation results while Event @ AV considers all audio and visual event categories for each sample instead of directly averaging results from different events.
Implementation Details. We conduct the training and evaluation processes on single NVIDIA GTX 2080 Ti GPU with GB memory. Following the data preprocessing in previous works, we decode a 10-second video at fps into 10 segments. Instead of using pre-trained ResNet- 152 [45] and R(2+1)D [51], we also utilize CLIP's [7] image encoder (i.e., pre-trained ViT-B/16 [52]) to extract visual features with richer semantic information. Furthermore, we leverage CIAAP [8] pretrained on Audioset [53] extract the audio features instead of using VGGish [46]. Following HAN [4] we adopt the Adam optimizer and the learning rate 2c-4 drops by factor of 0.25 for every 6 epochs. We train the model with a batch size of 32 for 20 epochs. We also add layers HAN [4] to increase the complexity of the network for better performance. ar is set to 4 and is 0.4.
4.2 Comparison with State-of-the-art Methods
To demonstrate the effectiveness of our method, we compare it to several baseline works including the audio-visual event localization approach AVE [1] and AVSDN [16] and the state-of-the-art methods for the weakly-AVVP task, such as HAN [4], MA [5], JoMoLD [6], etc.

---

Table Analysis of different strategies to indicate that no event appears in the prompt.
Segment-Level
Event-level
Method
V
A-V
Type
Event
A
A-V
Type
Event
Class, Not Class]
40.8
63.1
39.3
47.7
51.9
33.8
59.8
33.5
42.3
44.4
Class, None]
66.3
71.4
61.2
66.3
67.1
57.9
67.8
53.5
59.7
58.7
Class, Other]
68.7
71.3
63.4
67.8
68.2
61.5
67.4
55,9
61.6
60.6
Table 4: Algorithm analysis of dynamic re-weighting. All results are of visual metric.
(a) Analysis different and 3 values.
(b) Analysis of different re-weighting strategies.
B
Visual
0.2
0.4
0.6
0.8
Method
Segment-Leva
Event-Lovel
3.5
71.1
71.1
71.0
weighttht w/ event
70.4
66.8
71.2
71.3
70.9
70.6
re-weight on w/o event
70.7
67.0
4.5
70.3
70.9
70.3
70.2
stable re-weight
70.5
66.8
69.0
70.3
70.8
69.9
dynamic re-weight
67.8
As shown in Table 1, we conduct our experiments on the LLP dataset with all state-of-the-art methods. We observe that our method has competitive performance all evaluation metries. Compared with our baseline method HAN [4], the proposed method can significantly improve the performances on visual metrics. Especially, our method gains 14.2 and 10.5 points on segment-level visual parsing and audio-visual parsing metrics with ResNet and VGGish features, respectively. Similar improvements also observed on event-level metries.
Compared with state-of-the-ie art methods, our method still yields better performance on all visual- related metrics. Especially, compared with JoMoLD, achieved 3.3, 2.2, and 2.3 points of improvement in terms of segment-level visual, audio-visual, and type parsing. With the CLAP and CLIP features, improve the segment-level audio metries from 62.7% 68.7%, and visual metrics from 67.1 1% to 71.3%.
4.3 Ablation Studies
Effectiveness of language guided segment-level denoising. The ablation study of denoising is shown in Table 2. We observe that the results on segment-level audio/visual parsing are improved by 3.2%//1.25% when we construct prompts and remove segment-level noisy labels by language guidance. The significant improvement demonstrates that with the help of language description, our method does provide more accurate segment-level labels for both audio and visual modality, which enhances network learning.
Analysis of different types of prompts. As shown in Table 3, investigate the influence of using different language descriptions to express that no event appears in the segment. We observe that when using prompts like [Class, Not Class), the performance is much lower than method. We suspect the reason is that the text encoder is more sensitive to meaningful words like event class while neglecting the negative word not. When using none, the result in audio-visual metrics is about 2.2 points lower than ours. The reason is that though the pre-defined events do not appear, background scenes like sky and trees may exist. When using the description other, obtain the best performance, where we neither throw away the possibility of the appearance of background classes nor introduce event categories that do not appear.
Effectiveness of dynamic re-weighting. The ablation study of dynamic re-weighting shown in Table 2. We observe that the results on visual parsing improved by 1.2% and 1.1% segment- level and event-level, respectively. The improvement demonstrates that our method accurately finds the possible mislabeled segments to learn them softly.
Analysis of different a and 3 values. According to Eq. 4, the hyper-parameter a and 3 are used to be multiplied with similarity to serve the weight for unreliable segments with/ without the event, respectively. In Table (a), as a increased from 3.5 to the results are usually improved, as our denoising method identifies most of the segments into the correct category, larger weight should

---

Souach
2oodt
Secech wel Speach
Chideen rocater
Ryyog_/oool
w/ Chicken J906641
wel Frying (beei
0.35
1.55
0.0
210
Cante 3m11
Costa
(a) Speech.
(b) Chicken_ mooster_
(c) Frying food.
Figure Similarity percentage distribution of the visual modality. The similarity is calculated between the visual segments and the event description. We split the segments into with (blue)/ without (orange) the event. Examples of three events are shown.
COMMUNITY
CO COMUUNTTV
COM UNITY COMMUNITY
JoMoLD
Speech
GT
Violin_fiddee
JoMoLD
LSLD
GT
JoMoLD
LSLD
GT
JoMoLD
Violin_ Fiddle
15LD
Figure 3: Qualitative result on the LLP test The results of method (LSLD) and the state-of- the-art method JoMoLD [6] are shown. "GT" denotes the ground truth annotations.
be given. However, when too big, the performance drops, because most of the labels will remain at without any we-weighting to unreliable segments. A similar observation is found in 3.
Analysis of different re-weighting strategies. As shown in Table 4 (b), four different strategies are compared. The first two rows denote conducting dynamic re-weighting on only segments with without the event, where the results are 0.9% and 0.6% lower than ours segment-level visual metrics, respectively. "stable re-weight" denotes using stable value instead of weighted similarity as the soft label, where find the results are 0.8% and 1% lower than ours in terms of segment-level and event-level. This demonstrates that since segments with higher similarity are more likely to belong the category, re-weighting with dynamic similarity helps better learn the unreliable segments.
In addition, visualize three examples of similarity distribution with/ without event in Fig. 2. We observe that in most cases, such as chicken rooster and frying food, the similarity distribution is well separated, while only small area is taken unreliable segments. However, there special case, speech, where we can hardly recognize whether it happens through the similarity with the language prompt of the class. We suspect the reason is that speech is difficult to recognize in vision. For example, the vision is about a basketball game while speech could only occur in the audio track.
4.4 Qualitative Results
Visualization of video parsing results. compare our method with the SOTA method JoMoLD [6]
by visualizing the audio and visual parsing results in Fig. 3. As for the visual parsing task, the visual track only contains speech on the segments, where JoMoLD recognizes it as the Violin_ Violinfiddle, and our method identifies the speech event and localizes the temporal boundaries correctly. The results prove

---

Car and Speech
0.190
6:228
0.16
Cella and Vislin_nddlle
Speech
IL299
0.118
K:44
6:24
0,25
0:23
Calo
R319
6.181
6.18
other
0.331
0.254
0247
other
JoMoLD
V: LSLD
GT
V: JoMoLDD
LSLD
GT
(a) Video Spoech and Car
(b) Video with Cello and Fiolin fiddle
Baby Janphter Speech
Singing
6:254
0.218
Speech
other
9.192
0.241
0.231
Raby Laughter
6,25
0.37
0.332
Ja
LSLD
GT
V:
JoMoLD
LSLDD
GT
(c) Video with Singing
(d) Video with Baby lorghter and Speech
Figure 4: Visualization of segment-level label denoising on visual track. The similarities between our prompts and segments are presented. Results on JoMoLD [6] and our method are shown. Each example presents three segments of a video.
that our approach is more effective in recognizing events in visual segments. As for the audio parsing task, our LSLD properly identifies the segments with speech, yet JoMoLD only recognizes speech on some of the segments. This demonstrates that LSLD not only works well on the visual track but is also capable of identifying the events that appear on the audio segment well. However, when identifying the Violin_fiddl, both LSLD and JoMoLD only recognize the event in some segments due to the interference of the speech voice, which impels to further refine the method on the audio modality.
Visualization of the denoising procedure. We visualize the segment-level label denoising results of our method and JoMoLD [6] in Fig. 4. We observe that, in all examples, our prompt of the highest similarity captures the appeared events accurately. As shown in Fig. 4 (a), in the last two segments there is no event exists, where JoMoLD still recognizes Car because Car appears in the first segment and JoMoLD can only denoise video-level. In contrast, LSLD correctly assigns labels for all the segments. In Fig. 4 (b), LSLD recognizes different event appearances on the segment level, while JoMoLD fails to do so, which demonstrates the importance of segment-specifee label assignment and the effectiveness of our method. In Fig. 4 (c), LSLD successfully removes Singing on the first segment, yet JoMoLD is unable remove it. A similar observation could be found in Fig. 4 (d).
5 Conclusion and Discussion
In this paper, we address the weakky-suuprvisee audio-visual video parsing task. Instead of focusing on the overall video instance level denoising, propose to assign specific segment-level labels for each video on the audio and visual track. Since each segment label is not predefined but could be any combination of events that occur in the video, introduce the language modality into task and construct language prompts to represent all cases of event appearance. The similarities between each segment and prompts are calculated, where the prompt with the highest similarity is considered the segment-level label. To further alleviate the effect of segment-level label noise, we apply dynamic weighting on visual segments that may be mislabeled. Extensive experiments and qualitative results show that our method outperforms the state-of- the-ter methods on the visual metrics by large margin.
Limitation. Since only the LLP dataset currently has fine-grained (segment-leve and modality annotations and labels for audio-visual understanding, only perform LSLD on the LLP dataset. In future work, more effort will be made to explore the effectiveness of our method on other audio-visual datasets.
Broader impact. Audio- Visual Video Parsing (AVVP) has numerous potential real-life applications, such as professional media production, audiovisual archive management, entertainment, etc. Our

---

method provides new perspective to address AVVP, i.e., through the guidance of natural language, which could benefit from large-scale vision-language models and will inspire future works.
Acknowledgment. This work was supported in part by the National Natural Science Foundation of China under Grant 62001331, in part by the Natural Science Foundation of Hubei Province under Grant 2021 and in part by the Special Fund of Hubei Luojia Laboratory under Grant 220100018.
References
[1] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings the European Conference Computer Vision (ECCV), pages 247-263, 2018.
[2] Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9248-99-89999 0208-925,,222 2019.
[3] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12475-12486, 2021.
[4] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly- supervised audio-visual video parsing. In Proceedings of the European Conference on Computer Vision (ECCV), pages 436-454, 2020.
[5] Yu Wu and Yi Yang. Exploring heterogeneous clues for sepkly-supervised audio-visual video parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1326-1335, 2021.
[6] Haoyue Cheng, Zhaoyang Liu, Hang Zhou, Chen Qian, Wayne Wu, and Limin Wang. Joint- modal label denoising for suaely-supervised audio-visual video parsing. In Proceedings of the European Conference Computer Vision (ECCV), pages 431 431-448 2022.
[7] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8763. 2021.
[8] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Kerg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-aude pretraining with feature fusion and keyword- to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
[9] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Advances in neural information processing systems, 2016.
[10] Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2827-2836, 2016.
[11] Relja Arandjelovic and Andrew Zisserman. Look, listen and leamn In Proceedings of the IEEE international conference on computer vision, pages 609-617, 2017.
[12] Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. Advances in Neural Information Processing Systems, 2018.
[13] Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. Advances Neural Information Processing Systems, pages 9758-9770 2020.
10

---

[14] Pedro Morgado, Yi Li, and Nuno Nvasconcelos. Learning representations from audio-visie spatial alignment. Advances in Neural Information Processing Systems, pages 2020.
[15] Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R Glass. Contrastive audio-visual masked autoencoder. In The Eleventh International Conference on Learning Representations, 2022.
[16] Yan-Bo Bo Lin, Yu-Jhe Li, and Yu-Chiang Frank Wang. Dual-modality seq2seq network for audio- visual event localization, In ICASSP 2019-2019 IEEF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2002-2006, 2019.
[17] Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang. Dual attention matching for audio-visual event localization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6292-6300, 2019.
[18) Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, and Meng Wang. Positive sample propagation along the audio-visual event line. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8436 8444, 2021.
[19] Yan Xia and Zhou Zhao. Cross-modal background suppression for audio-visual event localiza- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19989-19999999 19998. 2022.
[20] Varshanth Rao, Md Ibrahim Khalil, Haoda Li, Peng Dai, and Juwei Lu. Dual perspective network for audio-visual event localization. In Proceedings of the European Conference on Computer Vision (ECCV), pages 689-704, 2022.
[21] Tanvir Mahmud and Diana Mareulescu. Ave-clip: Audioclip- multi-windox temporal transformer for audio visual event localization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5158-5167, 2023.
[22] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker- independent audio-visual model for speech separation. arXiv preprint 1rXiv::804.000 03619, 2018.
[23] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference computer vision (ECCV), pages 570-586, 2018
[24] Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-69, 2020.
[25] Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross- modal consistency. In 202/ IEEF/CVE Conference on Computer Vision and Patlern Recognition (CVPR), pages I 55900-15000, 2021.
[26] Sagnik Majumder, Ziad Al-Halah, and Kristen Grauman. Move2hear: Active audio-visual source separation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 275-285, 2021.
[27] Efthymios Tzinis, Scott Wisdom, Tal Remez, and John R Hershey. Audioscopev2: Audio-visual attention architectures for calibrated open-domain on-screen sound separation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 368-385, 2022.
[28] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358-4366, 2018
[29] Relja Arandjelovic and Andrew Zisserman. Objects that sound. In Proceedings of the European conference on computer vision (ECCV), pages 435-451, 2018.
11

---

[30] Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse fine. In Proceedings of the European Conference on Computer Vision (ECCV), pages 292-308, 2020.
[31] Hanyu Xuan, Zhiliang Wu, Jian Yang, Yan Yan, and Xavier Alameda- A proposal- based paradigm for self-supervised sound source localization in videos. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 1029-1038, 2022.
[32] Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.08744 2020.
[33] Ruohan Gao, Tae- the- Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10457-10467, 2020.
[34] Rameswar Panda, Chun-Fu Richard Chen, Quanfu Fan, Ximeng Sun, Kate Saenko, Aude Oliva, and Rogerio Feris. Adamml: Adaptive multi-modal learning for efficient video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7576-7585, 2021.
[35] Saghir Alfasly, Jian Lu, Chen Xu, and Yuru Zou. Learnable irrelevant modality dropout for multimodal action recognition on modality-specific annotated videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20208-20217, 2022.
[36] Shentong Mo and Yapeng Tian. Multi-modal grouping network for weakly-supervised audio- visual video parsing. In Advances in Neural Information Processing Systems, 2022.
[37] Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data, In Proceedings of the European Conference on Computer Vision (ECCV), pages 67-84, 2016.
[38] Ang Li, Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. Learning visual n-grams from web data. In Proceedings of the IEEE International Conference on Computer Vision, pages 4183-4192, 2017.
[39] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfee100m: The new data in multimedia research. Communications of the ACM, (2):64-7)) 73, 2016.
[40] Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In Proceedings of the European Conference on Computer Vision (ECCV), pages 153-170, 2020.
[41] Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annota- tions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11162-11173, 2021.
[42] Xinlei Chen, Hao Fang. Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 05355 2015.
[43] Manuele Barraco, Marcella Cornia, Silvia Cascianelli, Lorenzo Baraldi, and Rita Cucchiara. The unreasonable effectiveness of clip features for image captioning: an experimental analysis. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 0662-4770, 2022.
[44] Mingkang Tang, Zhanyu Wang, Zhenhua Liu, Fengyun Rao, Dian Li, and Xiu Li, ClipAcaption: Clip for video caption. In Proceedings of the 29th ACM International Conference on Multimedia, pages 4858 4862, 2021.
[45] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
12

---

[46] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Chan- ning Moore, Manoj Plakal, Devin Platt, Rif Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In 20/7 ieee international conference on acoustics, speech and signal processing (icassp), pages 131-135, 2017.
[47] Yan-Bo Yan-Bo Lin, Hung- Yu Tseng, Hsin- Ying Lee, Yen- Yu Lin, and Ming-Hsuaa Yang. Explor- ing cross-video and cross cross- modallty signals for superv-su audio-visual video parsing. Advances in Neural Information Processing Systems, pages 1449-11461] 2021.
[48] Xun Jiang, Xing Xu, Zhiguo Chen, Jingran Zhang, Jingkuan Song, Fumin Shen, Huimin Lu, and Heng Tao Shen. Dhhn: Dual hierarchical hybrid network for weakly-supervised audio-visual video parsing. In Proceedings of the 30th ACM International Conference on Multimedia, pages 719-727, 2022.
[49] Jiashuo Yu, Ying Cheng, Rui- Wei Zhao, Rui Feng. and Yuejie Zhang. Mm- Mm-pyramid: Multi- modal pyramid attentional network for audio-visual event localization and video parsing. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6241-6249, 2022.
[50] Junyu Gao, Mengyuan Chen, and Changsheng Xu. Collecting cross-modal presence-absence evidence for weakly audio-visual event perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18827-18836, 2023.
[51] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450-6459, 2018.
[52] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1199 1929, 2020.
[53] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeles dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780, 2017.
13


Reviews:
[{"review_id": "Ispj5DR5AV", "reviewer": "Reviewer_Q15u", "summary": "The authors have observed that weakly-supervised labels in audio-visual tasks are noisy at the segment-level label. To solve the problem, the authors introduce language as an additional source of information to assign soft labels to each of the 1-second video segments. More specifically, a similarity score is computed between each segment and possible prompts, and segment-level labels are softly assigned by the proposed dynamic re-weighting method. The experimental results show the performance boost with the re-weighing.", "questions": "1. Does ```other``` include the case where the segment is similar to some other labels than class A or class B?", "limitations": "The authors discussed the limitations of the work (e.g. segment-level labels for audio) in Section 5.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is mostly well-written, with details and figures.\n\n2. The idea of dynamic re-weighting for segment-level label noise is interesting. \n\n3. The authors show various experimental results, including the ablation study of different prompts.", "weaknesses": "1. Some manuscripts need to be clarified. \n\n* Section 3.4, especially Equation (4), needs to be clarified. According to Equation (4), $\\tilde{\\mathbf{y}}_t^v[c]$ is already assigned with 1 or 0. Where does the value come from? \n\n* Moreover, there is a discrepancy between the descriptions in Lines 171-179 and the equation. For example, how is *relatively high similarity* applied to the equation?\n\n* Figure 1 (c): Does```w/o Cello``` stand for a a prompt \"w/o Cello\"? \n \n* In Table 1's caption, it is written, \"The last 3 lines are all label denoising methods ...\"\nDo the last 3 lines indicate ```MA+Clip```, ```JoMoLD+Clip```, and ```LSLD (ours)```?\n\n2. Although Section 4 shows multiple experimental results, all of the results are on a single dataset (LLP dataset). Ideally, the proposed approach should be demonstrated on various datasets, especially those with no segment-level labels or longer video clips.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your reviews and address your concerns as follows.\n\n**Q1**: Section 3.4, especially Equation (4), needs to be clarified. According to Equation (4), $\\mathbf{\\widetilde{y}}_t^v[c]=1$is already assigned with 1 or 0. Where does the value come from?\n\n**A1**:  Sorry for the confusion. The value is derived from Sec 3.3, where each visual segment is assigned with segment-level label $\\mathbf{\\widetilde{y}}_t^v$ by segment-level denoising. After denoising, for each probable class $c$, $\\mathbf{\\widetilde{y}}_t^v[c]$ could be either 0 (not exist) or 1 (exist). An example of the label denoising process is illustrated below: \n\nSuppose we have a video containing two events, Violin and Cello, its video-level label is assigned as $\\mathbf{y}^v$['Violin']=1, $\\mathbf{y}^v$['Cello']=1. The labels of each segment are initially the same as the video-level label. To perform label denoising, the prompts are created as [Violin and Cello, Violin, Cello, Other], where ``Other'' means neither Violin nor Cello appears. Then we calculate the similarity between segments and prompts, and take the prompt with the highest similarity as the event that appears in this segment. Suppose for $t$-th segment, the prompt with the highest similarity is Violin, \nwe then adjust $\\mathbf{\\widetilde{y}}_t^v$['Cello'] to 0, while retaining $\\mathbf{\\widetilde{y}}_t^v$['Violin'] at 1. In this way, we obtain the segment-level denoised label $\\mathbf{\\widetilde{y}}_t^v[c]$ for each video segment.\n\n**Q2**: There is a discrepancy between the descriptions in Lines 171-179 and the equation. For example, how is relatively high similarity applied to the equation? \n\n**A2**: We apologize that we didn't describe it clearly. Lines 171-179 describe the procedure of finding unreliable segments, after that we apply dynamic reweighting to these unreliable segments by Eq.(4). \n\nFor positive samples ($\\\\mathbf{\\\\widetilde{y}}_t^v[c] = 1$), the unreliable one is that its prompt similarity score $\\\\mathbf{s}_c$ is lower than the maximum similarity scores of negative data ($Max_w/o$). For negative samples, the unreliable one is that its prompt similarity score $\\\\mathbf{s}_c$ is higher than the minimum similarity scores of positive data ( $Min_w$ ). Indeed the unreliable data are the intersection of the yellow region (negative data) and blue region (positive data) as shown in Fig. 2 of the original manuscript. We then change the labels of unreliable samples by a soft value (proportional to its prompt similarity), rather than the hard values (0 or 1). For those reliable ones, we keep their original label unchanged. \n\nThus the detailed procedure for applying dynamic re-weighting on unreliable segments is illustrated below,\n\n$$\\mathbf{\\hat{y}}_t^v[c]=\n\\begin{cases}\nMin(1,\\alpha \\times \\mathbf{s_c} ) & ,\\mathbf{\\widetilde{y}}_t^v[c] = 1 \\\\ and \\\\  \\mathbf{s_c} \\leq Max_w/o \\\\\\\\\nMin(1,\\beta \\times \\mathbf{s_c})&, \\mathbf{\\widetilde{y}}_t^v[c] = 0 \\\\ and \\\\ \\mathbf{s_c} \\geq Min_w\\\\\\\\\n\\widetilde{y}_t^v[c]& , otherwise\n\\end{cases}$$\n\nwhere $ \\mathbf{s_c}$ is the similarity between the segment and the event, $Min_w$ is the lowest similarity of the segments with the event, $Max_w/o$ is the highest similarity of segments w/o the event, $\\alpha$ and $\\beta$ are two scalars.\n\n**Q3**: Does w/o Cello stand for a prompt \"w/o Cello\"? \n\n**A3**: As illustrated in Q2, w/o Cello stands for segments that are labeled without the Cello event. We will make it clearer in our manuscript. \n\n**Q4**: In Table 1's caption, it is written, \"The last 3 lines are all label denoising methods ...\" Do the last 3 lines indicate MA+Clip, JoMoLD+Clip, and LSLD (ours)?\n\n**A4**: Yes, MA, JoMoLD, and LSLD(Ours) are all label denoising methods. And MA+Clip, JoMoLD+Clip means that we reproduce MA and JoMoLD with extracted features from Clip. \n\n**Q5**: Although Section 4 shows multiple experimental results, all of the results are on a single dataset (LLP dataset). Ideally, the proposed approach should be demonstrated on various datasets, especially those with no segment-level labels or longer video clips.\n\n**A5**: Sorry that we have not carry out experiments on additional datasets. Since LLP is the only dataset that has both segment-level and modality-level labels for test, all state-of-the-art AVVP methods were evaluated exclusively on the LLP dataset. In the future, we will try to conduct experiments on more general datasets.\n\n**Q6**: Does 'other' include the case where the segment is similar to some other labels than class A or class B?\n\n**A6**: Yes, `Other' means that neither class A nor class B appears. "}, {"review_id": "nn7oyCN9jv", "reviewer": "Reviewer_ULdQ", "summary": "This paper proposes LSLD, which leverages CLIP model to denoise unreliable segment-level video labels for audio-visual video parsing. ", "questions": "$-$ Do the baselines with labels denoising (e.g., MA) also leverage CLIP to do denoise? For example, for MA+CLIP baseline, the refined labels are from CLIP or ResNet?", "limitations": "The authors mentioned this in the paper. The proposed method can only work on visual segment-level labels. However, this claim might not be entirely correct. The possible way to do this is mentioned at weakness.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "$+$ The proposed method can achieve state-of-the-art results on LLP datasets in several metrics.\n\n$+$ The improvement of video and audio-visual events is significant.", "weaknesses": "$-$ The proposed method can only denoise labels for visual domains. It will limit the overall accuracy of audio-visual video parsing.  Although the authors mentioned CLAP model is not applicable to the proposed pipeline in L.59-L.60, the audio waveform can be simply blocked out to get segment-level audio information as well.\n\n$-$ Introducing the language model might not be necessary. Since LLP dataset is collected from AudioSet, pre-trained models from AudioSet can find the corresponding labels for LLP in both audio and visual labels. These models may contribute to stronger label-denoising. \n\n$-$ The effectiveness of LSLD is not clear.\n* In L.231, it mentioned that LSLD could also benefit audio accuracy. However, LSLD causes an accuracy drop from 62.4 to 62.3 (Segment-Leve) and from 53.9 to 53.0 (Evet-Level). Since LSLD is based on HAN, which leverages audio-visual feature aggregation, LSLD would be expected to improve audio results as well.\n* Does LSLD also benefit other approaches (e.g., MGN)? The proposed approach should also benefit other baselines including CLIP encoder or Resnet encoder.\n* Is LSLD complementary to other denoising approaches (i.e., MA and JoMoLD)? These approaches also denoise audio labels. They should be complementary to LSLD.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your reviews and address your concerns as follows. And LSLD means only visual denoise and LSLD* means both audio and visual label denoise in the following answers.\n\n**Q1**: The proposed method can only denoise labels for visual domains. \n\nNow we have extended our denoising method to the audio modality, which leads to further improvement compared to the visual-only denoising results (e.g., from 51.3 to 55.7 on the event-level audio accuracy), even with the VGGish feature.\n\nTo be specific, the audio-denoising is based on the audio-text similarity computed by LAION-CLAP [1], which is a large-scale audio-language model pre-trained on Audioset. Same to the denoising process of visual labels (Sec. 3.3), the similarity between prompts and segments is calculated, where the event of the most similar prompt is regarded as the denoised segment-level label. \n|Method||| Segment-Level||||| Event-Level|||\n| :--: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |:-: |\n| | A | V| A-V|Type|Event |A| V |A-V| Type |Event|\n| HAN  |  60.1| 52.9| 48.9 |54.0 |55.4 |51.3| 48.9 |43.0| 47.7 |48.0|\n| LSLD* |  62.7| 67.1| 59.4| 63.1| 62.2| 55.7|64.3| 52.6 |57.6 |55.2|\n|HAN+Clip+Clap| 66.9| 54.3| 50.0| 57.1| 60.2| 59.1| 50.4 |43.9 |51.2 |55.8\n|LSLD*+Clip+Clap| **68.7**| **71.3**| **63.4** |**67.8** |**68.2** |**61.5** |**67.4** |**55.9** |**61.6**| **60.6**|\n\n 'Clip + Clap' means we use the visual feature extracted from CLIP and the audio feature extracted from LAION-CLAP. \n\n**Q2**: Introducing the language model might not be necessary. \n\nThe AudioSet pre-trained model learns from audio-visual correlations, making it more suitable for handling videos whose audio and visual events are temporally. However, in AVVP, audio and visual are assumed to be misaligned, and the model needs to predict different labels for the two modalities. Thus the AudioSet pre-trained model cannot be directly used for label denoising. Also, it cannot be used to generate segment-level labels.\n\nIn contrast, a language model is more general but effective way to denoise the audio and visual tracks individually. Introducing CLIP or CLAP is crucial, since language models are flexible and can describe all events occurring in a video, enabling the generation of segment-level labels. Furthermore, using CLIP makes our method capable of extending to other datasets and tasks, not confined solely to a subset of AudioSet. Moreover, our audio backbone (VGGish) is pre-trained on AudioSet, but using CLIP still leads to better performance upon it.\n\n**Q3**:  LSLD would be expected to improve audio results as well.\n\nSince the original LSLD only performs visual denoising, the performance of audio accuracy is not that highly assured. Although in Table 1 of the manuscript, the audio accuracy is slightly decreased, LSLD indeed improves the audio accuracy upon DHHN[3] and MM-Pyramid[2] as shown in Sec. A of the appendix. Please see the table below.\n|Method||| Segment-Level||||| Event-Level|||\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |:-: |\n| | A | V| A-V|Type|Event |A| V |A-V| Type |Event|\n|  MM-Pyramid  |   60.9 |54.4 |50.0 |55.1 |57.6 |52.7 |51.8 |44.4 |49.9 |50.5|\n| MM-Pyramid + Clip | 62.8 |56.3 |51.6 |56.9 |60.0 |53.1 |52.2 |45.5 |50.3 |50.7|\n|MM-Pyramid + LSLD + Clip| **63.3** |**66.8** |**60.4** |**63.5**| **62.4**|**54.5**|**62.5**|**52.6**|**56.5**| **53.2**|\n|  DHHN  | 61.3| 58.3 |52.9 |57.5 |58.1 |54.0 |55.1 |47.3 |51.5 |51.5|\n| DHHN + Clip |  62.6 |59.1 |53.4 |58.4 |59.1 |54.3| 55.2| 46.8| 52.1| 52.5|\n|DHHN + LSLD + Clip|**64.1**|**69.0**|**60.7**|**64.6**|**64.0**|**56.2**|**66.2**|**53.7**|**58.7**|**56.4**|\n\nIn addition, we also performed LSLD* by extending our method into audio denoising, which leads to significant improvement over audio metrics. Please refer to the table of Q1.\n\n**Q4**: Does LSLD also benefit other approaches? The proposed approach should also benefit the CLIP encoder or Resnet encoder.\n\nIn the original supplementary files, we reported the results of applying LSLD on MM-Pyramid and DHHN. The results are shown in the table of Q3. We can see LSLD clearly improves both architectures. \n\nFollowing your suggestion, we also tested on the ResNet encoder with LSLD*, please see the results of 'LSLD*' from the table of Q1.\n\n**Q5**: Is LSLD complementary to other denoising approaches? \n\nYes, we apply LSLD* to MA and JoMoLD and see both of them obtain significant improvement. All the experiments are conducted with the VGGish feature and Resnet feature for a fair comparison. Please see the table below. \n|Method||| Segment-Level||||| Event-Level|||\n| :-------: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |:-: |\n| | A | V| A-V|Type|Event |A| V |A-V| Type |Event|\n|  MA  |   60.3| 60.0| 55.1| 58.9| 57.9 |53.6| 56.4| 49.0| 53.0| 50.6|\n|MA + LSLD* | **61.2**| **66.6**| **59.0**| **62.3**| **61.1**| **54.3**| **64.0**| **52.5**| **56.9**| **54.2**|\n|  JoMoLD  |  61.3 |63.8| 57.2| 60.8| 59.9| 53.9| 59.9| 49.6| 54.5| 52.5|\n|JoMoLD + LSLD* | **62.3**| **66.1**| **58.7**| **62.4**| **61.8**| **55.8**| **63.6**| **52.2**| **57.2**| **55.0**|\n\n**Q6**: Do the baselines with labels denoising also leverage CLIP to do denoise? For MA+CLIP baseline, the refined labels are from CLIP or ResNet?\n\nNo, MA and JoMoLD don't leverage CLIP to denoise labels. For MA+CLIP, the refined labels are from CLIP visual features, but CLIP similarity (e.g., computing similarity between the image and our constructed prompt) is not used in the denoising process.\n\n[1] Yusong Wu et al. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. \n\n[2] Jiashuo Yu et al. Mm-pyramid: Multimodal pyramid attentional network for audio-visual event localization and video parsing. \n\n[3] Xun Jiang et al. Dhhn: Dual hierarchical hybrid network for weakly-supervised audio-visual video parsing."}, {"review_id": "dbxzUNk3TM", "reviewer": "Reviewer_Nz2w", "summary": "This paper tackles the audio-visual parsing task by dividing the video-level label into segment-level labels with the help of language based on CLIP. Training with the fine-grained segment-level labels, instead of the coarse video-level labels, makes the model perform better. In the process, the processing for noisy and unreliable labels bring obvious improvement on the task.", "questions": "How to generate the prompts? how to obtain A and B? Randomly choose a negative class from 25 categories together with the original class to compose A and B respectively? And the paper says that “the language prompts of the video [could] be xx”. Are there any other formats besides the “other/none/notclass” format in experiment?", "limitations": "One main motivation of this paper it to tackle the problem of unmatching between the audio and visual modality. Then what’s the actual situation in the evaluated data? And it would be better to provide a statistics of this unmatching in this data to show that the data is proper to evaluate this point, together with quantitative comparison of the performance on the data with or without this unmatching. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The presentation is clear and it’s easy to follow.\nThe manner to introduce CLIP to generate segment-level labels to replace video-level labels for training is smart, simple and effective.\n", "weaknesses": "I think the main contribution in this paper is to introduce large models such as CLIP to help the audio-visual video parsing task, and the exploration could be more thorough. The main task is the “audio-visual” video parsing task, then how would the relationship between the two modalities influence the task? \nIn default case, would the audio modality perform better than visual modality? In Table 1,2 and so on, the results show that “A” always perform better than “V” when V has not been helped by CLIP; and Even when introducing CLIP, the “HAN +CLIP” also shows that “A” better than “V”. Does this mean that “A” always perform better than “V” by default? And also compared the results of “A-V” and others with “A”, the improvements are small. If so, enhancing the audio modality may be more effective for this task, while the audio modality in this work is weaken in the process. This is because that there is no proper audio-based large models like CLIP? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your reviews and address your concerns as follows.\n\n**Q1**: The main task is the “audio-visual” video parsing task, then how would the relationship between the two modalities influence the task? \n\n**A1**: Both modalities contribute to the recognition of events in audio-visual parsing. Even audio signals are still useful in visual event prediction. When we treat audio and visual modality separately, i.e., only using audio signals to predict audio events and only using visual frames to predict visual events, we found the segment-level overall metric drops from 48.9 to 43.1, and the event-level overall metric also decreases by 7.5 points, compared to the model with the interaction and cross attention of audio-visual modalities.\n\nIn addition, as can be seen from Table 1 of the manuscript, when we use the Clip visual feature in HAN and MGN, all the metrics including pure audio recognition performance have been improved, which means better visual features could benefit not only visual recognition but also audio recognition.  \n\n**Q2**: In default case, would the audio modality perform better than visual modality? \n\n**A2**: Thanks for the question. Audio doesn't always perform better than visual but indeed has better performance in more cases for audio-visual events. Visual frames usually have occlusion, camera movement, low resolution, and more variance, so the event may be hard to see from the visual modality. In contrast, audio signals are relatively easier to capture and clear to hear, thus they are easier to recognize different events. \n\n**Q3**: Compared the results of “A-V” and others with “A”, the improvements are small. If so, enhancing the audio modality may be more effective for this task, while the audio modality in this work is weaken in the process. This is because that there is no proper audio-based large models like CLIP?\n\n**A3**: Thanks for the question. The audio modality was weakened because we only perform visual-track denoising in the manuscript. To better address your concern, we further conduct experiments with a large-scale audio-language pre-trained model called LAION-CLAP [1] for audio denoising, and observe significant improvement in audio-related metrics. LSLD* means we perform both audio and visual label denoise.  \n|Method||| Segment-Level||||| Event-Level|||\n| :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: |:--: |\n| | A | V| A-V|Type|Event |A| V |A-V| Type |Event|\n| HAN  |  60.1| 52.9| 48.9 |54.0 |55.4 |51.3| 48.9 |43.0| 47.7 |48.0|\n| LSLD* |  62.7| 67.1| 59.4| 63.1| 62.2| 55.7|64.3| 52.6 |57.6 |55.2|\n|HAN+Clip+Clap| 66.9| 54.3| 50.0| 57.1| 60.2| 59.1| 50.4 |43.9 |51.2 |55.8\n|LSLD*+Clip+Clap| **68.7**| **71.3**| **63.4** |**67.8** |**68.2** |**61.5** |**67.4** |**55.9** |**61.6**| **60.6**\n\n**Q4**: How to generate the prompts? how to obtain A and B? Randomly choose a negative class from 25 categories together with the original class to compose A and B respectively? And the paper says that “the language prompts of the video [could] be xx”. Are there any other formats besides the “other/none/notclass” format in experiment?\n\n**A4**: We apologize that we didn't describe it clearly. Here is an example. Suppose the label annotation of a video is that the video contains Violin (denoted as A) and Cello (denoted as B). Then, we construct the prompts for this video as [Violin and Cello, Violin, Cello, Other]. There is no other format used in the paper. We tried to replace \"Other\" with \"None\" and \"Notclass\" and found \"Other\" achieves the best performance.\n\n**Limitations**: One main motivation of this paper is to tackle the problem of unmatching between the audio and visual modality. Then what’s the actual situation in the evaluated data? And it would be better to provide statistics of this unmatching in this data to show that the data is proper to evaluate this point, together with a quantitative comparison of the performance of the data with or without this unmatching.\n\n**A5**: Since the training set only has video-level labels, we investigate the unmatching percentage of videos and segments on the validation set. We discover that the unmatching ratio is 73% for videos and 48% for segments, which indicates that unmatching between the audio and visual modality appears extensively in the dataset. Since the training set does not have segment-level labels, we cannot train the model on the matching and unmatching data respectively. However, existing works have studied audio-visual synchronization [2] [3] and Asynchronous [4] [5] [6] phenomenon.\n\n**References**\n\n[1] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP, 2023. \n\n[2] Naji Khosravan, Shervin Ardeshir, and Rohit Puri. On attention modules for audio-visual synchronization. In CVPR Workshops, pages 25–28, 2019. \n\n[3] Yasheng Sun, Hang Zhou, Ziwei Liu, and Hideki Koike. Speech2talking-face: Inferring and driving a face with synchronized audio-visual representation. In IJCAI, volume 2, page 4, 2021. \n\n[4] Juergen Luettin, Gerasimos Potamianos, and Chalapathy Neti. Asynchronous stream modeling for large vocabulary audio-visual speech recognition. In 2001 ICASSP, pages 169–172, 2001. \n\n[5] Chuang Gan, Yi Gu, Siyuan Zhou, Jeremy Schwartz, Seth Alter, James Traer, Dan Gutfreund, Joshua B Tenenbaum, Josh H McDermott, and Antonio Torralba. Finding fallen objects via asynchronous audio-visual integration. In CVPR, pages 10523–10533, 2022.\n\n[6] Lee, Sangmin, Sungjune Park, and Yong Man Ro. Audio-Visual Mismatch-Aware Video Retrieval via Association and Adjustment. In ECCV, pages 497-514, 2022."}, {"review_id": "Npf521KvzI", "reviewer": "Reviewer_2DCt", "summary": "The goal of this paper is to create a model that can solve the task of audio-visual video parsing in a weakly-supervised way. For this, the authors propose to use the language modality which proves to provide benefits and improve the performance of the proposed model over the previous baseline. The language modality is used to provide additional supervision for the system. The paper provides an extensive comparison with previous baselines. Also, the authors provide extensive ablations showing the importance of each component and some qualitative examples which provide another perspective of how the model behaves.", "questions": "Most of the above questions can be easily fixed and clarified. For now, I think this paper looks good, so my score is weak accept. However, I would encourage the authors to clarify the mathematical equations, as this will make the proposed method easier to be understood. ", "limitations": "The authors discuss the limitations of their work and how the work can be extended.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed system performs better than the previous baselines.\n- Multiple baselines were used to compare against the proposed method.\n- Extensive ablation showing the importance of each component\n- Presence of qualitative results. \n", "weaknesses": "- Line 2: typo “andvisual”->”and visual”\n- Figure 1: last line of the description: the cross product looks different than in the figure. Make them consistent.\n- Line 127: Do the authors also train the Resnnet and VGGish? Or are they pre-trained and then frozen?\n- Equation 1. I have some questions here. What is the shape of w^a_t and w^v_t? Are these scalars or vectors? Moreover, p_t^a/p_t^v is a vector. What will p^a and p^v? Also vectors? If w is a scalar then p^a/p^v will be vectors. If w is a vector, then I would assume there is a dot product between w and p which may be wrong depending on the dimension of w. Is there a dot product in the p^a and p^v? Moreover, if there is a dot product then p^a and p^v will be a single scalar, and not a tensor. However, I think the authors want to obtain p^a which will have the same shape as p^a_t, but it will aggregate the scores from the whole video. I also observed that p^{av} uses a Hadamard product. The first thing would be to clarify the shape of the weights and that would also clarify a lot of things in the equations. Right now, for me, it is very hard to understand what is going on there. \n- Line 154: How are the features per segment obtained? CLIP gives features per frame. Do the authors average the per-frame features from CLIP to obtain per-segment features?\n- Line 179: How is the overlap region chosen? What are the thresholds? It is said before that “...some segments in orange are labeled as without car, their similarity is still high…”. How high and in comparison with what? Thus, provide some details on how it is judged if the similarity is still too high or too low.\n- Line 182: is this t-th segment one of the unreliable segments, or it is just the t-th segment of the video? Make this explicit, as right now it seems that it is just the t-th segment of a video, but I understood that the re-weighting is only applied to the unreliable segments, as mentioned in Line 170. \n- Line 184: What are the values of alpha and beta? \n- Line 191-194. I am a bit confused by these lines. First, I will refer to the figure. In the upper part of Figure 1 (d), it is seen that the method is learning in a supervised way to label segments correctly, based on the labels that are provided by the denoising and the re-weighting mechanism. However, for the lower part of Figure 1 (d), only the VGGish features are used for the MMIL. Is this correct? Shouldn’t both the visual and the audio features be used by MMIL, as in equation (1)? As far as I understand the only thing that changes is the loss of the video modality, which now can provide segment-level supervision. But everything else is staying the same.\n- Line 193: Here it is said that CLIP is used to extract the visual features during training. Does this mean that the authors train HAN but they replace the Resnet with CLIP? If yes, is CLIP trained, or it is frozen?\n- Line 230: typo “leave the label” -> “leave out the label”\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your reviews and address your concerns as follows.\n\n**Q1**: Do the authors also train the Resnet and VGGish? Or are they pre-trained and then frozen?\n\n**A1**: Yes, they are pre-trained and then frozen. The Resnet is pre-trained on ImageNet and VGGish is pre-trained on Audioset.\n\n**Q2**: What is the shape of $\\mathbf{w}^a_t$ and $\\mathbf{w}^v_t$? Are these scalars or vectors? Moreover, $\\mathbf{p}_t^a$ /$\\mathbf{p}_t^v$ is a vector. What is $\\mathbf{p}^a$ and $\\mathbf{p}^v$? Also vectors? Is there a dot product in the $\\mathbf{p}^a$ and $\\mathbf{p}^v$? Moreover, if there is a dot product then $\\mathbf{p}^a$ and $\\mathbf{p}^v$ will be a single scalar and not a tensor.\n\n**A2**: We apologize for not describing this information clearly in the paper. $\\mathbf{w}_t^a,\\mathbf{w}_t^v\\in\\mathbb{R}^{1\\times{C}}$ are all vectors, whose shape is the same as $\\mathbf{p}_t^a$ and $\\mathbf{p}_t^v$. $\\mathbf{p}^a$, $\\mathbf{p}^v$ are the weighted average of $\\mathbf{p}_t^a$, $\\mathbf{p}_t^v$ respectively. So $\\mathbf{p}^a$, $\\mathbf{p}^v$ are also vectors. We will describe these weights and equations more explicitly in the paper. \n\n**Q3**: How are the features per segment obtained? CLIP gives features per frame. Do the authors average the per-frame features from CLIP to obtain per-segment features?\n\n**A3**: Yes, since a segment/second is composed of 8 frames, we average the features of the 8 frames to get the feature of a segment.\n\n**Q4**: How is the overlap region chosen? What are the thresholds? It is said before that “...some segments in orange are labeled as without car, their similarity is still high…”. How high and in comparison with what? Thus, provide some details on how it is judged if the similarity is still too high or too low.\n\n**A4**: We apologize that we didn't describe it clearly. For positive samples ($\\mathbf{\\widetilde{y}}_t^v[c] = 1$), the unreliable one is that its prompt similarity score $\\mathbf{s}_c$ is lower than the maximum similarity scores of negative data ($Max_w/o$). For negative samples, the unreliable one is that its prompt similarity score $\\mathbf{s}_c$ is higher than the minimum similarity scores of positive data ($Min_w$). Indeed the unreliable data are the intersection of the yellow region (negative data) and blue region (positive data) as shown in Fig. 2 of the original manuscript. We then change the labels of unreliable samples by a soft value (proportional to its prompt similarity), rather than the hard values (0 or 1). For those reliable ones, we keep their original label unchanged. Thus the detailed procedure for applying dynamic re-weighting on unreliable segments is illustrated below.\n$$\\mathbf{\\hat{y}}_t^v[c]=\n\\begin{cases}\nMin(1,\\alpha \\times \\mathbf{s_c} ) & ,\\mathbf{\\widetilde{y}}_t^v[c] = 1 \\\\ and \\\\  \\mathbf{s_c} \\leq Max_w/o \\\\\\\\\nMin(1,\\beta \\times \\mathbf{s_c})&, \\mathbf{\\widetilde{y}}_t^v[c] = 0 \\\\ and \\\\ \\mathbf{s_c} \\geq Min_w\\\\\\\\\n\\widetilde{y}_t^v[c]& , otherwise,\n\\end{cases}$$\nwhere $ \\mathbf{s_c}$ is the similarity between the segment and the event, $Min_w$ is the lowest similarity of the segments with the event, $Max_w/o$ is the highest similarity of segments w/o the event, $\\alpha$ and $\\beta$ are two scalars.\n\n**Q5**: Is this t-th segment one of the unreliable segments, or it is just the t-th segment of the video? Make this explicit, as right now it seems that it is just the t-th segment of a video, but I understood that the re-weighting is only applied to the unreliable segments, as mentioned in Line 170.\n\n**A5**: Thanks for pointing out! In the original Eq. (4), t-th segment is one of the unreliable segments. The Eq.(4) will be updated as replied in the question above (Q4). \n\n**Q6**: What are the values of alpha and beta?\n\n**A6**: We set $\\alpha$ as 4 and $\\beta$ as 0.4, because the similarity of segments with the event is supposed to be higher than segments w/o the event. We'll add the value of these two parameters in the implementation details. Besides, the analysis of different $\\alpha$ and $\\beta$ values is shown in Table 4 (a) of the original manuscript.\n\n**Q7**: I am a bit confused by these lines. First, I will refer to the figure. In the upper part of Figure 1 (d), it is seen that the method is learning in a supervised way to label segments correctly, based on the labels that are provided by the denoising and the re-weighting mechanism. However, for the lower part of Figure 1 (d), only the VGGish features are used for the MMIL. Is this correct? Shouldn’t both the visual and the audio features be used by MMIL, as in equation (1)? As far as I understand the only thing that changes is the loss of the video modality, which now can provide segment-level supervision. But everything else is staying the same.\n\n**A7**: Yes, only the VGGish features (i.e. audio features) are used for the MMIL. The reason of using MMIL is to aggregate the segment-level predictions to the video-level features predictions for weakly supervised optimization. Since we generated the segment-level labels for the visual modality, we have already turned the weakly supervised labels into the fully supervised labels.  Thereby we do not need to perform the MMIL mechanism again. Sorry we didn't make it clear in Figure 1, actually we use the BCE (binary cross-entropy) loss for the video modality, and the final loss function is:\n\\begin{equation}\n    \\mathcal{L} = \\mathrm{BCE}(\\mathbf{y}^{av}, \\mathrm{p}^{av}) + \\frac{1}{T}\\sum_{t=1}^{T}\\mathrm{BCE}(\\mathbf{\\hat{y}}_t^v, \\mathbf{p}^v_t) + \\mathrm{BCE}(\\overline{\\mathbf{y}}^a, \\mathbf{p}^a),\n\\end{equation}\n\n**Q8**: Here it is said that CLIP is used to extract the visual features during training. Does this mean that the authors train HAN but they replace the Resnet with CLIP? If yes, is CLIP trained, or it is frozen?\n\n**A8**: Yes, we replace Resnet with CLIP Image Encoder. And CLIP is frozen during training."}]

Author rebuttal:

Meta-review: Initially, all four expert reviewers appreciated the paper’s contribution. They appreciated the simplicity of the approach and the demonstrated improvement over the prior art. The rebuttal addressed the reviewers’ concerns. All four reviewers recommended that the paper is above the bar for acceptance to NeurIPS. This AC agrees with their recommendation. Please take into account all reviewer feedback in the camera-ready version.
Decision: Accept (poster)