Exploring Cross-Video and Cross-Modality Signals for Weakly-Supervised Audio-Visual Video Parsing
Authors: Yan-Bo Lin
Hung-Yu Tseng
Hsin-Ying Lee
Yen-Yu Lin
Ming-Hsuan Yang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Audio-visual Video Parsing
Weakly-supervised Learning
Vision Applications and Systems

Abstract: The audio-visual video parsing task aims to temporally parse a video into audio or visual event categories. However, it is labor intensive to temporally annotate audio and visual events and thus hampers the learning of a parsing model. To this end, we propose to explore additional cross-video and cross-modality supervisory signals to facilitate weakly-supervised audio-visual video parsing. The proposed method exploits both the common and diverse event semantics across videos to identify audio or visual events. In addition, our method explores event co-occurrence across audio, visual, and audio-visual streams. We leverage the explored cross-modality co-occurrence to localize segments of target events while excluding irrelevant ones. The discovered supervisory signals across different videos and modalities can greatly facilitate the training with only video-level annotations. Quantitative and qualitative results demonstrate that the proposed method performs favorably against existing methods on weakly-supervised audio-visual video parsing.

Exploring Cross- Cides and Cross-Modality Signals for Weakly-Supervised Audio-Visual Video Parsing
Yan-Bo Lin
Hung-Yu Yu Tseng²
Hsin-Ying Lee*
Yen-Yu Lin
Ming-Hsuan Yang3.5.6
National Yang Ming Chiao Tung University UNC Chapel Hill
UCC Merced
ASnap Research Google Research Yonsei University
yblindunc. edu htseng edu hlee5@snap.com
lin0cs nctu. edu. tw mhyang@ucmerced.com
Abstract
The audio-visual video parsing task aims to temporally parse a video into audio or visual event categories. However, it is labor-intensive temporally annotate audio and visual events and thus hampers the learning of parsing model. To this end, we propose to explore additional cross-video and cross-modality supervisory signals to facilitate beakkl-suuperiseer audio-visual video parsing. The proposed method exploits both the common and diverse event semantics across videos identify audio or visual events. In addition, method explores event co-occurrence across audio, visual, and audio-visual streams. We leverage the explored moddlityy co-occurrence to localize segments of target events while excluding irrelevant ones. The discovered supervisory signals across different videos and modalities greatly facilitate the training with only video-level annotations. Quantitative and qualitative results demonstrate that the proposed method performs favorably against existing methods weakly- supervss audio-visuu video parsing.
Introduction
Humans perceive multisensory signals via seeing, hearing, touching, etc., and obtain multimodal information while exploring the surrounding environments. Visual and audio signals, the most common modalities, motivate researchers to jointly comprehend audio-visual events (e.g.. see people singing and hear their sounds) [1, 2, 3, 4, 5, 6, 7]. Events visible in images while hearable in audio are referred to audio-visual events. However, learning-based models tend to recognize particular audio-visual event by using the data from the dominant modality with richer information and overlook clues from either audio only or visual only events which still contribute to holistic video understanding. Therefore, the resultant models can generalize well on audio-visual events only instead of comprehensively understanding all kinds of video events. To address this issue, target at audio-visual video parsing [4, 6] where predictions for audio, visual, and audio-visue events with temporal boundaries all required but separately evaluated.
The time-consuming and visual video parsing task. To address this issue, Tian er al. [4] handle this task in weakly-supervised manner given only video-level labels, which indicate events presence without temporal bound- aries and detailed modalities. They develop audio-visual co-attention mechanism to assemble discriminative multimodal representations and use multiple instance learning to aggregate frame-level predictions into video-level ones. However, video-level labels alone cannot identify which modality events are from. Wu et al. [6] then propose to perform label refinement by swapping the audio and visual tracks of different videos to estimate and remove irrelevant event categories for each modality. They further adopt temporal contrastive learning to align audio and visual representations from the same frame. However, the contrastive learning is based on the assumption that audio and
annotation process poses a major challenge for the audio-
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

visual signals are synchronized, which may not hold in practical scenarios with complex events. Furthermore, these methods [4, 6] only consider audio and visual tracks of a single video without exploiting the relationship across categories and videos, which also provide rich shared semantics regarding event categories.
In this work, we propose to leverage audio and visual data across different videos to explore shared information of each category. For example, videos with singing events may have similar patterns whatever in an audio or a visual modality. By observing all videos in a training batch, we can not only explore the shared semantics among audio-visur data but also exclude unrelated events. In addition to the relationship across different videos, we exploit the dependency between event categories. For example, when people are singing, there is usually music accompaniment. Therefore, we propose to treat audio, visual, and audio-visual streams separately and adopt an audio-visual class co-occurrence module that jointly explores the relationship of different categories among all streams. By measuring the similarity of event categories from audio, visual, and audio- events, the correlated events are more likely to be correctly determined as the presence or absence of event categories. Such a strategy can robustly learn the correlation of categories within/across modality and fully exploit video data. The proposed strategy can be applied to existing methods on video parsing.
We evaluate the proposed method on the LLP [4] dataset. Videos are parsed into audio, visual, and auuio-visuall events under both segment and event levels, and evaluated with F-scores metrics. Both qualitative and quantitative results demonstrate the effectiveness of the proposed method on the audio-visual video parsing task. The main contributions of this work are summarized as follows:
We leverage audio and visual data across different videos and tracks, which can learn common semantics of the same events and discern unrelated clues.
We develop an audio- visual event co-occurrence module that jointly considers the relation- ship of categories in audio, visual, and audio-visual modalities, which can prevent models from differentiating the representations of the related events.
Qualitative and quantitative experimental results on the benchmark dataset demonstrate that the proposed method performs favorably against the state-of saate-of-the-arne the--rtss in various settings.
2 Related Work
Audio- Visual Representation Learning. Implicit correlation between audio and visual data from videos provides rich information for audio-visual representation learning. First, the audio-visual pairs from the same video clip [8, 9, 10, 11, 12, 13, 14, 15, 16, 17] are strongly correlated based on the assumption that audio and visual data from a video are synchronized and highly correlated. Moreover, features extracted from unpaired video clips tend to be more diverse than those from the same clips. Second, by exploring audio-visual temporal synchronization [18, 19], temporal information can be served as a training guidance. Given a video sequence, existing methods [18. 19] distinguish audio and visual features from different frames while correlating features from the same frames. Such an idea enhances robust audio-visual representation learning that essential several tasks such as audio-visual event eecalizatioo/parsnnnerooggg
[1. 2, 3, 4, 5, 6, 7, 20], sound separation [21, 22, 23, 24, 25, 26, 27, 28, 29, 30], audio spatialization [31, 32, 33, 34, 35, 36, 37, 38), and sound localization [39, 40, 41, 42, 43, 44]. Instead of random sampling sound and images, our method selects both related and irreverent videos to explore common semantics and discern dissimilar
events.
Audio-Visun Video Event Localization and Parsing. Audio-visual video parsing aims detect events in videos and identify audio, visual, and audio-visual events seeing the event and hearing its sound) and activities. Videos can be parsed with event categories and boundaries in both audio and visual modalities. Early researches [5, 7, 2, 3] aim to jointly derive audiovisual information in each local segment of the input video for audio-visual event localization, which emphasizes to detect only audio-visual events. However, due to the inconsistent information observed from audio and visual signals, data from either modality with insufficient clues may degrade the performance of prediction. Therefore, the work [7] focuses on audio/visual data with relevant categorical events to tackle this issue. Although methods of this category present favorable results, they are applicable to audio-visua event localization, which considers only synchronous audio-visual events or not. Recently, multi-modal multiple instance learning (MMIL) based methods with hybrid attention [4] carry out superiss audio-visuu video parsing. These methods aggregate segment-level
2

---

Autio
Video Inputs
Lnn
P
Papaees Pizz
Fatuu
Co
Palaree Piz
Coss Hodale Occantzze
[Na Yigi
Fine
Relltad Vidaos
MIP
Shard
Crocs-Mrd MrdallModallty
BRules Rug
Suturtii
Ulansizaed
HLLP
Addio
Uno
Class fustanee
Canoo-Mssdilyy
Festur
Croes Mobiiity CpeDremmeoo
Aggmeattnn
Visui
Figure Algorithmic overview. Our framework consists of a visual feature extractor, an audio feature extractor, feature aggregation module, MMII. pooling, shared cross-modality semantics, and an cross-modee moddlly co-occurrence module. Given n videos of seconds, the visual and audio feature extractors compute their visual and audio features. The feature aggregation module [4] conducts self- and cross modality attention aggregate segment- segment-wise audio fa and visual f" representations. We map segment- wise aggregated features to class-specific features by exploring cross- cross-modality co-occurrence. By performing self- and cross- coos--modality attention for class features, we identify within and cross modalities relationship between classes for event predictions. Note that & denotes matrix multiplication with the softmax operation performing on each row, and the green block only shows the example for segment-wise visual prediction at time f. We also leverage the aggregated features of all videos figure common semantics regrading events by maximizing the similarities between related videos while minimizing those between unrelated videos with Eq. 8, The MMIL Pooling [4] an attention-baset pooling function that aggregates segment-wise results produce video-level ones, which are optimized by the binary cross entropy loss described in Eq. 3 and Eq. 6.
predictions into video-level ones, with which optimizing model by using video- weak labels is enabled. Since video-level labels are typically insufficient to identify either audio or visual events, Wu et al. [6] generate pscudo labels for each modality by exchanging audio and visual tracks between unrelated videos. However, we notice that videos with replaced sounds or images may share some common semantics. Our method can exploit videos in training batch extract their common semantics for a categorical event and discern unrelated clues. Furthermore, we can leverage the relationship between event classes find out related events (e.g., singing may accompany music).
3 Proposed Method
In this paper, we propose a novel framework for weakly-supervised audio-visual video parsing. In order to explore common semantics across videos and dependency across event categories, the proposed model leverages all audio and visual signals across videos in a training batch and the correlation between classes for each training instance. Section 3. we first define the notations and settings considered in this paper and revisit the common backbone [4, 6] for weakly-supervised audio-visual video parsing, which consists of feature aggregation and multi-modal multiple instance learning (MMIL) pooling. Then in Section 3.2 and Section 3.3, we detail the modules we propose to capture dependency across different events and information across different videos, respectively.
3.1 Preliminaries
Problem Formulation and Notations. Given video sequence with T seconds long, we obtain T non-overlapping audio and visual segments where each segment is one-second long. Models are aiming to predict the event labels for each segment, which may contain several or no events. time t, there three targets for audio, visual, and audio-visun events: y" e R 1 ,y" e glxc

---

and yt" E IR 1x are multi-class event label with C' event categories, yf, Ył, and yt denote audio, visual, and audio-visual event labels, respectively. We note that detailed annotations (e.g., yř. y:. and y(") are not accessible during training and only available during evaluation. As for training, only video-level annotations are available during training. Video-level annotations only contain action event categories without indicating specific times slots or modalities (e.g., audio and visual event).
Revisit of Suuervy--pe Audio- Visual Video Parsing. The previous method [4] presents promising results with feature aggregation based on transformers and multimodal multiple instance learning (MMIL) pooling. Given video sequence of T frames, we denote audio and visual feature sets by F [f ....... (%) IR and FV = {f[ ...., [ E RT'xd, respectively, where d is the feature dimension. The transformer encoder [45] employed to aggregate both muthitymdlality and cross-modality information using multi-head attention blocks:
(fr,....
F
s F F Fa) Softmax
F
(I)
f F
Ccoosa F", F*) = Softmax(
vd
where &eets(-) (-) and (-roas(-) are self-attention and cross-modality attention functions respectively. They perform dot- hoo-product on features across time stamps by using non-shared MLPs. Then the jointly aggregated representations are described as follows:
fe f + Deels (F. F Fa) Pcross (f.. F", F",F". FF)),
f." f." + dsels (f,', F", F") + Φcross (f. F FF),
With the aggregated audio and visual features fo and f.", we can obtain the Trame-wist event prediction P E R. RIIC and P RIxC and the attention weights computed by MLPs and normalized by softmax function for audio, visual, and audio-visual streams (i.e., w e RixC' wi E RIxC and wi E R2xC), Then the video-level prediction is gathered with the MMIL pooling:
T
pd Σ WEP:
p Σ W. and p TM Σ
w au [0 WOD + "["[[]](p"
(3)
The model can then be optimized using the binary cross-entropy loss function between p and vidco-level weak label y 1xC which does not indicate time boundaries and modalities for events.
3.2 Cross-Modality Co- Co-Oceurrence
Videos with multi-label events contain rich information among event categories because the related events are likely to present the same time. The correlation is useful for models to robustly predict the presence or absence of events.
Similar to [46), explicitly model the relationship between event categories in different modalities. we first obtain the representations for each class and then measure the correlation. We note that the class relationships may be different in audio and visual modalities. That is why the work [46] cannot be directly applied to audio-visual video parsing since audio or visual events can be partially single frame. Thus, jointly understanding the class relationship within modality and across two modalities can benefit the audio-visual video parsing task.
or jointly presented
In order to map the frame-wise audio and visual features into class-level ones, the nonlinear transfor- mation with MLPs is formulated as follows:
at.c ReL.U + bg),
Vt.c = ReLU(f + b"),
where ag, and Vz,c are audio and visual class-level features for class c at time with dimension I x d,, respectively. The weights and biases for class e for audio and visual features are denoted as M M.¹ E Rddd, and be. by E Rixd, With class-level representations, we can further model the relationship between event categories within and across modalities by self-attention and cross- modality co-attention mechanism:

---

atc at,c + Pacty(at,cs A1, Ar) ) + Peroso Vo Vt), Vr. = Vt.c + Psels v,, Vi) + Pcross (V Ar),
(5)
where A; {ar,1. a..c) and V, (Vi,1,.... Vt,c) are sets of audio and visual class features at time t. âr. and Vt,e are now co-occcurence features that consider the relationships between categories within and across modalities. We can then predict the probability for each event at time t by MLPs and aggregate every segment-wise predictions into video-level ones i.e.,
pº = ((MLPa((@r..... p" Th MMILL(P). ................................ ................................ P7)), (Pi.....PP7) pT.))
()), P = (MLP.
vi.c)),,
(6)
where the sigmoid function, and M-) is the multi-modal multiple instance learning pooling described in Eq. 3 taking all segment -iise predictions as inputs. The video-level prediction can be optimized by the binary cross-entropy loss function with a video-level weak label y.
3.3 Shared Cross-Modality Semanties across Videos
The information across different videos provides rich supervisory signals that benefit the training of weakly- seakky-superviied mudio-visual video parsing. By observing videos in training batch, can discover both the common and diverse event semantics. With video-level labels, we can initially associate related and irrelevant videos. In order to obtain discriminative categorical representation, we would like to encourage audio and visual representations from related events to be similar and differentiate those from irrelevant videos. However, targeting at segment wise representations with specific events is difficult due to the lack of temporal annotations. Therefore, we seek event-related frames through the weights from MMIL pooling in Eq.
T exp(ely @ wP))
T
exp(gly ww))
f" =
20
(7)
[=1 E explgly O ow))
Er= exp(e(y O w)))
1=1
where @ and g()) are element-wise dot product and summation function over all elements respectively.
With video-level labels and features (f" and f"), adopt contrastive learning [47, 48, 49] to encourage features across modalities with the same event category (at least one) to be close and those with different events to be far away from each other. We leverage all videos in a batch to explore diverse semanties, where the sets of audio and visual features are denoted as (f@))-... fin)) and
( .... ..,,) respectively with video-level labels (F(o),...) F(.)) The relationship across videos can be optimized by the proposed training objective follows:
2
Σ fly; 8j) exp(x, (/g//T)
Lcontrast
log
(8)
Σj= exp(()) /T)
where f()) clipping function that clips values over 1, and T denotes hyper-parameter controlling the temperature. Thus, the proposed method can be optimized by joint the binary cross-entropy loss mentioned in Section 3.1 and the contrastive learning loss in Eq. 8. Our training strategy exploit cross-modality information across videos and event categories to understand common semantics while ignoring irrelevant ones.
4 Experimental Results
Datasets. We use the Look, Listen and Parse (LLP) Dataset [4] for all experiments. The LLP dataset consists of 11, 849 10-seconds video clips annotated with 25 event categories. It covers various real-life scenes such as speech, music performances, car, cheering, dog, etc. Particularly, there 7202 video clips labeled with with only video-level event annotations for model training. The detailed annotations (e.g., individual audio and visual events per second) are available for the remaining 1849 validation and test videos. For all experiments, we use the official data splits from the LLP dataset.
than one event category. We use the 10000 video clips
Evaluation Metries. Following previous work [4, 6). we adopt F-scores as the evaluation metrics. Note that all types of events (audio, visual, and vidiol)isual) are measured under both segment- level and event-level metries. The segment-level metries evaluate snippet-wise prediction
5

---

Table 1: Quantitative results of weakly-supervised audio-visual video parsing. We evaluate all methods on the LLP dataset [4] with F-scores in five different event types and two kinds of segments. The first row indicates five different event types (audio, visual, audio-visual, Type @ AV, and Event @ AV). In the second row, two kinds of segments are shown: Seg. and Event are segment-
level and event-level; and indicates only label refinement is utilized for fair comparisons.
Audio
Visual
Audio-visual
Type AV
Event @ AV
Method
Seg.
Event
Seg.
Event
Seg.
Event
Seg.
Event
Seg.
Event
AVE [5]
47.2
40.4
37.1
34.7
35,4
31.6
39.9
35.5
41.6
36.5
AVSDN [2]
47.8
34.1
52.0
46.3
37.1
26.5
45.7
35.6
50.8
37.7
AVSDN + Ours
48.3
41.2
52.4
48.5
46.9
40.0
49.2
43.2
53.2
40.1
HAN [4]
60.1
51.3
52.9
48.9
48.9
43.0
54.0
47.7
55.4
48.0
HAN + Ours
59.2
51.3
59.9
55.5
53.4
46.2
57.5
51.0
58.1
49.7
MA [6]
60.3
53.6
60.0
56.4
55.1
49.0
58.9
$3.0
$7.9
50.6
MA
59.8
52.1
57.5
54.4
52.6
45.8
56.6
50.8
56.6
49.4
MA + Ours
60.8
53.8
63.5
58.9
57.0
49.5
60.5
54.0
59.5
52.1
results. As for the event-level metrics, the clips are extracted by concatenating positive consecutive segments in the same events. Then, compute the event-level F-scores with mloU 0.5 as the threshold. Furthermore, the overall Type@AY AV performance on audio-visus scene is also considered by computing the averaged audio, visual, and vusio-visual event evaluation results. Instead of directly averaging results from different event types, Event @ AV considers all audio and visual event categories for each sample.
Implementation Details. We implement the proposed method using Py PyTochh [50], and conduct the training and evaluation processes
single NVIDIA GTX 1080 Ti GPU with 11 GB memory.
Following [4, 6]. we use the same visual and audio encoders for fair comparisons. We adopt both ResNet-152 [51] pre-trained on ImageNet [52] and 3D ResNet [53] pre-trained
Kinetics-400 [54] as visual feature extractors. Visual frames are sampled at 8 fps and their 2D and 3D visual features concatenated and then processed by MI. the segment-wise representations. As for audio data, we utilize VGGish [55] pre-trained on AudioSet [56] to extract (28-dimensional audio features. The code and models are publicly available.
extracted. The 2D and 3D) visual features
Evaluated methods. We compare the proposed method based on several baselines to the following waukly-unsupervisce approaches to the audio-visual video parsing task:
AVE consists of an audio-guided co-attention mechanism to adaptively learn the sounding regions. We note that AVE [5] deals with the audio-visun event localization task. Thus, we follow [4] and add additional audio and visual parsing branches for the weakly- supervis audio-visual video parsing task as a baseline.
AVSDN [2] sequence-to-see -o-seuuence- based model to integrate global audio and visual features to local ones. Since AVSDN [2] also deals with the audio-visual event localization task, we make the same modifications to AVSDN as those to AVE.
HAN [4] is multi-modal multiple instance learning- based method with hybrid attention network.
MA [6] reports the state-of-the the-art performance on the weakly-supervised audio-visual video parsing task. a method based on HAN with the label refinement and the audio-visual contrastive learning differentiating temporal segments.
4.1 Quantitative Evaluation
Table 1 shows the quantitative comparisons on the LLP dataset [4]. The proposed method performs favorably against the competing approaches on the -upervisd audio-visual video parsing task. Since our method can be easily extended to existing methods, we extend the proposed on three baselines. The third, fifth, and last rows Table 1 indicate that the proposed method generally benefits three baselines on several metrics of the audio-visual video parsing task by large margin. We note that MA * [6] only utilizes label refinement to refine labels for each modality, and temporal difference andio-visual contrastive learning [6] is not implemented.

---

Table 2: Ablation study. We investigate the effect of using different design components in the proposed method. We show how proposed cross-modalin co-occurrence (CM-Co) in Section 3.3 and shared cross-modality semantics across videos (CM-S) module in Section 3.2 improve the baselines.
Audio
Visual
Audio-visure
Type AV
Event @ AV
Method
Seg.
Event
Seg.
Event
Seg.
Event
Seg.
Event
Seg.
Event
HAN [4]
60.1
51.3
52.9
48.9
48.9
43.0
54.0
47.7
55.4
48.0
HAN + CM-S
58.1
49.6
58.3
53.6
53.2
46.3
56.5
49.8
55.9
47.5
HAN + CM-Co
59.7
51.4
57.4
52.4
51.9
44.2
56.3
49.3
57.4
48.5
HAN + Ours
59.2
51.3
59.9
55.5
53.4
46.2
57.5
51.0
58.1
49.7
MA [28]
60.3
53.6
60.0
56.4
55.1
49.0
58.9
53.0
57.9
50.6
MA CM-Co
61.1
53.3
61.7
57.3
56.3
49.0
59.7
53.0
58.9
51.2
MA
59.8
52.1
57.5
54.4
52.6
45.8
56.6
50.8
56.6
49.4
MA " + CM-S
60.4
53.5
60.7
56.5
55.8
47.5
58.9
52.5
58.6
51.0
MA * CM-Co
60.5
53.6
61.3
56.5
54.9
46.7
58.9
52.3
59.1
51.4
MA* + Ours
60.8
53.8
63.5
58.9
57.0
49.5
60.5
54.0
59.5
52.1
We notice that our method significantly improves baselines in the metrics of visual, audio-visual, Type AV, and Event AV. By observing the class distribution training sets, find that 31% 7%, and 9% training videos contain speech, singing, and violin events. These events are more likely to present in the audio modality. Therefore, the video-level labels would limit the performance regarding visual events. The proposed method can leverage additional cross-video and cross-modality supervisory signals explore common semantics, which can improve results in vision-related metrics.
4.2 Ablation Study
Cross-Modality Co-Occurrence and Semantics across Video. We conduct the ablation study to analyze the individual impact of each developed component in the proposed method. The results are presented in Table 2. CM-Co represents the usage the cross-modality co-occurrence module described in Section 3.2, which leverages the relationship between categories within and cross modalities. CM-S indicates the shared cross-modality semantics across videos module described in Section 3.3, which considers all audio and visual information across videos in batch.
In Table 2, we note that both CM-S and CM-Co can improve basclines several metrics. By exploring common semantics among training videos (CM-S), we improve the performance visual and oudio-visual evaluation by a large margin. Such a strategy can exploit additional information from videos to address the potential drawback of video-level labels described in Section 4.1. Furthermore, the proposed cross-modality no-occurrence module (CM-Co) also presents favorable results. We note that the significant improvement Event AV evaluation with the usage of CM-Co can verify the efficacy of considering the relationship between categories within and across modalities. Since Event AV considers all audio and visual events for the F-score (e.g., truth positive from both audio and visual events), the improvement Event AV indicates our mrods-modalltyy co-occurrence can perform well on video parsing when events present in an audio visual modality.
In the second group of the evaluated methods in Table 2, we verify the proposed CM-S works better than the contrastive learning method in MA. We perform our CM-S on the MA model. The CM-S exploits information across different videos to address the issue that audio and visual tracks may not be synchronized. Instead, the contrastive learning method in MA is developed based on the assumption synchronization to associate the audio-vivne visual representation in a single video. Since CM-S learns diverse and common semantics, is effective and complementary to the contrastive learning approach in MA performing on a single video. We note that our CM-S generally improves the performance over all segment-level metrics, which supports our claim.
Self-attention and Cross- Modaliyy Co-attention in Co-Occurrence. Since our cross-modality co-occurrence module exploits self-attention among class-level features in the same modality and cross-modality co-attention on cross-modality class-level representations to model the relationship between categories in the same and different modalities. Taking class-level audio features in Eq. 5 as
7

---

Table 3: Ablation study. We investigate the effect of different developed mechanisms in the proposed cross-modality co-occurrence (CM-Co) module in Section 3.2. In Eq. 5, class-level features are processed by self-attention and cross-modality co-attention mechanisms. A Only and V Only indicate only self-attention performs for individual audio and visual events respectively. AV denotes performing self-attention for audio and visual events. CM-Co is the proposed method that considers relationship between categories within and cross modalities by both self-attention and cross-modality co-attention mechanisms.
Audio
Visual Event
Audio-visual
Type@ AV
Event @ AV
Method
Seg.
Event
Seg.
Seg.
Event
Seg.
Event
Seg.
Event
HAN [4]
60.1
51.3
52.9
48.9
48.9
43.0
54.0
47.7
55.4
48.0
HAN + A Only
60.5
52.3
49.8
43.9
45.6
38.3
52.0
44.8
55.7
45.9
HAN + V Only
56.1
44.5
56.8
53.2
49.7
40.7
54.2
46.1
54.1
44.6
HAN + AV
59.5
50.3
55.1
50.5
48.6
40.3
54.4
47.0
56.0
47.4
HAN + CM-Co
59.7
51.4
57.4
52.4
51.9
44.2
56.3
49.3
57.4
48.5
MA [6]
59.8
52.1
57.5
54.4
52.6
45.8
56.6
50.8
56.6
49.4
MA + A Only
60.7
52.7
53.9
47.9
50.1
42.2
54.9
47.6
57.0
47.1
MA R Only
46.8
34.4
60.8
57.0
42.8
31.1
50.1
40.9
52.6
40.4
MA + AV
58.3
50.4
59.4
55.2
53.9
46.9
57.2
50.8
56.7
48.5
MA CM-Co
60.5
53.6
61.3
56.5
54.9
46.7
58.9
52.3
59.1
51.4
Table 4: Ablation study. We evaluate the proposed method in accuracy, efficiency, and model sizes. We show the numbers of parameters and FLOPs for the proposed cross cooss-modllity co-occurrence (CM-Co) and HAN [4] with a few layers.
Note that the results are all in the segment level.
Method
Audio
Visual
Audio-visual
Type@AV AV
Event AV
GFLOPs
Params
HAN Layer
60.1
52.9
48.9
54.0
6.63
2.4M
HAN 2 Layers
58.2
55.4
50.6
54.7
54.9
7.28
2.9M
HAN 3 Layers
58.1
55.2
50.3
54.5
54.6
7.97
3.5M
HAN CM-Co
59.7
57.4
51.9
56.3
57.4
6.99
2.8M
example, the class-level self-attention and cross-modality co-attention atin(ar,es A,, Ar) and attn(a,,.> V,, Ve), respectively.
Table 3 presents the results in various modifications of the cross-modality co-occurrence module. We note that the design co-occurrence the same and cross modalities can generally improve the results in several metries. We also evaluate the co-occurrence module in single modality. The results are shown the second, third, seventh, and eighth rows in Table 3, where Only and V Only indicate the co-oceurrence module only leverages the relationship between categories in audio or visual data respectively. As the results shown in the second and seventh rows, training with co-occurrence in audio events only (i.e., Only) events. Similarly, considering visual event only (i.e., V Only) can benefit the results regarding visual events. Furthermore, the co-occurrence for both audio and visual categories (AV) in the fourth and ninth rows can contribute to the results in general metrics such as Type @ AV and Event @ AV. We then further consider the correlation between events across modalities. That is the cross-modality co-occurrence module (CM-Co) in the fifth and tenth rows. The results can confirm the efficacy of the proposed eross cross- modalityy co-occurrence module in all metrics except segment-leve audio events caused by similar reasons discussed in Section 4.1.
slightly improve the performance
audio
Model Capacity. Since our cross-modality co-occurrence module leverages class-level representa- tions, it would increase the capability models on capturing information. For fair comparisons, we add extra parameters to HAN [4] analyze whether parameters contribute performance gain. Specifically, we increase the number layers in its transformer-based feature aggregation to and 3, respectively.
In Table 4, we report the results in accuracy, computational costs, model sizes. The first three rows show the performance of HAN with different numbers of layers. We note that HAN with one

---

GT Va
Violin
Cali
GT A:
Prdd V
Vioiin
Calo
Pod A
Calo
MA" 4Ours
Ped V:
Calo
Pred A:
Callo
MA
Figure 2: Qualitative comparisons. We compare the proposed method with the state-of satt--of-thh-arr
weakly-supervised audio-visuse video parsing method on the LLP dataset [4]. The frame-wiss annotations are shown in gray and purple bars. The gray bar denotes visual events, and the purple bar represents audio events. GT and GT are the ground-truth visual and audio events respectively. Our results are shown in the green block, and the results by the competing method, MA [6], are present in the blue block.
MA' Ours
MA'
Figure 3: Audio feature distribution by using t-SNE. The upper figure shows the distribution by our method. The lower figure presents that by MA" The legend lists all event combinations.
extra layer has more parameters than the proposed co-occurrence module. However, the results of HAN with extra layers indicate that using more parameters/layers for HAN does not improve the performance. The proposed cross-modality co-occurrence module enhances HAN more effectively.
4.3 Qualitative Evaluation
Qualitative Results. We present the qualitative results of the evaluated methods in Figure 2. GT V and GT show the ground-truth annotations for visual and audio events, respectively. Pred and Pred_A A present the predictions made by our method and the state-of-the-art competing method, MA [6], respectively. Our results are shown the green block, while the results MA are present in the blue block. In general, method presents more accurate predictions in both audio and visual

---

Dra Soutin
you Moth CLP0
+uurs
MA'
Figure 4: Visual feature distribution by using t-SNE. The upper figure shows the distribution by our method. The lower figure presents that by MA" The legend lists all event combinations.
events than MA. We note that the whole violin is shown after 7 seconds. That would hamper models for understanding visual events e.g., MA predicts wrong results on violin visual events before seconds. Since our method leverages the relationship between categories, it can still predict correct
temporal boundaries for guitar events by jointly considering cello events the videos.
Feature Distribution Visualized by I-SNE. We apply t-SNE to the aggregated audio and visual features from each segment described in Eq. 2. The visualization results are present in Figure and Figure 4, respectively. The legends list all the combinations of multiple labels. For example, in Figure 3, audio events of singing violin are shown as purple spots. We note that the related events including multiple events are shown in similar colors. In Figure 4, the proposed method achieves better performance in the sense that similar color spots are closer than the spots in MA*
present blue spots, and the mixed sounds of singing and
Conclusions
In this paper, present novel audio-visual video parsing framework in weakly-supervised manner that can be applied to existing methods. We propose two modules to exploit the relationship across videos, modalities, and event categories, and explore additional supervisory signals that can benefit audio-visual video parsing. The shared cross-modalin semantics module leverages common and diverse event semantics across videos learn robust cross-modality representations that facilitate models to identify audio, visual, and audio-visual events. Furthermore, the cross-modality co-occurrence module aims learn the relationship between event categories. It helps localize segments of target events and can exclude irrelevant ones by performing self-attention and cross- modality co-attention on class-wise features, Extensive experimental results show that our approach substantially improves several baselines and performs favorably against the kate-of-the-ee art methods.
Acknowledgments. This work was supported in part by the Ministry of Science and Technology under grants 109- 2221-E-009-1 13-MM, 1002628-E-A49-00 also funded in part by Qualcomm through Taiwan University Research Collaboration Project, the Higher Education Sprout Project the National Yang Ming Chiao Tung University, and Ministry of Education.
and 100-2644F000-0155 was
10

---

References
[1] Jun-Tae Lee, Mihir Jain, Hyoungwoo Park, and Sungrack Yun. Cross- attssttattntionll audio-visual fusion for weakly-supervised action localization. In ICLR, 2021. 1.2
[2] Yan-Bo Lin, Yu-Jhe Li, and Yu Yu-Chiang Frank Wang. Dual-modality seq2seq network for audio-visual event localization. In ICASSP, 2019. 1, 1,
[3] Yan-Bo Lin and Yu-Chiang Frank Wang. Audiovisual transformer with instance attention for audio-visual event localization. In ACCV, 2020. 1,2
[4] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly- supervised audio-visual video parsing. In ECCV, 2020. 1, 2, 3, 4, 5, 6, 7, 8, 9
[5] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In ECCV, 2018. 2,
[6] Yu Wu and Yi Yang. Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In CVPR, 2021. 1, 2. 3, 5, 6, 8, 9
[7] Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang. Dual attention matching for audio-visual event localization. In ICCV, 2019. 1,2
[8] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV, 2017. 2
[9] Relja Arandjelović and Andrew Zisserman. Objects that sound. In ECCV, 2018. 2
[10] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In NeurlPS, 2016. 2
[11] Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In ECCV, 2016.
[12] Jean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapu- ram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. In NeurIPS, 2020. 2
[13] Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- supervised learning by cross
cross-modal modal audio-video clustering. In NeurIPS, 2020. 2
[14] Yuki M Asano, Mandela Patrick, Christian Rupprecht, and Andrea Vedaldi. Labelling unlabelled videos from scratch with multi-modal self-supervision. In NeurIPS, 2020. 2
[15] Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. Active contrastive learning of audio-visual video representations. In ICLR, 2021. 2
[16] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination
with cross-modal agreement. In CVPR, 2021. 2
[17] Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimina- tion. In CVPR, 2021. 2
[18] Andrew Owens and Alexei A. Efros. Audio-visual scene analysis with self-supervised multisen- sory features. In ECCV, 2018. 2
[19] Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. NeurlPS, 2018.
[20] Jinxing Zhou, Liang Zheng. Yiran Zhong, Shijie Hao, and Meng Wang. Positive sample propagation along the audio-visual event line. In CVPR, 2021. 2
[21] Chuang Gan, Deng Huang, Hang Zhao, Joshua B Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. CVPR, 2020.
[22] Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In ECCV, 2018. 2

---

[23] Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. In ICCV, 2019. 2
[24]
Xudong Xu, Bo Dai, and Dahua Lin. Recursive visual sound separation using minus-plus net. In ICCV, 2019. 2
[25] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In ICCV,
2019. 2
[26] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In ECCV, 2018. 2
[27] Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Dan Ellis, and John R. Hershey. Into the wild with audioscope: Unsupervised audio-visual separation of on-screen sounds. In ICIR, 2021. 2
[28] Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. In CVPR, 2021. 2,7
[29] Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross- modal consistency. In CVPR, 2021. 2
[30] Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. In CVPR, 2020. 2
[31] Ruohan Gao and Kristen Grauman. 2.5d sodd-d.uull-ounnd In CVPR, 2019. 2
[32] Pedro Morgado, Yi Li, and Nuno Vasconcelos. Learning representations from mudiolvisual spatial alignment. In NeurIPS, 2020. 2
[33] Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video. In NeurIPS, 2018. 2
[34] Karren Yang, Bryan Russell, and Justin Salamon. Telling left from right: Learning spatial correspondence of sight and sound. In CVPR, 2020.
[35] Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In ECCV, 2020.
[36] Yan-Bo Lin and Yu-Chiang Frank Wang. Exploiting audio-visual consistency with partial supervision for spatial audio generation. AAAI, 2021.
[37] Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural audio generation without binaural audios. In CVPR, 2021. 2
[38] Yu-Ding Lu, Hsin-Ying Lee, Hung- Yu Tseng, and Ming- Hsuan Yang. Self-supervieed audio spatialization with correspondence classifier. ICIP, 2019.
[39] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self- supervised learning of audio-visual objects from video. In ECCV, 2020. 2
[40] Arda Senocak, Tae- Mhy- Oh, Junsik Kim, Ming Hsuan Yang, and So Kweon. Learning to localize sound source in visual scenes. In CVPR, 2018. 2
[41] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound sources visual scenes: Analysis and applications. TPAMI, 2019.
[42] Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In ECCV, 2020. 2
[43] Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. In CVPR, 2019. 2
[44] Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Discriminative sounding objects localization via self-supervised audiovisual matching. In NewrIPS, 2020. 2
12

---

[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017. 4
[46] Praveen Tirupattur, Kevin Duarte, Yogesh Rawat, and Mubarak Shah. Modeling multi-label action dependencies for temporal action localization. In CVPR, 2021. 4
[47] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv Preprint, 2018. 5
[48] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018. 5
[49] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 5
[50]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019. 6
[51] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[52] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009. 6
[53] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018. 6
[54] João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. CVPR, 2017.
[55] Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron Weiss, and Kevin Wilson. Can architectures for large-scale audio classification. In ICASSP, 2017.
[56] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labed labeled dataset for audio events. In ICASSP, 2017. 6
13


Reviews:
[{"review_id": "jcHrwm5LwwE", "reviewer": "Reviewer_nrEQ", "summary": "The authors introduce an interesting audio-visual video parsing framework method in a weakly-supervised manner, which contains two part of *the shared cross-modality semantics module* and *the cross-modality co-occurrence module*. Specifically, the former leverages common and diverse event semantics across videos to learn robust cross-modality representations that facilitate models to identify differently modalities’ events. And the latter joint considers the relationship of event categories, which helps localize segments of target events and can exclude irrelevant ones by performing self-attention and cross modality co-attention on class-wise features. Additionally, the experiments conducted on the LLP dataset demonstrate the proposed model obtained the most advanced results in downstream classification with various settings.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This audio-visual video parsing(AVVP) task proposed by Tian et al[a] in 2020, which aims at providing a detailed analysis of audio, visual, and audio-visual events in videos without such alignment assumptions. The method in this paper is unique, but it is more like a combination of existing technologies[a][b]. Practically, relative to existing random sampling sound and images, the proposed method selects both related and irreverent videos to explore common semantics and discern dissimilar events. And the method can exploit videos in a training batch to extract their common semantics for a categorical event and discern unrelated clues. In addition, most of the related work are cited.\n\nQuality: The submission sounds technically, and it's well supported by experimental results, and the methods used are appropriate. This is a complete piece of work，and the authors are careful and honest about evaluating both the strengths and weaknesses of their work\n\nClarity: The writing and organization of the submission is relatively clear, but some content and existing methods have too much overlap, mainly reflected in *Revisit of Weakly-Supervised Audio-Visual Video Parsing*.\n\nSignificance: The experimental results have a positive effect on related downstream tasks, and others researchers likely to use the ideas or build on them. The submission provide a new exploration for solving the audio-visual video parsing task, and to a certain extent solve the task better than the previous work. And the current model is the state-of-the-art by the experimental results, and some ablation experiments have been used to prove it.\n\n[a] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly supervised audio-visual video parsing. In *ECCV*, 2020.\n[b] Yu Wu and Yi Yang. Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In *CVPR*, 2021\n\nPost Rebuttal:\n\nThe authors' response has partially addressed my previous concerns, such as the difference between this work and existing technologies [4, 6], as well as some method details. Overall, I think this is an incremental work on Weakly-Supervised Audio-Visual Video Parsing, where the main contribution is as what the authors emphasize, \"the conventional contrastive learning is conducted over frame pairs of the same video, while contrastive learning in Eq. 8 of this work is applied to video pairs\" in the weakly-supervised learning scheme. The proposed methods is somewhat similar to the previous ones, except that they utilize the video-pairs annotated by video-level annotations. Based on the above considerations, I decide to hold my previous rating of 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper still has some limitations, which are mainly reflected as follows:\n\n- How to ensure that at least two of the n videos in each batch are related? Have you considered the extreme case where all videos in a batch are irrelevant?\n- Since the video is weakly labeled, it is impossible to determine the specific moment when the labeled event occurs. At each time $t$ of the class-level operation, the corresponding label may not occur at that time. So, how to solve this problem?\n- The Cross-modality Co-Occurrence module in Figure 1 has some minor problems, which is the output of the example is the probability of $\\hat{p}^a_t$, but it is indeed $Query$ made by visual on the audio feature. I think this is the author accidentally wrote the opposite.\n- There are too many similarities between the Preliminaries part of the article and the literature [c], and it is recommended to adjust.\n- It is recommended to bold the best experimental results in Table 3.\n\n[c] Yu Wu and Yi Yang. Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In *CVPR*, 2021.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "f0wPty7Pd4b", "reviewer": "Reviewer_ffnZ", "summary": "The paper describes a method for audiovisual event parsing. It proposes a method based on attention based features aggregation along with cross-modal co-occurrences. A contrastive learning based loss is also added to encourage features across modalities to be similar for the same event. Experiments are done on a single audiovisual dataset - LLP dataset -- which is an audiovisual dataset of primarily sound events.  ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes methods for audio-visual event recognition and detection. It is primarily an attention driven method in which attention mechanisms are used for feature aggregation as well as for learning audio-visual correlations.  \n\n1. The method is primarily based on prior works [4] and [6]. Taking these two works into consideration, the method does not seem particularly novel and a minor addition over these works. The attention based feature aggregation is based on these prior works. The contrastive learning loss which is aimed at encouraging similarities between audio and visual features for the same event also seems to be based on a similar loss in [6]. So the main addition is cross-modality co-occurrence, in which the key differentiating factor is class-specific formulation. Otherwise, it is mere repetition of the feature aggregation process (another block of layers doing the same thing as feature aggregation). \n\n\n2. The class-specific attention in Sec 3.2 leads to learning  2xC (#num classes) matrices of size d x d_c. This can easily blow up as C increases, even if it reaches a few hundred ? The LLP dataset used in the experiments is derived from Audioset which itself has 500+ classes. I would like to hear authors comment on how it will scale as C becomes large. Also what if we dont do it in a class-specific manner ? How much worse that is compare to the current formulation ? \n\n\n3. How are the weights w’s in Eq 3 computed ? Its not clear from the text. \n\n\n4. There are plenty of works on audio-visual learning with different forms of attention - , temporal, self, cross-modal etc. R1 and R2 below to mention just a couple. There are likely more. The paper does not really put itself against these other formulations of audio-visual attention and does not explain why and how its different and perhaps better.  Authors should refer to these works and some sort of technical or empirical comparison is expected. R1 for example applies a relatively simpler attention mechanism for weakly supervised sound event recognition. It shows experimental results on full Audioset which is the source of the LLP dataset used here in experiments. \n\n\n5. The empirical evaluations and analyses are rather weak. Experimental results on only 1 dataset are shown. Not much insight is provided into the framework. Like when cross-modal is important when it is not. What type of weights do we learn for different classes - are they meaningful ? What if in Eq 5 only self-modal attention (without the visual  terms for audio and vice versa) ? The qualitative analysis is also not providing anything interesting. The ground truths for both classes are themselves spread over the whole video duration which is not really an interesting case to get an idea of localization capabilities. It would have been more interesting to see how well the method can localize when the events starts and end somewhere in the recording. \n\n\n[R1] Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data, IJCAI 2020. \n\n[R2] Cross-Attentional Audio-Visual Fusion for Weakly-Supervised Action Localization, ICLR 2021. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not pointed out any specific limitations. Audiovisual learning in general can have negative societal impact when not carefully done. However, in the context of current paper there might not be obvious critical ones. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "SNDBaGz14H6", "reviewer": "Reviewer_2QRh", "summary": "The authors address weakly-supervised audio-visual video parsing problem, which aims to temporally parse a video into audio, visual, and audio-visual events learning from video-level labels. To improve parsing performance, they propose to explore additional cross-video and cross-modality supervisory signals. Experiments on the LLP dataset can validate the effectiveness of the proposed method. \n\n**Post rebuttal:**\n\nI've read comments from fellow reviewers and the rebuttal from the authors. \n\nIn this paper, the authors propose an interesting method to address a new audio-visual video understanding problem: audio-visual video parsing. Two key technical contributions: co-attention with class-level representation and audio-visual contrastive objective are well motivated and technically sound. Experimental results show that it can outperform very recent state-of-the-art methods. \n\nI had some concerns about the limitations of the proposed loss term and missing experimental results. But, the rebuttal can successfully address my major concerns. Thus, I would like to upgrade my rating to 7.  \n \n**To authors:** \n\nI do encourage the authors to add the discussion about the limitations of the proposed method and new results to the paper. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n+ The proposed methods are well motivated and technically sound. With class-level audio and visual features, the co-attention mechanism can explicitly model the relationship between event categories cross different modalities. With the proposed contrastive objective, it will tend to encourage features across modalities with the same event category to be close and those with different events to be far away from each other. \n+ Extensive experiments on the LLP dataset validate that the proposed method is superior over recent state-of-the-art audio-visual video parsing approaches.\n\nCons:\n\n-  In Eq. 8, the proposed contrastive objective encourages audio and visual features from two videos with at least one overlapping category to be close and those with non-overlapping categories to be far away from each other.  However, audio and visual tracks in a video can contain totally different events and the categories in the video-level labels could be from either single modality (audio or visual) or both. The loss will introduce learning noises when the overlapping categories only appear in the same single modality and two videos have no similar cross-modal content. For example,  given two videos A and B and their labels are {dog, speech} and {dog, guitar}, respectively, and dog is an audio event in A and B. Clearly, they have the same dog category. The loss term will push audio feature from A be close to visual feature from B. However, the dog event is an audio event in B. It means that the visual feature of B does not have any information about dog. Then, the loss term will introduce learning noises. \n\n- The MA method also used contrastive learning (CL) to improve audio-visual video parsing. The authors mentioned that the CL method in MA is based on a strong assumption that audio and visual signals are synchronized and the proposed CM-S will have no the problem. However, there are no stratforward experimental results to verify the superiority of the proposed CM-S over CL in MA. To validate the effectiveness of the proposed method: CM-S, the authors should compare CM-S with the CL method in MA using HAN and MA as baselines in ablation study. \n\n- Why HAN + Ours achieves even worse audio parsing results than HAN? It seems that the proposed method mainly improve visual parsing .  The authors should provide more analysis and explanation on the results. \n\n- The contrastive objective is computed by pairing all videos in training batches. Whether it will significantly increase training cost? \n\nOverall, I think the proposed method is technically sound. But, I still have some concerns about the method and experiments.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It would be good if the authors can provide some failure examples in the paper and analyze the limitations of the proposed method. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "ClyEaAkrH9h", "reviewer": "Reviewer_jrWL", "summary": "The paper presents a procedure for weakly supervised audio-visual semantic parsing, an interesting (relatively new) audio-visual task, where the goal is to predict and localize both audio and visual events, while leveraging only video level (ie non localized) annotations for training. \n\nThe paper presents two main contributions. The first is the introduction of the so-called \"cross-modality co-occurrence module\" and the second is the observation that a cross-modal weakly-supervised contrastive loss enhances audio-visual parsing, by aligning video-level audio and visual representations of semantically similar samples. One of the main concerns about this paper is with respect to the motivation and experimental evaluation of the first contribution (see main review section).\n\nThe paper achieves good performance on the audio-visual parsing task, outperforming prior state-of-the-art by significant margins.\n\n------\n\nPost rebuttal:\n\nAfter carefully considering the comments of all reviewers and the rebuttal, I will increase my rating to 7. The paper makes progress in an interesting and relatively recent audio-visual understanding problem. The proposed approach is sound and achieves significant gains over prior methods. The authors also addressed my main concerns in the rebuttal, and I strongly encourage the authors to add the new results to the paper, as well as the distinctions between the proposed method and [46].", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I'm overall positive about the paper (due to the reasons outlined above), I hope the authors can address the following concerns. I will reassess my recommendation based on the responses.\n\n1) It was not clear to me why the module introduced in section 3.2 is called cross-modality co-occurrence (especially the co-occurrence part). From equations 4-5, it seems that this module computes \"class-wise\" features, and then performs aggregation over the class dimension using attention.\n\n2) What are the parameter and FLOP count for the \"cross-modality co-occurrence\" (CM-Co) module? If this is significant, it may not be fair to compare prior work like MA or HAN to this method. In other words, it may not be possible to conclude that the reason why CM-Co works is that it provides a better inductive prior for the model to generalize faster, or is it only due to the higher parameter or FLOP counts. One possibility to control for this would be to implement HAN and MA with a feature aggregation module twice as deep.\n\n3) What differences between the CM-Co and [46] should be highlighted in the paper. As is, it is unclear what are the differences, if any?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper aims to improve audio-visual semantic parsing (an established research area). As such, it does not introduce new negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours split over 2 days"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes a method for audiovisual event parsing. It proposes a method based on attention based features aggregation along with cross-modal co-occurrences. A contrastive learning based loss is also added to encourage features across modalities to be similar for the same event. There was some disagreement between reviewer scores (three positives one mildly negative), although not for the reasons behind those scores, The reviewers agreed that the paper addresses a recent and interesting problem - weakly supervised audio-visual semantic parsing, where the goal is to predict and localize both audio and visual events, while leveraging only video level (ie non localized) annotations for training - , that the paper is technically sound, and that the approach produces good results. The main limitation was that there is limited technical novelty. While the solution is novel, it is obtained by combination of variants on previously used building blocks. After discussion, it was concluded that the paper addresses an interesting problem in an interesting way. Although somewhat incremental, the paper deserves publication.
Decision: Accept (Poster)