Exploring Cross-Video and Cross-Modality Signals for Weakly-Supervised Audio-Visual Video Parsing
Authors: Yan-Bo Lin
Hung-Yu Tseng
Hsin-Ying Lee
Yen-Yu Lin
Ming-Hsuan Yang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Audio-visual Video Parsing
Weakly-supervised Learning
Vision Applications and Systems

Abstract: The audio-visual video parsing task aims to temporally parse a video into audio or visual event categories. However, it is labor intensive to temporally annotate audio and visual events and thus hampers the learning of a parsing model. To this end, we propose to explore additional cross-video and cross-modality supervisory signals to facilitate weakly-supervised audio-visual video parsing. The proposed method exploits both the common and diverse event semantics across videos to identify audio or visual events. In addition, our method explores event co-occurrence across audio, visual, and audio-visual streams. We leverage the explored cross-modality co-occurrence to localize segments of target events while excluding irrelevant ones. The discovered supervisory signals across different videos and modalities can greatly facilitate the training with only video-level annotations. Quantitative and qualitative results demonstrate that the proposed method performs favorably against existing methods on weakly-supervised audio-visual video parsing.

Exploring Cross- Cides-Video and Cross-Modality Signals for Weakly-Supervised Audio-Visual Video Parsing
Yan-Bo Lin
Hung-Yu Yu Tseng²
Hsin-Ying Lee*
Yen-Yu Lin
Ming-Hsuan Yang3.5.6
National Yang Ming Chiao Tung University UNC Chapel Hill
UCC Merced
ASnap Research Google Research Yonsei University
yblindunc. edu htseng edu hlee5@snap.com
lin0cs nctu. edu. tw mhyang@ucmerced.com
Abstract
The audio-visual video parsing task aims to temporally parse a video into audio or visual event categories. However, it is labor-intensive temporally annotate audio and visual events and thus hampers the learning of parsing model. To this end, we propose to explore additional cross-video and cross-modality supervisory signals to facilitate beakkl-suuperiseer audio-visual video parsing. The proposed method exploits both the common and diverse event semantics across videos identify audio or visual events. In addition, method explores event co-occurrence across audio, visual, and audio-visual streams. We leverage the explored moddlityy co-occurrence to localize segments of target events while excluding irrelevant ones. The discovered supervisory signals across different videos and modalities greatly facilitate the training with only video-level annotations. Quantitative and qualitative results demonstrate that the proposed method performs favorably against existing methods weakly- supervss audio-visuu video parsing.
Introduction
Humans perceive multisensory signals via seeing, hearing, touching, etc., and obtain multimodal information while exploring the surrounding environments. Visual and audio signals, the most common modalities, motivate researchers to jointly comprehend audio-visual events (e.g.. see people singing and hear their sounds) [1, 2, 3, 4, 5, 6, 7]. Events visible in images while hearable in audio are referred to audio-visual events. However, learning-based models tend to recognize particular audio-visual event by using the data from the dominant modality with richer information and overlook clues from either audio only or visual only events which still contribute to holistic video understanding. Therefore, the resultant models can generalize well on audio-visual events only instead of comprehensively understanding all kinds of video events. To address this issue, target at audio-visual video parsing [4, 6] where predictions for audio, visual, and audio-visue events with temporal boundaries all required but separately evaluated.
The time-consuming and visual video parsing task. To address this issue, Tian er al. [4] handle this task in weakly-supervised manner given only video-level labels, which indicate events presence without temporal bound- aries and detailed modalities. They develop audio-visual co-attention mechanism to assemble discriminative multimodal representations and use multiple instance learning to aggregate frame-level predictions into video-level ones. However, video-level labels alone cannot identify which modality events are from. Wu et al. [6] then propose to perform label refinement by swapping the audio and visual tracks of different videos to estimate and remove irrelevant event categories for each modality. They further adopt temporal contrastive learning to align audio and visual representations from the same frame. However, the contrastive learning is based on the assumption that audio and
annotation process poses a major challenge for the audio-
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

visual signals are synchronized, which may not hold in practical scenarios with complex events. Furthermore, these methods [4, 6] only consider audio and visual tracks of a single video without exploiting the relationship across categories and videos, which also provide rich shared semantics regarding event categories.
In this work, we propose to leverage audio and visual data across different videos to explore shared information of each category. For example, videos with singing events may have similar patterns whatever in an audio or a visual modality. By observing all videos in a training batch, we can not only explore the shared semantics among audio-visur data but also exclude unrelated events. In addition to the relationship across different videos, we exploit the dependency between event categories. For example, when people are singing, there is usually music accompaniment. Therefore, we propose to treat audio, visual, and audio-visual streams separately and adopt an audio-visual class co-occurrence module that jointly explores the relationship of different categories among all streams. By measuring the similarity of event categories from audio, visual, and audio- events, the correlated events are more likely to be correctly determined as the presence or absence of event categories. Such a strategy can robustly learn the correlation of categories within/across modality and fully exploit video data. The proposed strategy can be applied to existing methods on video parsing.
We evaluate the proposed method on the LLP [4] dataset. Videos are parsed into audio, visual, and auuio-visuall events under both segment and event levels, and evaluated with F-scores metrics. Both qualitative and quantitative results demonstrate the effectiveness of the proposed method on the audio-visual video parsing task. The main contributions of this work are summarized as follows:
We leverage audio and visual data across different videos and tracks, which can learn common semantics of the same events and discern unrelated clues.
We develop an audio- visual event co-occurrence module that jointly considers the relation- ship of categories in audio, visual, and audio-visual modalities, which can prevent models from differentiating the representations of the related events.
Qualitative and quantitative experimental results on the benchmark dataset demonstrate that the proposed method performs favorably against the state-of saate-of-the-arne the--rtss in various settings.
2 Related Work
Audio- Visual Representation Learning. Implicit correlation between audio and visual data from videos provides rich information for audio-visual representation learning. First, the audio-visual pairs from the same video clip [8, 9, 10, 11, 12, 13, 14, 15, 16, 17] are strongly correlated based on the assumption that audio and visual data from a video are synchronized and highly correlated. Moreover, features extracted from unpaired video clips tend to be more diverse than those from the same clips. Second, by exploring audio-visual temporal synchronization [18, 19], temporal information can be served as a training guidance. Given a video sequence, existing methods [18. 19] distinguish audio and visual features from different frames while correlating features from the same frames. Such an idea enhances robust audio-visual representation learning that essential several tasks such as audio-visual event recalizatioo/parsnnnrrreege
[1. 2, 3, 4, 5, 6, 7, 20], sound separation [21, 22, 23, 24, 25, 26, 27, 28, 29, 30], audio spatialization [31, 32, 33, 34, 35, 36, 37, 38), and sound localization [39, 40, 41, 42, 43, 44]. Instead of random sampling sound and images, our method selects both related and irreverent videos to explore common semantics and discern dissimilar
events.
Audio-Visun Video Event Localization and Parsing. Audio-visual video parsing aims detect events in videos and identify audio, visual, and audio-visual events seeing the event and hearing its sound) and activities. Videos can be parsed with event categories and boundaries in both audio and visual modalities. Early researches [5, 7, 2, 3] aim to jointly derive audiovisual information in each local segment of the input video for audio-visual event localization, which emphasizes to detect only audio-visual events. However, due to the inconsistent information observed from audio and visual signals, data from either modality with insufficient clues may degrade the performance of prediction. Therefore, the work [7] focuses on audio/visual data with relevant categorical events to tackle this issue. Although methods of this category present favorable results, they are applicable to audio-visua event localization, which considers only synchronous audio-visual events or not. Recently, multi-modal multiple instance learning (MMIL) based methods with hybrid attention [4] carry out superiss audio-visuu video parsing. These methods aggregate segment-level
2

---

Autio
Video Inputs
Lnn
P
Papaees Pizz
Fatuu
Co
Palaree Piz
Coss Hodale Occantzze
[Na Yigi
Fine
Relltad Vidaos
MIP
Shard
Croce-Mrd MrdallModallty
BRules Rug
Suturtii
Ulansizaed
HLLP
Addio
Uno
Class fustanne
Canoo-Mssdilyy
Festur
Croes Mobiiity CpeDremmeoo
Aggmeattnn
Visui
Figure Algorithmic overview. Our framework consists of a visual feature extractor, an audio feature extractor, feature aggregation module, MMII. pooling, shared cross-modality semantics, and an cross- mooss-modality co-occurrence module. Given n videos of seconds, the visual and audio feature extractors compute their visual and audio features. The feature aggregation module [4] conducts self- and cross modality attention aggregate segment- segment-wise audio fa and visual f" representations. We map segment- wise aggregated features to class-specific features by exploring cross- cross-modality co-occurrence. By performing self- and cross- coos--modality attention for class features, we identify within and cross modalities relationship between classes for event predictions. Note that & denotes matrix multiplication with the softmax operation performing on each row, and the green block only shows the example for segment-wise visual prediction at time f. We also leverage the aggregated features of all videos figure common semantics regrading events by maximizing the similarities between related videos while minimizing those between unrelated videos with Eq. 8, The MMIL Pooling [4] an attention-baset pooling function that aggregates segment-wise results produce video-level ones, which are optimized by the binary cross entropy loss described in Eq. 3 and Eq. 6.
predictions into video-level ones, with which optimizing model by using video- weak labels is enabled. Since video-level labels are typically insufficient to identify either audio or visual events, Wu et al. [6] generate pscudo labels for each modality by exchanging audio and visual tracks between unrelated videos. However, we notice that videos with replaced sounds or images may share some common semantics. Our method can exploit videos in training batch extract their common semantics for a categorical event and discern unrelated clues. Furthermore, we can leverage the relationship between event classes find out related events (e.g., singing may accompany music).
3 Proposed Method
In this paper, we propose a novel framework for weakly-supervised audio-visual video parsing. In order to explore common semantics across videos and dependency across event categories, the proposed model leverages all audio and visual signals across videos in a training batch and the correlation between classes for each training instance. Section 3. we first define the notations and settings considered in this paper and revisit the common backbone [4, 6] for weakly-supervised audio-visual video parsing, which consists of feature aggregation and multi-modal multiple instance learning (MMIL) pooling. Then in Section 3.2 and Section 3.3, we detail the modules we propose to capture dependency across different events and information across different videos, respectively.
3.1 Preliminaries
Problem Formulation and Notations. Given video sequence with T seconds long, we obtain T non-overlapping audio and visual segments where each segment is one-second long. Models are aiming to predict the event labels for each segment, which may contain several or no events. time t, there three targets for audio, visual, and audio-visun events: y" e R 1 ,y" e glxc

---

and yt" E IR 1x are multi-class event label with C' event categories, yf, Ył, and yt denote audio, visual, and audio-visual event labels, respectively. We note that detailed annotations (e.g., yř. y:. and y(") are not accessible during training and only available during evaluation. As for training, only video-level annotations are available during training. Video-level annotations only contain action event categories without indicating specific times slots or modalities (e.g., audio and visual event).
Revisit of Suuervy--pe Audio- Visual Video Parsing. The previous method [4] presents promising results with feature aggregation based on transformers and multimodal multiple instance learning (MMIL) pooling. Given video sequence of T frames, we denote audio and visual feature sets by F [f ....... (T) IR and FV = {f[ ...., [ E RT'xd, respectively, where d is the feature dimension. The transformer encoder [45] employed to aggregate both muthitymdlality and cross-modality information using multi-head attention blocks:
(fr,....
F
s F F Fa) Softmax
F
(I)
f F
Ccoosa F", F*) = Softmax(
vd
where &eets(-) (-) and (-roas(-) are self-attention and cross-modality attention functions respectively. They perform dot- hoo-product on features across time stamps by using non-shared MLPs. Then the jointly aggregated representations are described as follows:
fe f + Deels (F. F Fa) Pcross (f.. F", F",F". FF)),
f." f." + dsels (f,', F", F") + Φcross (f. F FF),
With the aggregated audio and visual features fo and f.", we can obtain the Trame-wist event prediction P E R. RIIC and P RIxC and the attention weights computed by MLPs and normalized by softmax function for audio, visual, and audio-visual streams (i.e., w e RixC' wi E RIxC and wi E R2xC), Then the video-level prediction is gathered with the MMIL pooling:
T
pd Σ WPP:
p Σ W. and p TM Σ
w au [0 WOW + "["[[]](p"
(3)
The model can then be optimized using the binary cross-entropy loss function between p and vidco-level weak label y 1xC which does not indicate time boundaries and modalities for events.
3.2 Cross-Modality Co- Co-Oceurrence
Videos with multi-label events contain rich information among event categories because the related events are likely to present the same time. The correlation is useful for models to robustly predict the presence or absence of events.
Similar to [46), explicitly model the relationship between event categories in different modalities. we first obtain the representations for each class and then measure the correlation. We note that the class relationships may be different in audio and visual modalities. That is why the work [46] cannot be directly applied to audio-visual video parsing since audio or visual events can be partially single frame. Thus, jointly understanding the class relationship within modality and across two modalities can benefit the audio-visual video parsing task.
or jointly presented
In order to map the frame-wise audio and visual features into class-level ones, the nonlinear transfor- mation with MLPs is formulated as follows:
at.c ReL.U + bg),
Vt.c = ReLU(f + b"),
where ag, and Vz,c are audio and visual class-level features for class c at time with dimension I x d,, respectively. The weights and biases for class e for audio and visual features are denoted as M M. E Rddd, and bg. by E Rixd, With class-level representations, we can further model the relationship between event categories within and across modalities by self-attention and cross- modality co-attention mechanism:

---

atc at,c + Pacty(at,cs A1, Ar) ) + Peroso Vo Vt), Vr. = Vt.c + Psels v,, Vi) + Pcross (V As),
(5)
where A; {ar,1. a..c) and V, (Vi,1,.... Vt,c) are sets of audio and visual class features at time t. âr. and Vt,e are now co-occcurence features that consider the relationships between categories within and across modalities. We can then predict the probability for each event at time t by MLPs and aggregate every segment-wise predictions into video-level ones i.e.,
pº = ((MLPa((@r..... p"
()), P = (MLP. MMILL(P). ................................ ................................ P7)), (Pi.....PP7) pT.))
vi.c)),,
(6)
where the sigmoid function, and M-) is the multi-modal multiple instance learning pooling described in Eq. 3 taking all segment -iiee predictions as inputs. The video-level prediction can be optimized by the binary cross-entropy loss function with a video-level weak label y.
3.3 Shared Cross-Modality Semanties across Videos
The information across different videos provides rich supervisory signals that benefit the training of weakly- seakky-superviied mudio-visual video parsing. By observing videos in training batch, can discover both the common and diverse event semantics. With video-level labels, we can initially associate related and irrelevant videos. In order to obtain discriminative categorical representation, we would like to encourage audio and visual representations from related events to be similar and differentiate those from irrelevant videos. However, targeting at segment wise representations with specific events is difficult due to the lack of temporal annotations. Therefore, we seek event-related frames through the weights from MMIL pooling in Eq.
T exp(ely @ wP))
T
exp(gly ww))
f" =
20
(7)
[=1 Er= explgly O ow))
Er= exp(e(y O w)))
1=1
where @ and g()) are element-wise dot product and summation function over all elements respectively.
With video-level labels and features (f" and f"), adopt contrastive learning [47, 48, 49] to encourage features across modalities with the same event category (at least one) to be close and those with different events to be far away from each other. We leverage all videos in a batch to explore diverse semanties, where the sets of audio and visual features are denoted as (f@))-... fin)) and
( .... ..,,) respectively with video-level labels (F(o),...) F(.)) The relationship across videos can be optimized by the proposed training objective follows:
2
Σ fly; 8j) exp(x, (/g//T)
Lcontrast
log
(8)
Σj= exp(()) /T)
where f()) clipping function that clips values over 1, and T denotes hyper-parameter controlling the temperature. Thus, the proposed method can be optimized by joint the binary cross-entropy loss mentioned in Section 3.1 and the contrastive learning loss in Eq. 8. Our training strategy exploit cross-modality information across videos and event categories to understand common semantics while ignoring irrelevant ones.
4 Experimental Results
Datasets. We use the Look, Listen and Parse (LLP) Dataset [4] for all experiments. The LLP dataset consists of 11, 849 10-seconds video clips annotated with 25 event categories. It covers various real-life scenes such as speech, music performances, car, cheering, dog, etc. Particularly, there 7202 video clips labeled with with only video-level event annotations for model training. The detailed annotations (e.g., individual audio and visual events per second) are available for the remaining 1849 validation and test videos. For all experiments, we use the official data splits from the LLP dataset.
than one event category. We use the 10000 video clips
Evaluation Metries. Following previous work [4, 6). we adopt F-scores as the evaluation metrics. Note that all types of events (audio, visual, and vidiol)isual) are measured under both segment- level and event-level metries. The segment-level metries evaluate snippet-wise prediction
5

---

Table 1: Quantitative results of weakly-supervised audio-visual video parsing. We evaluate all methods on the LLP dataset [4] with F-scores in five different event types and two kinds of segments. The first row indicates five different event types (audio, visual, audio-visual, Type @ AV, and Event @ AV). In the second row, two kinds of segments are shown: Seg. and Event are segment-
level and event-level; and indicates only label refinement is utilized for fair comparisons.
Audio
Visual
Audio-visual
Type AV
Event @ AV
Method
Seg.
Event
Seg.
Event
Seg.
Event
Seg.
Event
Seg.
Event
AVE [5]
47.2
40.4
37.1
34.7
35,4
31.6
39.9
35.5
41.6
36.5
AVSDN [2]
47.8
34.1
52.0
46.3
37.1
26.5
45.7
35.6
50.8
37.7
AVSDN + Ours
48.3
41.2
52.4
48.5
46.9
40.0
49.2
43.2
53.2
40.1
HAN [4]
60.1
51.3
52.9
48.9
48.9
43.0
54.0
47.7
55.4
48.0
HAN + Ours
59.2
51.3
59.9
55.5
53.4
46.2
57.5
51.0
58.1
49.7
MA [6]
60.3
53.6
60.0
56.4
55.1
49.0
58.9
$3.0
$7.9
50.6
MA
59.8
52.1
57.5
54.4
52.6
45.8
56.6
50.8
56.6
49.4
MA + Ours
60.8
53.8
63.5
58.9
57.0
49.5
60.5
54.0
59.5
52.1
results. As for the event-level metrics, the clips are extracted by concatenating positive consecutive segments in the same events. Then, compute the event-level F-scores with mloU 0.5 as the threshold. Furthermore, the overall Type@AY AV performance on audio-visus scene is also considered by computing the averaged audio, visual, and vusio-visual event evaluation results. Instead of directly averaging results from different event types, Event @ AV considers all audio and visual event categories for each sample.
Implementation Details. We implement the proposed method using Py PyTochh [50], and conduct the training and evaluation processes
single NVIDIA GTX 1080 Ti GPU with 11 GB memory.
Following [4, 6]. we use the same visual and audio encoders for fair comparisons. We adopt both ResNet-152 [51] pre-trained on ImageNet [52] and 3D ResNet [53] pre-trained
Kinetics-400 [54] as visual feature extractors. Visual frames are sampled at 8 fps and their 2D and 3D visual features concatenated and then processed by MI. the segment-wise representations. As for audio data, we utilize VGGish [55] pre-trained on AudioSet [56] to extract (28-dimensional audio features. The code and models are publicly available.
extracted. The 2D and 3D) visual features
Evaluated methods. We compare the proposed method based on several baselines to the following ceakly-unsuperviseee approaches to the audio-visual video parsing task:
AVE consists of an audio-guided co-attention mechanism to adaptively learn the sounding regions. We note that AVE [5] deals with the audio-visun event localization task. Thus, we follow [4] and add additional audio and visual parsing branches for the weakly- supervis audio-visual video parsing task as a baseline.
AVSDN [2] sequence-to-see -o-sequence- based model to integrate global audio and visual features to local ones. Since AVSDN [2] also deals with the audio-visual event localization task, we make the same modifications to AVSDN as those to AVE.
HAN [4] is multi-modal multiple instance learning- based method with hybrid attention network.
MA [6] reports the state-of-the the-art performance on the weakly-supervised audio-visual video parsing task. a method based on HAN with the label refinement and the audio-visual contrastive learning differentiating temporal segments.
4.1 Quantitative Evaluation
Table 1 shows the quantitative comparisons on the LLP dataset [4]. The proposed method performs favorably against the competing approaches on the -upervisd audio-visual video parsing task. Since our method can be easily extended to existing methods, we extend the proposed on three baselines. The third, fifth, and last rows Table 1 indicate that the proposed method generally benefits three baselines on several metrics of the audio-visual video parsing task by large margin. We note that MA * [6] only utilizes label refinement to refine labels for each modality, and temporal difference andio-visual contrastive learning [6] is not implemented.

---

Table 2: Ablation study. We investigate the effect of using different design components in the proposed method. We show how proposed cross-modalin co-occurrence (CM-Co) in Section 3.3 and shared cross-modality semantics across videos (CM-S) module in Section 3.2 improve the baselines.
Audio
Visual
Audio-visure
Type AV
Event @ AV
Method
Seg.
Event
Seg.
Event
Seg.
Event
Seg.
Event
Seg.
Event
HAN [4]
60.1
51.3
52.9
48.9
48.9
43.0
54.0
47.7
55.4
48.0
HAN + CM-S
58.1
49.6
58.3
53.6
53.2
46.3
56.5
49.8
55.9
47.5
HAN + CM-Co
59.7
51.4
57.4
52.4
51.9
44.2
56.3
49.3
57.4
48.5
HAN + Ours
59.2
51.3
59.9
55.5
53.4
46.2
57.5
51.0
58.1
49.7
MA [28]
60.3
53.6
60.0
56.4
55.1
49.0
58.9
53.0
57.9
50.6
MA CM-Co
61.1
53.3
61.7
57.3
56.3
49.0
59.7
53.0
58.9
51.2
MA
59.8
52.1
57.5
54.4
52.6
45.8
56.6
50.8
56.6
49.4
MA " + CM-S
60.4
53.5
60.7
56.5
55.8
47.5
58.9
52.5
58.6
51.0
MA * CM-Co
60.5
53.6
61.3
56.5
54.9
46.7
58.9
52.3
59.1
51.4
MA* + Ours
60.8
53.8
63.5
58.9
57.0
49.5
60.5
54.0
59.5
52.1
We notice that our method significantly improves baselines in the metrics of visual, audio-visual, Type AV, and Event AV. By observing the class distribution training sets, find that 31% 7%, and 9% training videos contain speech, singing, and violin events. These events are more likely to present in the audio modality. Therefore, the video-level labels would limit the performance regarding visual events. The proposed method can leverage additional cross-video and cross-modality supervisory signals explore common semantics, which can improve results in vision-related metrics.
4.2 Ablation Study
Cross-Modality Co-Occurrence and Semantics across Video. We conduct the ablation study to analyze the individual impact of each developed component in the proposed method. The results are presented in Table 2. CM-Co represents the usage the cross-modality co-occurrence module described in Section 3.2, which leverages the relationship between categories within and cross modalities. CM-S indicates the shared cross-modality semantics across videos module described in Section 3.3, which considers all audio and visual information across videos in batch.
In Table 2, we note that both CM-S and CM-Co can improve basclines several metrics. By exploring common semantics among training videos (CM-S), we improve the performance visual and oudio-visual evaluation by a large margin. Such a strategy can exploit additional information from videos to address the potential drawback of video-level labels described in Section 4.1. Furthermore, the proposed cross-modality no-occurrence module (CM-Co) also presents favorable results. We note that the significant improvement Event AV evaluation with the usage of CM-Co can verify the efficacy of considering the relationship between categories within and across modalities. Since Event AV considers all audio and visual events for the F-score (e.g., truth positive from both audio and visual events), the improvement Event AV indicates our mrods-modalltyy co-occurrence can perform well on video parsing when events present in an audio visual modality.
In the second group of the evaluated methods in Table 2, we verify the proposed CM-S works better than the contrastive learning method in MA. We perform our CM-S on the MA model. The CM-S exploits information across different videos to address the issue that audio and visual tracks may not be synchronized. Instead, the contrastive learning method in MA is developed based on the assumption synchronization to associate the audio-vivne visual representation in a single video. Since CM-S learns diverse and common semantics, is effective and complementary to the contrastive learning approach in MA performing on a single video. We note that our CM-S generally improves the performance over all segment-level metrics, which supports our claim.
Self-attention and Cross- Modaliyy Co-attention in Co-Occurrence. Since our cross-modality co-occurrence module exploits self-attention among class-level features in the same modality and cross-modality co-attention on cross-modality class-level representations to model the relationship between categories in the same and different modalities. Taking class-level audio features in Eq. 5 as
7

---

Table 3: Ablation study. We investigate the effect of different developed mechanisms in the proposed cross-modality co-occurrence (CM-Co) module in Section 3.2. In Eq. 5, class-level features are processed by self-attention and cross-modality co-attention mechanisms. A Only and V Only indicate only self-attention performs for individual audio and visual events respectively. AV denotes performing self-attention for audio and visual events. CM-Co is the proposed method that considers relationship between categories within and cross modalities by both self-attention and cross-modality co-attention mechanisms.
Audio
Visual Event
Audio-visual
Type@ AV
Event @ AV
Method
Seg.
Event
Seg.
Seg.
Event
Seg.
Event
Seg.
Event
HAN [4]
60.1
51.3
52.9
48.9
48.9
43.0
54.0
47.7
55.4
48.0
HAN + A Only
60.5
52.3
49.8
43.9
45.6
38.3
52.0
44.8
55.7
45.9
HAN + V Only
56.1
44.5
56.8
53.2
49.7
40.7
54.2
46.1
54.1
44.6
HAN + AV
59.5
50.3
55.1
50.5
48.6
40.3
54.4
47.0
56.0
47.4
HAN + CM-Co
59.7
51.4
57.4
52.4
51.9
44.2
56.3
49.3
57.4
48.5
MA [6]
59.8
52.1
57.5
54.4
52.6
45.8
56.6
50.8
56.6
49.4
MA + A Only
60.7
52.7
53.9
47.9
50.1
42.2
54.9
47.6
57.0
47.1
MA R Only
46.8
34.4
60.8
57.0
42.8
31.1
50.1
40.9
52.6
40.4
MA + AV
58.3
50.4
59.4
55.2
53.9
46.9
57.2
50.8
56.7
48.5
MA CM-Co
60.5
53.6
61.3
56.5
54.9
46.7
58.9
52.3
59.1
51.4
Table 4: Ablation study. We evaluate the proposed method in accuracy, efficiency, and model sizes. We show the numbers of parameters and FLOPs for the proposed cross cooss-modllity co-occurrence (CM-Co) and HAN [4] with a few layers.
Note that the results are all in the segment level.
Method
Audio
Visual
Audio-visual
Type@AV AV
Event AV
GFLOPs
Params
HAN Layer
60.1
52.9
48.9
54.0
6.63
2.4M
HAN 2 Layers
58.2
55.4
50.6
54.7
54.9
7.28
2.9M
HAN 3 Layers
58.1
55.2
50.3
54.5
54.6
7.97
3.5M
HAN CM-Co
59.7
57.4
51.9
56.3
57.4
6.99
2.8M
example, the class-level self-attention and cross-modality co-attention atin(ar,es A,, Ar) and attn(a,,.> V,, Ve), respectively.
Table 3 presents the results in various modifications of the cross-modality co-occurrence module. We note that the design co-occurrence the same and cross modalities can generally improve the results in several metries. We also evaluate the co-occurrence module in single modality. The results are shown the second, third, seventh, and eighth rows in Table 3, where Only and V Only indicate the co-oceurrence module only leverages the relationship between categories in audio or visual data respectively. As the results shown in the second and seventh rows, training with co-occurrence in audio events only (i.e., Only) events. Similarly, considering visual event only (i.e., V Only) can benefit the results regarding visual events. Furthermore, the co-occurrence for both audio and visual categories (AV) in the fourth and ninth rows can contribute to the results in general metrics such as Type @ AV and Event @ AV. We then further consider the correlation between events across modalities. That is the cross-modality co-occurrence module (CM-Co) in the fifth and tenth rows. The results can confirm the efficacy of the proposed eross- modalityy co-occurrence module in all metrics except segment-leve audio events caused by similar reasons discussed in Section 4.1.
slightly improve the performance
audio
Model Capacity. Since our cross-modality co-occurrence module leverages class-level representa- tions, it would increase the capability models on capturing information. For fair comparisons, we add extra parameters to HAN [4] analyze whether parameters contribute performance gain. Specifically, we increase the number layers in its transformer-based feature aggregation to and 3, respectively.
In Table 4, we report the results in accuracy, computational costs, model sizes. The first three rows show the performance of HAN with different numbers of layers. We note that HAN with one

---

GT Va
Violin
Call
GT A:
Prdd V
Vioiin
Calo
Pod A
Calo
MA" 4Ours
Ped V:
Calo
Pred A:
Callo
MA
Figure 2: Qualitative comparisons. We compare the proposed method with the state-of satt--of-thh-arr
weakly-supervised audio-visuse video parsing method on the LLP dataset [4]. The frame-wiss annotations are shown in gray and purple bars. The gray bar denotes visual events, and the purple bar represents audio events. GT and GT are the ground-truth visual and audio events respectively. Our results are shown in the green block, and the results by the competing method, MA [6], are present in the blue block.
MA' Ours
MA'
Figure 3: Audio feature distribution by using t-SNE. The upper figure shows the distribution by our method. The lower figure presents that by MA" The legend lists all event combinations.
extra layer has more parameters than the proposed co-occurrence module. However, the results of HAN with extra layers indicate that using more parameters/layers for HAN does not improve the performance. The proposed cross-modality co-occurrence module enhances HAN more effectively.
4.3 Qualitative Evaluation
Qualitative Results. We present the qualitative results of the evaluated methods in Figure 2. GT V and GT show the ground-truth annotations for visual and audio events, respectively. Pred and Pred_A A present the predictions made by our method and the state-of-the-art competing method, MA [6], respectively. Our results are shown the green block, while the results MA are present in the blue block. In general, method presents more accurate predictions in both audio and visual

---

Dra Soutin
you Moth CLP0
MA' +Ours
MA'
Figure 4: Visual feature distribution by using t-SNE. The upper figure shows the distribution by our method. The lower figure presents that by MA" The legend lists all event combinations.
events than MA. We note that the whole violin is shown after 7 seconds. That would hamper models for understanding visual events e.g., MA predicts wrong results on violin visual events before seconds. Since our method leverages the relationship between categories, it can still predict correct
temporal boundaries for guitar events by jointly considering cello events the videos.
Feature Distribution Visualized by I-SNE. We apply t-SNE to the aggregated audio and visual features from each segment described in Eq. 2. The visualization results are present in Figure and Figure 4, respectively. The legends list all the combinations of multiple labels. For example, in Figure 3, audio events of singing violin are shown as purple spots. We note that the related events including multiple events are shown in similar colors. In Figure 4, the proposed method achieves better performance in the sense that similar color spots are closer than the spots in MA*
present blue spots, and the mixed sounds of singing and
Conclusions
In this paper, present novel audio-visual video parsing framework in weakly-supervised manner that can be applied to existing methods. We propose two modules to exploit the relationship across videos, modalities, and event categories, and explore additional supervisory signals that can benefit audio-visual video parsing. The shared cross-modalin semantics module leverages common and diverse event semantics across videos learn robust cross-modality representations that facilitate models to identify audio, visual, and audio-visual events. Furthermore, the cross-modality co-occurrence module aims learn the relationship between event categories. It helps localize segments of target events and can exclude irrelevant ones by performing self-attention and cross- modality co-attention on class-wise features, Extensive experimental results show that our approach substantially improves several baselines and performs favorably against the kate-of-the-ee art methods.
Acknowledgments. This work was supported in part by the Ministry of Science and Technology under grants 109- 2221-E-009-1 13-MM, 1002628-E-A49-00 also funded in part by Qualcomm through Taiwan University Research Collaboration Project, the Higher Education Sprout Project the National Yang Ming Chiao Tung University, and Ministry of Education.
and 100-2644F000-0155 was
10

---

References
[1] Jun-Tae Lee, Mihir Jain, Hyoungwoo Park, and Sungrack Yun. Cross- attssttattntionll audio-visual fusion for weakly-supervised action localization. In ICLR, 2021. 1.2
[2] Yan-Bo Lin, Yu-Jhe Li, and Yu Yu-Chiang Frank Wang. Dual-modality seq2seq network for audio-visual event localization. In ICASSP, 2019. 1, 1,
[3] Yan-Bo Lin and Yu-Chiang Frank Wang. Audiovisual transformer with instance attention for audio-visual event localization. In ACCV, 2020. 1,2
[4] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly- supervised audio-visual video parsing. In ECCV, 2020. 1, 2, 3, 4, 5, 6, 7, 8, 9
[5] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In ECCV, 2018. 2,
[6] Yu Wu and Yi Yang. Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In CVPR, 2021. 1, 2. 3, 5, 6, 8, 9
[7] Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang. Dual attention matching for audio-visual event localization. In ICCV, 2019. 1,2
[8] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV, 2017. 2
[9] Relja Arandjelović and Andrew Zisserman. Objects that sound. In ECCV, 2018. 2
[10] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In NeurlPS, 2016. 2
[11] Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In ECCV, 2016.
[12] Jean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapu- ram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. In NeurIPS, 2020. 2
[13] Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- supervised learning by cross
cross-modal modal audio-video clustering. In NeurIPS, 2020. 2
[14] Yuki M Asano, Mandela Patrick, Christian Rupprecht, and Andrea Vedaldi. Labelling unlabelled videos from scratch with multi-modal self-supervision. In NeurIPS, 2020. 2
[15] Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. Active contrastive learning of audio-visual video representations. In ICLR, 2021. 2
[16] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination
with cross-modal agreement. In CVPR, 2021. 2
[17] Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimina- tion. In CVPR, 2021. 2
[18] Andrew Owens and Alexei A. Efros. Audio-visual scene analysis with self-supervised multisen- sory features. In ECCV, 2018. 2
[19] Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. NeurlPS, 2018.
[20] Jinxing Zhou, Liang Zheng. Yiran Zhong, Shijie Hao, and Meng Wang. Positive sample propagation along the audio-visual event line. In CVPR, 2021. 2
[21] Chuang Gan, Deng Huang, Hang Zhao, Joshua B Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. CVPR, 2020.
[22] Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In ECCV, 2018. 2

---

[23] Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. In ICCV, 2019. 2
[24]
Xudong Xu, Bo Dai, and Dahua Lin. Recursive visual sound separation using minus-plus net. In ICCV, 2019. 2
[25] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In ICCV,
2019. 2
[26] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In ECCV, 2018. 2
[27] Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Dan Ellis, and John R. Hershey. Into the wild with audioscope: Unsupervised audio-visual separation of on-screen sounds. In ICIR, 2021. 2
[28] Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. In CVPR, 2021. 2,7
[29] Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross- modal consistency. In CVPR, 2021. 2
[30] Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. In CVPR, 2020. 2
[31] Ruohan Gao and Kristen Grauman. 2.5d sodd-d.uull-ounnd In CVPR, 2019. 2
[32] Pedro Morgado, Yi Li, and Nuno Vasconcelos. Learning representations from mudiolvisual spatial alignment. In NeurIPS, 2020. 2
[33] Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video. In NeurIPS, 2018. 2
[34] Karren Yang, Bryan Russell, and Justin Salamon. Telling left from right: Learning spatial correspondence of sight and sound. In CVPR, 2020.
[35] Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In ECCV, 2020.
[36] Yan-Bo Lin and Yu-Chiang Frank Wang. Exploiting audio-visual consistency with partial supervision for spatial audio generation. AAAI, 2021.
[37] Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural audio generation without binaural audios. In CVPR, 2021. 2
[38] Yu-Ding Lu, Hsin-Ying Lee, Hung- Yu Tseng, and Ming- Hsuan Yang. Self-supervised audio spatialization with correspondence classifier. ICIP, 2019.
[39] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self- supervised learning of audio-visual objects from video. In ECCV, 2020. 2
[40] Arda Senocak, Tae- Mhy- Oh, Junsik Kim, Ming Hsuan Yang, and So Kweon. Learning to localize sound source in visual scenes. In CVPR, 2018. 2
[41] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound sources visual scenes: Analysis and applications. TPAMI, 2019.
[42] Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In ECCV, 2020. 2
[43] Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. In CVPR, 2019. 2
[44] Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Discriminative sounding objects localization via self-supervised audiovisual matching. In NewrIPS, 2020. 2
12

---

[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017. 4
[46] Praveen Tirupattur, Kevin Duarte, Yogesh Rawat, and Mubarak Shah. Modeling multi-label action dependencies for temporal action localization. In CVPR, 2021. 4
[47] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv Preprint, 2018. 5
[48] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018. 5
[49] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 5
[50]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019. 6
[51] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[52] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009. 6
[53] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018. 6
[54] João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. CVPR, 2017.
[55] Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron Weiss, and Kevin Wilson. Can architectures for large-scale audio classification. In ICASSP, 2017.
[56] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labed labeled dataset for audio events. In ICASSP, 2017. 6
13


Reviews:
[{"review_id": "lkod2qcgfcu", "reviewer": "Reviewer_4KfH", "summary": "The paper proposes a method for weakly supervised audio-visual video parsing task where the goal is to segment video/audio streams into different event categories. Given video level label during training, the authors propose an audio-visual class co-occurrence module to capture the relationships between event categories. They also explore the shared cross-modality semantics and use contrastive losses between modality across videos to improve the categorical representation. Results are reported on the Look, Listen and Parse benchmark. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The audio-visual video parsing task is formulated in a weakly-supervised setting, making itself challenging. The authors take advantage of the given video-level label and propose two novel modules to leverage the shared information of event category across modalities and videos. These two modules can also be integrated into the existing SOTA methods to boost their performance. Although the contrastive learning is not new in audio-visual learning, the way the authors collecting positive samples is interesting, rather than just using the synchronization. \n\nQuality and clarity: the writing of paper is good and easy to follow.  The comparison with other methods is adequate, and the experimental results seems convincing. However, the efficacy of CM-S module and its difference with the vanilla contrastive learning used in MA [6] may need more discussion.\n\nSignificance: the results reported on LLP dataset surpass the previous SOTA significantly. As the proposed modules can plug and play, future works on audio-visual video parsing task can also adopt this design.\n\n****Post-Rebuttal:****\nI appreciate thoughtful reviews from other reviewers and a detailed rebuttal from the authors. The paper makes good progress on a new audio-visual video parsing problem. However, the main concern is that the techniques lack novelty, as pointed out by other reviewers, with which I agree. Therefore, I would not be upset if the paper was eventually rejected. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have some questions about the following points in the paper:\n\n1) Is the non-linear transformation MLPs in Eq(4) unique for each category? How to ensure that the class-level features are learned correctly and what is the learning dynamic? If certain visualizations on a_{t,c}/v_{t,c} are provided, it can better demonstrate the class-level features are well-learned. Otherwise, the claim that the relationships between event categories are captured is less convincing.\n\n2) Does the dot notation in Eq(8) mean element-wise dot-product?  Since the video-level labels y is a vector with value {0, 1}, why the values of (y_i \\dot y_j) would exceed 1? As this is one of the two main contributions of the paper, it is crucial for this to be clear for the reader. \n\n3) Since MA also uses contrastive learning to improve performance, MA* + CM-S (in table 2) achieves similar parsing accuracy compared to the MA (in table 1), which weakens the efficacy of the proposed CM-S module. \n\n4) How to decide the batch size? Is the performance sensitive to the batch size? As the authors are using a contrastive loss, it may need explanation/discussion on the choice.\n\nSome terms lack definition: \n\n1) The dimension of the non-linear transformation layers M_{c}^{a} in line143, is a_{t,c} of the same dimensionality as f_{t}^{a}?\n\n2) Is MlP_{a} different from MLP_{v}? As in the previous works [4, 6], a shared linear layer is used. What is the difference between this and previous works?\n\nTypos: there are some typos in the paper:\n1) L77: irreverent -> irrelevant?\n2) Figure 1 caption: regrading -> regarding\n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "IrkuDfe61b", "reviewer": "Reviewer_aCaK", "summary": "This paper studies the problem of weakly-supervised autio-visual video parsing. The authors consider the dependency between diffeent event categories which may occur simultaneously in a video, and the shared cross-modality semantics across different videos by contrastive learning. State-of-the-art performances have been achieved and sufficient ablation studies are provided to validate the effectiveness of the proposed methodology.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Informative figures and clear paper structure.\n2. State-of-the-art empirical performances and convincing ablation studies.\n\n\nWeaknesses:\n1. Technically sound but not inspiring methodology. The adopted techniques are not brand new, e.g., intra- and cross- modality modeling using transformer, feature enhancement through contrastive learning.\n2. Since the proposed method models class-specific representations, it means that the parameters of the model linearly increase as #class, leading to low efficiency for large amounts of event categories.\n\nQuestions:\n1. The contrastive learning looks more like a generic feature enhancement technique that can be applied in other domains. What is the special consideration when applying it to the studied problem?\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No limitations provided. No further suggestions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "3lbg6cIXbJY", "reviewer": "Reviewer_6GpX", "summary": "This paper proposes a new method for the audio-visual video parsing task, which is proposed in prior work Tian et al. ECCV 2020. There are two major new ideas 1) cross-modality co-occurrence, and 2) shared cross-modality semantics across videos. Experiments on the LLP dataset demonstrate some performance gains of these two ideas.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The paper is generally well written with clear formulations and nice illustrations. \n\n- The two new ideas are well conveyed: cross-modality co-occurrence and shared cross-modality semantics across videos.\n\n- Nice ablation studies to demonstrate the effectiveness of different components of the proposed framework.\n\nWeakness:\n- The main concern is the lack of novelty. The whole framework is largely based on prior work [1], and the main contributions are these two new ideas of 1) cross-modality co-occurrence and 2) shared cross-modality semantics across videos. While 1) is similar to [2] and 2) is based on the latest contrastive learning frameworks [3]. \n\n- Related to the above concern, why modeling the relationship across different videos, and why modeling the dependency between event categories are important for this particular task:  audio-visual event parsing should be better motivated, and why it is not addressed well in prior work. Since these two ideas are the major contributions, it is important to justify their importance.\n\n- Figure 2 shows an example qualitative results, but the example shown is rather simple where violin and cello are always visible and only cello is audible. It would be more convincing to show a more challenging example where there are some more variations, to better demonstrate the advantage of the proposed two ideas.\n\n\n[1] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly- supervised audio-visual video parsing. In ECCV, 2020. \n\n[2] Praveen Tirupattur, Kevin Duarte, Yogesh Rawat, and Mubarak Shah. Modeling multi-label action dependencies for temporal action localization. In CVPR, 2021\n\n[3] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.\n\n\n###################AFTER REBUTTAL#################\nThanks for the reply and I enojoyed reading other reviews. After the response, I still learn towards rejection, for the reasons in my original review. I think that this is an interesting research problem, but the current experimental evaluation and novelty is a bit limited and not well motivated. I think that the paper will be stronger in a future revision, with the suggested changes described in the response.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no Broader Impact section in the paper, although I don't any major potential negative societal impact of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "1sG1V0XzTeK", "reviewer": "Reviewer_2ihx", "summary": "The task is that of audio video parsing. Given the audio and visual features extracted for every segment (1 second) of a video, the task is to predict the events occurring in each segment. \nTraining data is weakly labelled while evaluation data is strongly labelled. Authors use the base architecture of MMIL used in [4] and [6] and propose further modifications:\n1. Authors introduce more learnable parameters to capture the class correspondence between the audio and video modalities. \nFor each segment, the audio and video features are transformed using MLP and attention layers to get class-level features. \nClass level features are then aggregated using MMIL pooling to get the predictions for each segment and the entire video. \n2. Contrastive learning based loss to enforce cross-modal similarity in the feature vectors for videos with similar events occurring. \nSegment wise features are aggregated using the temporal attention weight vector generated by the MMIL instance to get a video level representation. \nSimilarity in the contrastive loss term is weighted by the similarity in labels.\nThey show improvement over state-of-art methods for audio-video parsing.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Both the ideas of capturing class correspondence and contrastive learning have been proposed in [46] and [47, 48, 49] respectively. The novelty of this paper lies in applying it to the task of multi-label event classification. Strength of the paper is that they show improvement with baseline methods. \n2. I am unsure if the proposed loss (eqn 8) would count as contrastive loss because there is no explicit separation between \"positive\" and \"negative\" examples as is the case in papers mentioned before.\nEven the positive samples have been used in the denominator where only negative examples should be used. The denominator should have been multiplied with a term like (1-f) to discard the positive examples from being treated as negative samples.\n3. It is not clear to me why the contrastive loss was used to capture only the cross-modal similarity and not within modal similarity of feature vectors from the different samples. \n4. It is not clear to me why the authors didn't use the contrastive loss framework of [6] to enforce frame-level feature similarity for videos along with their proposed contrastive loss. \n5. How were the two losses (supervised ce loss and contrastive loss) weighted when training the model?\n6. The dimensions of tensors being multiplied in eqn 7 dont seem to match. w_a^t is of dimension 1xC and is being multiplied with a tensor of dimension 1xd. Also I am unclear how and why y_bar is used in eqn 7.\n7. Eqn 6 is unclear. According to notations, dimension of p_at^ should be 1xC where C is number of classes. The matrix A_t being multiplied is of dimension (T x d_c). Authors should clarify the dimensions of MLP layers that are used to convert that matrix into a vector of size 1xC.\n8. If the labels y for each event are binary then taking a dot product might not be the best way to get similarity between the labels in eqn 8. [1 0 0 0] would be equidistant from two examples with labels [1 1 0 0] and [1 1 1 1] if '.' refers to simple dot product. Authors should mention what the operation '.' is in equation 8 in case it is not dot product.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations not discussed. The authors can do qualitative error analysis and downsides of using their approach.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Obviously, this work is of generally reasonable quality, but well-informed reviewers had serious questions as to the novelty of the work and strength of the evaluations.   I think the reviewers were generally fair, so I will go with their overall average score in making my recommendation, but I recognize that this paper is very close to the border.  
Decision: Accept (Poster)