Multi-modal Grouping Network for Weakly-Supervised Audio-Visual Video Parsing
Authors: Shentong Mo
Yapeng Tian
Venue: NeurIPS 2022 Accept (2022)
Keywords: Weakly-Supervised Audio-Visual Video Parsing
Multi-modal Grouping

Abstract: The audio-visual video parsing task aims to parse a video into modality- and category-aware temporal segments. Previous work mainly focuses on weakly-supervised approaches, which learn from video-level event labels. During training, they do not know which modality perceives and meanwhile which temporal segment contains the video event. Since there is no explicit grouping in the existing frameworks, the modality and temporal uncertainties make these methods suffer from false predictions. For instance, segments in the same category could be predicted in different event classes. Learning compact and discriminative multi-modal subspaces is essential for mitigating the issue. To this end, in this paper, we propose a novel Multi-modal Grouping Network, namely MGN, for explicitly semantic-aware grouping. Specifically, MGN aggregates event-aware unimodal features through unimodal grouping in terms of learnable categorical embedding tokens. Furthermore, it leverages the cross-modal grouping for modality-aware prediction to match the video-level target. Our simple framework achieves improving results against previous baselines on weakly-supervised audio-visual video parsing. In addition, our MGN is much more lightweight, using only 47.2% of the parameters of baselines (17 MB vs. 36 MB). Code is available at https://github.com/stoneMo/MGN.

Multi-modal Grouping Network for Weakly-Supervised Audio-Visual Video Parsing
Shentong Mo Carnegie Mellon University
Yapeng Tian" University of Texas at Dallas
Abstract
The audio-visual video parsing task aims to parse video into modality- and category-aware temporal segments. Previous work mainly focuses on weakly- supervised approaches, which learn from video-level event labels. During train- ing. they do know which modality perceives and meanwhile which temporal segment contains the video event. Since there is no explicit grouping in the existing frameworks, the modality and temporal uncertainties make these meth- ods suffer from false predictions. For instance, segments in the same category could be predicted different event classes. Learning compact and discrimina- tive multi-mode subspaces is essential for mitigating the issue. To this end, in this paper, we propose a novel Multi-modal Grouping Network, namely MGN, for explicitly semantic-aware grouping. Specifically, MGN aggregates event- aware unimodal features through unimodal grouping in terms of learnable cate- gorical embedding tokens. Furthermore, it leverages the cross-moda grouping for modality-aware prediction to match the video-level target. Our simple framework achieves improving results against previous baselines visual video parsing. In addition, our MGN is much more lightweight, using only 47.2% of the parameters of baselines (17 MB 36 MB). Code is available https:/ bttps://githuue /ggttti....u...///gg com/ tttm/ssoon sttssttteeMMMMM..
weakly-supervised audio-
Introduction
Humans understand the surrounding environment by integrating signals from different senses. In daily life, sound and sight are two of the most commonly used modalities, which have drawn much attention from researchers to explore computational audio-visual scene understanding.
Previous audio-visual work [1, 2] usually assumes audio and visual data are temporally aligned. However, the alignment does not always exist in real-world videos. For example, sounding objects in many videos are outside of the of (FoV). For these non-aligned cases, audio signals become more reliable in understanding the events of interest. In this work, we address the audio- visual video parsing (AVVP) task [3] that aims to parse a video into temporal event segments and predict the audible, visible, or audio-visible event categories.
Existing approaches [3, 4, 5] usually focus on learning to leverage the unimodal and cross-modal temporal contexts from weak supervisions. HAN [3] introduced a simple Multimodal Multiple Instance Learning framework with cross-modal and self-modal attention layers to utilize the video- level labels. Recent state-of-the-a art methods usually use the HAN as the baseline and modify it to further improve parsing performance. Particularly, contrastive learning and label refinement proposed in Wu and Yang [4], where they adopted contrastive loss enforce the temporal aligmment between the audio and visual features at the same timestamp and augmented training data with modality-awand event labels generation. Furthermore, Lin et al. [5] proposed to leverage
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

audio-visual class co-oceurrence to jointly explore the relationship of different categories among all modality streams.
Our key motivation is to learn compact and discriminative audio and visual representations by explicit multi-modal grouping for mitigating the modality and temporal uncertainties in the weakly-supervised audio-visual video parsing problem. During training, segment-wise event labels are unavailable for audio and visual temporal segments in videos. Thus, the multi-modal temporal modeling in the above existing methods might aggregate and implicitly group irrelevant semantic information due to lacking of fine-gras -eeegggadedd supervisions, which causes false positives for predicting categories of events. That is, audio or visual temporal segments in the same event category might be far away from the class center in the embedding space since there are no segment-level and modality-wiss constraints during training. In the meanwhile, there is no constraint for modality category prediction to match the video-level target at the end.
Different from past approaches, we propose a new Multi-modal Grouping Network, namely MGN, to explicitly group semantic-aware multi-modal contexts, which enables learning more compact and discriminative audio and visual representations. Specifically, we first extract event-aware unimodal features through unimodal grouping in terms of learnable categorical embedding tokens for each individual modality. Then, we introduce a cross-attention layer with a hard attention mechanism to aggregate cross-modal temporal contexts. Finally, we utilize a cross-modes grouping module to predict the modality category from updated class-aware unimodal embeddings.
Experimental results the L.PP [3] dataset validate that new audio-visual video parsing frame- work achieves superior results over previous state-of-the-art methods [1, 2, 3, 4]. Empirical results also demonstrate the generalizability of our approach to contrastive learning and label refinement proposed in MA [4]. In addition, we substantially reduce the parameters of previous work by using only 47.2% parameters of baselines (17 MB vs. 36 MB).
Our main contributions can be summarized as follows:
We propose new audio-visual video parsing baseline: Multi-modal Grouping Network (MGN) that enables explicit grouping in multi-modal network to learn compact and discriminative audio and visual embeddings.
We introduce class-aware unimodal grouping and modality-aware cross-modal grouping modules to aggregate multi-modal temporal contexts.
The experiments can demonstrate the superiority of our MGN over state-of-ther the--rr AVVP approaches and its generalizability to contrastive learning and label refinement.
2 Related Work
Audio-Visual Learning, Audio-visual learning has addressed in many previous works [6, 7, 8, 9, 10, 11. 12, 13, 14, 15, 16] to learn the audio-visual association between the two distinct yet correlated modalities. Such audio-visual temporal association is crucial to several tasks, such as audio-visual spatialization [17, 18, 19, 14], speech/audio separation [13, 20, 21, 11, 12, 22, 23, 24], visual sound source localization [10, 25, 26, 27, 28, 291. A comprehensive survey [30] covers recent advances in audio-visual learning. In this work, we mainly focus on audio-visus visual video parsing that aims to parse a video into temporal segments in terms of audio, visual, and audio-visual events with only video-level annotations for training. addition, different from existing audio-visual architectures, we firstly introduce explicit multi-modal grouping into audio-visun learning.
Audio- Visual Video Parsing, Audio-visual video parsing aims at temporally localizing audio, visual, and audio-visual events in videos and predicting their event categories. Early approaches [1, 2, 11, 32] tried to localize only audio-visual events by aggregating cross-modal information in each local segment of the input video. Due to exhaustive labeling costs on segment- wise audio, visual, and audio-visuu events, recent work [3, 4] explored the seakky--uperrisee audio-visur visual video parsing with only video-level activity categories for training. For example, HAN [3] introduced the multi- modal multiple instance learning mechanism with a hybrid attention network to aggregate segment- wise representations and video-level features. To alleviate the harm brought by the audio-visua asynchrony, MA [4] proposed the audio-visus contrastive learning from different frames and refined modality-aware aware labels by exchanging audio and visual tracks between unrelated videos. More recently.
2

---

Speech
Dog
Car
Cat
Video-level Prediction
Modality-aware
Cross-modal Grouping
Sigmoid
fol")"
........
Cross-modal Grouping Block
Transformer Layers
Uni-modal Grouping Block
Uni-modal Grouping Block
Transformer Layers
Transformer Layers
(cf MEl
Class-aware Unimodal Grouping
(ci)
(
Of
Figure 1: Illustration of our Multi-modal Grouping Network (MGN). The Class-aware Uni- modal Grouping module takes input features ( 1" and learnable class tokens audio and visual events for C' categories to generate aware embeddings {g. fi=1- The aggregated Mlas--warre representations are fed into the Modality Cross-modks Grouping module to merge all the audio features with video features into cross- modal modality-aware features {g. [gar]e -==1- Finally, an FC layer and sigmoid function are used to predict the video-level target of audio- visual events.
{c [c, [ C
{g"
Lin er al. [5] leveraged the common semantics shared by videos with replaced audios or frames in mini batch to detect relevant events.
Different from these baselines based on HAN, develop fully novel network architecture alleviate implicit audio-visual matching and modality category uncertainty in the hybrid attention network. We the first exploit unimodal grouping for learning audio-visual representations with class-aware semantics. Furthermore, we introduce modality cross mross-modal grouping module to match the video-level label, although the given target does indicate modalities.
Method
Given video with both audio and visual tracks, our goal is parse the video into temporal segments associated with audible, visible, and audio-visible event categories. We propose novel baseline: Multi-modal Grouping Network (MGN) address the task, which mainly consists of two modules: cross-modal modal grouping (Sec. 3.3).
class-aware unimodal grouping (Sec. 3.2) and modality-awar cross
3.1 Preliminaries
In this section, we first describe the problem setup and notations and revisit Multi-modal Multiple Instance Learning in HAN [3] for the audio-visual video parsing (AVVP) task.
Problem Setup and Notations. Given video with non-owe -veellppeippigggge audio and visual segments, our goal is to temporally localize and recognize audio, visual, and audio-visual events that existed the video. For the multi-label events with event categories at time I, we have audio, visual, and audio-visual event labels for evaluation, which are denoted as: y. yeu E RIxC. During training, we do not have the segment- and modality-level annotations. Therefore, we can only use the video-level label ev cxxc to perform -uuklysupereisedd learning.

---

Revisit Multimodal Multiple Instance Learning. To address the weakly audio-visum video parsing problem. HAN [3] introduced Multimodal Multiple Instance Learning (MMIL)
framework based on transformers [33]. Given set of audio-visudl features F {f J1=1: F {fr T segments, HAN applied self-attention and cross-attention layers to aggregate the unimodal and cross-modal information at each timestamp:
f f + Ooa F Fo) + dea [. F", F").
f. f dsa (f," F", F") + Dea (f,' F", F*),
where 2sa (.), (.) denote the self-attention and cross-attention functions:
f F
Usa fr. F Fo) = Softmax
F
(3)
vd
F vd
Sea (Tg F", F") Softmax
F
(4)
and ff; fl" E IRld Fo, F RTxd d is the dimension of audio-visual features. Then, the probability of segment-wise categories for audio and visual events (py, p E RIxC) is predicted by a shared fully-connected (FC) layer and sigmoid function. With the attentive MMIL pooling, the video-level prediction is formulated as:
T p Σ
T pd Σ WIP..
Σ T
M
w P
WiP
pdd
[m]]]] @ P.[m]]
(5)
where W, = =, [Wf, w.] is the temporal attention weights computed by a FC layer and normalized by softmax function. P, {p is the probability set of audio-visum predictions. M = 2 denotes audio and visual modalities. Finally, the model is trained to optimize a seakly-supervieed loss of a and guided loss of p", with label smoothing:
Siase = O(yn0 "oly",,p")) + O(y",, + +Oy"",
(6)
where O(-) is binary cross-entropy function-based loss term, which summarizes binary cross-entropy for all categories and O(y, p) = E; BC',,, y" are video-level audio and visual labels generated by smoothing you to decrease the confidence of positive labels.
However, such a training mechanism will pose two main challenges. First, these methods without explicit grouping suffer from false predictions due the modality and temporal uncertainties. Second, there is no constraint for modality category prediction to match the video-level target at the end. To address these challenges, inspired by [34], propose novel Multi-modal Grouping Network (MGN) with class-aware unimodal grouping and modality-aware cross-modal grouping modules, as shown in Figure 1.
3.2 Class-aware Unimodal Grouping
In order to explicitly group class-aware matching semantics for audio-visual representations, we introduce novel class-aware unimodal grouping module by incorporating learnable modality-speciffc class tokens {c ) [c to help to group raw input unimodal features {f,' )T (f," 171-
We first use self attenttention transformers: (.),()) and d% (-) to temporally aggregate unimodal features from audio and visual inputs and align the features with the categorical token embeddings:
(T (2 = (0 (x). X 2+ X = (x]) [( (c))
(7)
(C ( (x" X",
x - ((
(8)
where I; denotes the concatenation operator. Then, the unimodal grouping blocks (+), take the learned audio and visual event class tokens and aggregated features inputs to generate class-aware audio and visual embeddings as:
[g (L ( (
[ = "
(9)

---

During grouping, we merge all the unimodal features that belong to the same class token into new unimodal class-aware feature, by computing the similarity matrices A A between unimodal features and class tokens via softmax operation formulated as
A "o Wee.). A (17tma Wg
(10)
where Waa Wa and W W are the weights of the learned linear projections for the features and class tokens of audio and visual modalities, respectively. Based this similarity, we calculate the class-aware features with the weighted sum of all segment features assigned to that class:
E A Wefe ΣT
x - g E.) = c + W.
AC
(11)
T
ē) W g. =
ΣE A
where Wo, W and W , :.:' denote the learned weights linear projections for audio and visual modalities, separately. Note that, the audio features {f())T= are merged with discriminative visual features {fr) ]t= via similar grouping block with hard-softmax operation. order constrain the independence of each class token c; c, we apply FC layers and softmax operation to generate the category probability ef, eç of class tokens for audio and visual modalities with class-constrained loss as:
Cets E CE(b, et) CE(h, e
(12)
where CEC) refers to cross-entropy loss; h; is one-hot encoding vector and only its element for the target class entry is After the class-aware unimodal grouping, the video-level category predictions p", p" of audio and visual events is simply computed by a FC layer and sigmoid operator:
pd Sigmoid(FC((g)) (())) p" (igmoid(FC((g:)S)) (())))
(13)
With the help of the proposed classcoontrrained loss, we generate class-aware representations {g.) - ) of audio and visual modalities for audio-visual matching.
3.3
-odal Cross-modal Grouping
The second challenge requires us to predict the modality category for matching with the only given video-level target in explicit way. To achieve this, propose modality- cross-modal grouping module composed of cross-modal transformers (.a(.) and grouping blocks g²t()) to aggregate class-aware representations {g {g]) 1 Based the audio-visual similarity in
the grouping stage, we combine all the audio features with visual features into new cross-modal modality-aware features {g."} _
(g = dea (lgg [g
(14)
{g. 20 = "
(15)
where (.) denote the grouping operator similar to (-) and g"(-) in Eq. 11. Then, leverage the joint audio-visual representations "g"" to predict the video-level target of audio and visual events via FC layer and sigmoid function as:
p Sig (C [))
(16)
The whole model can be optimized in an end-to-end manner in terms of the objective function:
L = Lbase + Lets
(17)
At inference time, the unimodal class-aware similarity is used predict the audio, visual, and audio-visual events for each segment t:
D p A = O A. P = = G P where p", p" I 1xC and A A TXCC
(18)
5

---

Table Quantitative results of contrastive learning and label refinement proposed in MA [4], respectively.
audio-visual video parsing. 'C' and 'R' denote the
Segment-
Event-Level
Method
A
A-V
Type
Event
A
A-V
Type
Event
AVE [1]
47.2
37.1
35.4
39,9
41.6
40.4
34.7
31.6
35.5.
36.5
AVSDN [2]
47.8
52.0
37.1
45.7
50.8
34.1
46.3
26.5
35.6.
37.7
HAN [3]
60.1
52.9
48.9
54.0
55.4
51.3
48.9
43.0
47.7
48.0
MGN (ours)
60.7
55.5
50.6
55.6
57.2
51.0
52.4
44.4
49.3
49.2
MA [4] (w C)
61.9
53.1
49.7
54.9
56.2
52.8
49.4
43.8
48.7
49.0
MGN (w C)
60.6
56.7
52.5
56.6
57.4
51.4
53.2
46.4
50.3
49.4
MA [4] (w R)
59.8
57.5
52.6
56.6
56.6
52.1
54.4
45.8
50.8
49.4
MGN (w R)
60.0
60.6
54.0
58.2
58.2
50.3
58.4
47.9
52.2
49.1
MA [4] (w C+R)
60.3
60.0
55.1
58.9
57.9
53.6
56.4
49.0
53.0
50.6
MGN (w C+R)
60.2
61.9
55.5
59.2
58.7
50.9
59.7
49.6
53.4
49.9
4 Experiments
4.1 Experimental Setup
Dataset. The Look, Listen and Parse (LLP) Dataset [3] contains 11,849 YouTube video clips of 10-seconds long from 25 different event categories, such as car, music, cheering, speech, etc. Note that each video includes at least Is audio or visual events and 7202 video clips are annotated with more than one event categories. We use 10,000 video clips with only video-level event labels for training. Following the official splits [3] of validation and test sets, we develop and test the model on the remaining 1879 videos with the segment-leve annotations, i.e., the speech event for audio starts at Is and ends at 5s.
Evaluation Metrics. We follow the prior work [3, 4] and use F-scores evaluate both segment-level and event-level predictions for audio, visual, and audio-visual events. The segment-level metrics can evaluate snippet-wise event prediction performance. For the event-level metrics, we concatenate positive consecutive segments in the same events and compute F-score based on mloU=0.5 as the threshold. Type@ @ AV and Event @ AV are reported for the overall evaluation of audio viiuall video parsing performance. Type @ AV is the averaged audio, visual, and audio-visue event evaluation results. Different from Type AV directly averaging results from different event types, Event AV considers audio and visual events for each sample.
Implementation. Following the data pre-processing in previous work [3], we sample video frames at 8 fps from the 10-second videos with 10 non-overlapping snippets of second. For low-level visual features, we concatenate 2D and 3D visual features extracted by ResNet-152 [35] pre-trained on ImageNet [36] and 3D ResNet [37] pre- traitrained on Kinetics-400 [38]. We utilize VGGish [39] pre-trained on AudioSet [40] to extract the audio features. The model is trained with Adam [41] optimizer with 3,=0.9, 82=0.999 and with an initial learning rate of 3e-4. We train the model with batch size of 16 for 40 epochs.
4.2 Comparison to Prior Work
In this work, we propose a novel and effective training framework for weakly-supervised audio-visual video parsing. To demonstrate the effectiveness of our approach, we first compare it to previous network baselines 2: 1) AVE [1]: an audio-guided co-attention network with additional audio-visum parsing branches; 2) AVSDN [2]: a
model with additional audio-visum parsing branches to merge global audio-visual features into local ones; 3) HAN [3]: a hybrid attention network with multi-modal multiple instance learning pooling. Furthermore, we compare MA [4], method based on HAN with the audio visual contrastive learning (C) and the label refinement
NNot that we will not include the recent work [5] into comparison since there is no source code for it. Meanwhile, they are based HAN [3]. but our MGN is fully novel backbone.

---

GT
Vacuum cleaner
Visual
HАN
MA
V
GT
Speech
Audio
HAN
MGN
Figure 2: Qualitative comparisons with baselines. "V_c" denotes the "Vacuum "hccuummcleanrr" class. The proposed MGN achieves much better performance parsing audio and visual events. Note that the audio-visun event prediction is an intersection of audio and visual event predictions.
(R). The MA series baselines include MA (w C). MA (w R), and MA (w C+R). The quantitative comparisons the LLP dataset reported in Table 1.
As can seen, the proposed MGN achieves the overall best results against previous network baselines in terms most of metrics. For the overall evaluation of segment-level predictions, achieve significant performance gains of 1.6 Type@ AV and 1.8 Event @ AV. When evaluated on segment-level predictions of each sample, MGN also improves the baseline by large margins, 2.6 Visual and 1.7 Audio-Visual. Meanwhile, our MGN outperforms baselines by 3.5 Visual, 1.4 Audio- Visual, and 1.6 Tyep @ AV for event-level predictions. These results demonstrate the effectiveness of our approach in supervied audio-visua video parsing against prior network architectures.
Furthermore, significant gains can be observed in the setting of using the audio-visual contrastive learning and label refinement. Adding the contrastive learning to our MGN achieves the segment-level performance gain of 3.6 Visual and 2.8 Audio- Visual, and the event-level gain of 3.8 Visual and 2.6 Audio- Visual. With the label refinement in MA, we significantly improve MA (w R) by 3.1 segment-level and 4.0 event-level for visual predictions. Our framework with both contrastive learning and label refinement achieves the best segment-level performance in terms of Visual, Audio- Visual, Type AV, and Event @ AV. These improvements imply the strong generalizability of the proposed MGN to the audio- visual contrastive learning and the label refinement.
In order to qualitatively evaluate the predictions of audio and visual events, we compare the proposed MGN with HAN [3], MA [4] in Figure We can observe three main things from previous approaches: First, existing methods can miss the snippet-wise predictions for some hard segments in the video, e.g., MA does not predict the vacuum cleaner for visual events after 3s. Second, prior work can miss one class prediction if there are two categories happening in the same modality. Third, the previous method can miss one modality prediction for the class existing in both modalities. When it comes to the proposed MGN, we superiorly achieve high Fl scores of segment-level and event-level prediction. benefiting from the well-designed class-aware unimodal grouping and modality-awar cross-modal grouping modules.
4.3 Experimental Analysis
In this section, we conducted ablation studies to validate the benefit of Class-aware Unimodal Grouping (CUG) and Modality-aware Cross-modal (MCG) Grouping strategies. We also performed extensive experiments to investigate the false positive issue in HAN and the learned class tokens.
7

---

Table 2: Ablation studies on Class-awar Unimodal Grouping (CUG) and Modality- Cross-
modal Grouping (MCG) blocks. Segment-level audio-visual video parsing results are reported.
CUG
MCG
Audio
Visual
Audio- Visual Type@AV Event@ AV
X
x
60.1
52.9
48.9
54.0
55.4
x
58.9
55.3
49.8
54.7
55.9
/
60.7
55.5
50.6
55.6
57.2
Class-aware Unimodal Grouping & Modality-aware Cross-modal Grouping. In order to demon- strate the effectiveness of the proposed class-aware unimodal grouping (CUG) and modality-aware cross-modal grouping (MCG), we ablated the necessity and strategy of grouping blocks. The results of segment-leve predictions are reported in Table 2. We can observe that adding CUG to the vanilla baseline achieves significant gains of 2.4 Visual, indicating the effectiveness of grouping class-aware semantics predicting snippet-wise categories for visual events. Incorporating MCG with CUG highly increases Audio-Visual. Tyep @ AV, Event @ AV by 1.7, 1.6 and 1.8. These results show the importance of modality grouping on predictions of audio-visual events. Besides effectiveness, model is also more efficient. When the depth of CUG and MCG is 3 and 6, the proposed MGN with only 47.2% parameters of the vanilla baseline performs the best on Type Svp@AAV AV and Event AV, especially Audio. These results further show the advantage of our MGN in real applications with lightweight parameters against the prior work [3, 4]. The detailed results are in Appendix.
False Predictions. In order to demonstrate the effectiveness of the proposed MGN in mitigat- ing false predictions against baselines, calcu- late the total amount of false positives for all 25 classes in the test set. The comparison results of event-level audio, visual and audio- visual met- rics shown in Figure 3. We observe that our MGN with the class-aware unimodal grouping modules decreases the false positives of audio and visual events by large margins, 381 and 494. Furthermore, the number of false pos- itives of audio-visual events drops down by 678, which verifies the importance modality-aware cross-modal grouping in mitigating the modal- ity uncertainty. Overall, our MGN with explicit grouping mechanisms significantly eliminates false predictions caused by the modality and temporal uncertainties existing in the baseline.
1500
HAN
1250
MGN
1000
750
Event A
Event V
Event AV
Figure 3: Comparison results of the total amount of false positives for all 25 classes between HAN [3] and the proposed MGN terms of event-level au- dio, visual and audio-vise visual metrics, i.e., Event A, Event_ V, and
Learned Class-aware Features. The learned class tokens are essential to grouping class-aware semantics from audio and visual features. To better evaluate the quality of those learned class-level features, we visualize the learned audio and visual representations 25 categories by t-SNE [42], as shown in Figure 4. It noted that each spot denotes the feature of one audio or visual event, while each color represents each class, such as "Speech" in brown and "Dog" in green. As can be seen in the last column, features extracted by the proposed MGN are intra-class compact and inter-class separable. However, there still exists mixtures of multiple categories for audio and visual events among the representations of HAN and MA. For the fub-figure on the bottom right, we can observe large cluster of brown spots for the "Speech" class of audio events in the test set, while brown spots prior work are distributed more sparsely. These meaningful visualizations further demonstrate that our MGN successfully learns compact and discriminative features for each modality.
4.4 Limitation
Although the proposed MGN achieves superior results on visual events and audio-visual events, the gains of audio events are not significant compared to the visual modality. We notice that there are 1628 visual instances and 2663 audio instances in the test set, which implies the audio modality is much harder than the visual one this setting. One possible solution to introduce small number of segment- beggeettwiis audio-visual parsing annotations supervision for sunirvipee training, In the

---

TAN
MGN lours)
Visual
Mongeeegg
Audio
Figure 4: Qualitative visualizations of audio (Top rows) and visual (Bottom rows) features learned by HAN, MA and the proposed MGN. Note that each spot denotes the feature of one audio visual event, while each color represents each class, such as "Speech" in brown and "Dog" in green.
meanwhile, our MGN performs worse with the increase the depth of transformer layers in grouping modules. This is caused by such weakly-supervisee setting with only video-level annotations that do indicate either segments or modalities. However, the model is expected parse video into events with different categories and modalities. Therefore, the potential future work is to add more grouping stages with learned class-tokens as supervision for each one.
Conclusion
In this work, present MGN, fully novel Multi-modal Grouping Network to explicitly group class-aware matching semantics for suueelysupervieed audio-visual video parsing. We introduce the class-aware unimodal grouping module to generate class-aware unimodal representations with learnable tokens by using unimodal grouping blocks for each modality. Furthermore, we leverage the modality-aware cross-modal grouping to match the video-level target with the cross-modal grouping blocks. Experimental results demonstrate the effectiveness and superiority our MGN against previous baselines. We also show the generalizability of our simple framework the audio-visual contrastive learning and label refinement.
Broader Impact. The proposed method detects video events in audio and visual modalities based on the learned statistics of the training dataset. It could capture internal biases in the data, which may have negative socictal impacts. For example, the model might not be able to discover rare but important events, such as fire warnings. Thus, before deploying our audio-visual video parsing model into real-world applications, need to carefully address these issues.

---

References
[1] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of European Conference Computer Vision (ECCV), 2018.
[2] Yan-Bo Lin, Yu-Jhe Li, and Yu-Chiang Frank Wang. Dual-modality seq2seq network for audio-visual event localization. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2002-2006, 2019.
[3] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly- supervised oudio-visual video parsing. In Proceedings of European Conference on Computer Vision (ECCV), page 436-454, 2020.
[4] Yu Wu and Yi Yang. Exploring heterogeneous clues for parsing. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 1326-1335 2021.
audio-visual video
[5] Yan-Bo Lin, Hung- Yu Tseng, Hsin- Ying Lee, Yen- Yu Lin, and Ming- Hisng-ssaan Yang. Exploring video and cross- moddaity signals for supervised audio-visual video parsing. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2021.
[6] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2016.
[7] Andrew Owens, Jiajun Wu, Josh H. McDermott, William T. Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 801-816, 2016.
[8] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pages 609-617, 2017.
[9] Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2018.
[10] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuun Yang, and In So Kweon. Learning localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition (CVPR), pages 4358-4366, 2018
[11] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European Conference on Computer Vision (ECCV), pages 570-586, 2018
[12] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 1735-1744, 2019.
[13] Chuang Gan, Deng Huang, Hang Zhao, Joshua B. Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. In /EEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10478-10487, 2020.
[14] Pedro Morgado, Yi Li, and Nuno Nvasconcelos. Learning representations from audio-visual spatial alignment. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 4733-4744, 2020.
[15] Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimina- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 12945, 2021.
[16] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12475-12486, June 1021.
10

---

[17] Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360evideo. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2018.
[18] Ruohan Gao and Kristen Grauman. 2.5d visual sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 324-333, 2019.
[19] Changan Chen, Unnat Jain, Carl Schissler, S. V. A. Garí, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In Proceedings of European Conference on Computer Vision (ECCV), pages
17-36, 2020.
[20] Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In Proceedings of the European Conference on Computer Vision (ECCV), pages 35-53, 2018.
[21] Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3879-3888, 2019.
[22] Ruohan Gao. Tae- Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10457-10467, 2020.
[23] Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2745-2754, 2021.
[24] Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross- modal consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15495 15505, 2021.
[25] Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9248-9257, 2019.
[26] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self- supervised learning of audio-visual objects from video. In Proceedings of European Conference on Computer Vision (ECCV), pages 208-224, 2020.
[27] Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In Proceedings of European Conference on Computer Vision (ECCV), pages 292-308, 2020.
[28] Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16867-16876, 2021.
[29] Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. arXiv preprint arXiv:2203.09324 2022.
[30] Yake Wei, Di Hu, Yapeng Tian, and Xuelong Li. Learning in audio-visual context: A review, analysis, and new perspective. arXiv preprint arXiv:2208.09577 2022.
[31] Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang. Dual attention matching for audio-visual event localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 6291-6299, 2019.
[32] Yan-Bo Lin and Yu-Chiang Frank Wang. Audiovisual transformer with instance attention for audio-visual event localization. In Proceedings of the Asian Conference on Computer Vision (ACCV), 2020.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). Curran Associates, Inc., 2017.
11

---

[34] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. arXiv preprini arXiv:2202. 1094, 2022.
[35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[36] Jia Deng, Wei Dong, Richard Socher, Li-Jia. Li, Kai Li, and Li Fei-Fei. ImageNet: A Large- Scale Hierarchical Image Database. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-2555 2009.
[37] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2018.
[38] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 6299-6308, 2017.
[39] Shawn Hershey. Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Chan- ning Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin W. Wilson. CNN architectures for large-scale audio classification. In Proceedings of IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2017.
[40] Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Chan- ning Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
[41] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[42] Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):257- 2008.
[43] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax.
In Proceedings of International Conference on Learning Representations (ICLR), 2017.
[44] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: continuous relaxation of discrete random variables. In Proceedings of International Conference on Learning Representations (ICLR), 2017.
12


Reviews:
[{"review_id": "lZByZKH5iqP", "reviewer": "Reviewer_f8HF", "summary": "This paper addresses the problem of predicting event labels over time in audio visual data with weak labels.  A method is proposed that uses attention between audio visual features and learned class embeddings, and extracts class-specific embeddings for use in event detection.  The model is evaluated on the Look, LIsten, and Parse dataset, using 11K video clips with video labels for training, and around 2K video clips with both audio and video labels for evaluation.  ", "questions": "See Weaknesses above for some questions. ", "limitations": "The authors acknowledge that performance on audio events is not improved.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\nThe proposed method is an interesting idea, and seems to help with the task. \nSmall improvements are achieved on a challenging task evaluated on a sufficiently large test set. \n\nThe ablations in Table 2 are informative.  \n\nWeaknesses: \nThe performance on audio events is not improved.\n\nThe analysis of false positives in Figure 3 is useful, but it seems like the models are just biased to different operating points, since their F-scores in table 1 are similar.  So the reduction of false positives in Figure 3 must come at the cost of a decrease in recall.   It would be better to show both numbers, or calibrate to the same operating point. \n\n\nClarity: \n\nThe paper is a bit difficult to understand due to notational issues\n\n> These methods without explicit grouping suffer from false predictions due to the modality and temporal uncertainties.\n\nIt is not clear to me why that should be at this point in the paper.  Is this an empirical finding or a hypothesis?\n\nRe: equation (6), cross entropy is usually from labels to estimates, so I would expect $CE(p,\\\\hat{p})$, but here it seems to be used the reverse way.  Also this looks like a vectorized version with binary probabilities in the elements.  It might be good to define it since technically cross entropy is defined over a distribution, rather than a vector of distributions.  That is, I think what is meant is something like $CE(\\\\mathbf{y}, \\mathbf{p}) = \\\\sum_i CE(y_i, p_i)$.  If not please clarify. \n\nIn the cross-modal attention $\\\\phi_{ca}(q,K,V)$, my understanding is that the key and value should be of the same modality, and the query of the other modality.  \nSo if I understand correctly, in equation (1)  $\\\\phi_{ca}(f^v_t, F^v, F^a)$  is probably supposed to be $\\\\phi_{ca}(f^a_t, F^v, F^v)$ , and in equation (2) $\\\\phi_{ca}(f^v_t, F^a, F^v)$  should be $\\\\phi_{ca}(f^v_t, F^a, F^a)$.\n\nFrom equation (7) on, the notation for self-attention changes without warning from a three-argument function to a one argument function $\\\\phi_{sa}(X)$, which I assume in the notation of (3) would be $\\\\{\\\\phi_{sa}(x_i, X, X)\\\\}_i$, but it would be kinder to the reader to just define it. \n\nIn equation (10), it is not clear why learned weights are needed to transform both the class tokens and the modality specific features.  Is it not equivalent to just transform the features?  That is $Ax \\\\cdot By = x^T A^T B y = B^T A x \\\\cdot  y = W x \\\\cdot y $, where $W = B^T A$.     \n\nIn equation (12), the notation is not clear.   The $\\\\mathbf{1}$ is not clear to me.  I was expecting to see a label for the presence of the class.  I'm probably misunderstanding something, but in any case, I think this needs to be better explained. Perhaps writing out the CE formula element by element would clarify.   Also, maybe (13) should come before (12) for clarity. \n\n\nErrata:\n\n> In order to explicitly group~~ing~~ class-aware...\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "LNfHalPt0y9", "reviewer": "Reviewer_KNZ9", "summary": "This paper tackles weakly-supervised audio-visual video parsing, and proposes a Multi-modal Grouping Network to explicitly group class-aware matching semantics with class-aware unimodal grouping module and modality-aware cross-modal grouping module. Experimental results are shown on the Look, Listen and Parse (LLP) Dataset with the generalizability to the audio-visual contrastive learning and label refinement.", "questions": "1. L58: \"We propose a new audio-visual video parsing baseline.....\"     Why is the proposed model called a \"baseline\"?\n2.  For the results in Table 1, the best result for \"Event-Level\"-\"A\" is not highlighted. Why?", "limitations": "The authors list the model limitations of limited data, and worse performance with deeper network, and point out the possible solutions of semi-supervised training and incorporating more intermediate supervision. The authors also point out the case of rare events in real deployment, which is practical and has corresponding techniques focusing on the case.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths:\nThis paper tackles weakly-supervised audio-visual video parsing considering the asynchronous possibility of the two modalities.\n\n- Weaknesses:\n1. The introduction of this paper is not strongly motivated, but only with general and brief description. The introduction lists several related papers but the relations of these papers to the proposed model are not well illustrated.\n\n2. L63: \"The experiments can demonstrate the superiority of our MGN over state-of-the-art AVVP approaches and its generalizability to contrastive learning and label refinement.\"     When the authors fist mention \"contrastive learning and label refinement\" [4] in the introduction, these two concepts are not explained but directly appear with citation to another paper, which makes the paper reading confusing.\n\n3. For the results in \"Table 2: Ablation studies on Class-aware Unimodal Grouping (CUG) and Modality-aware Cross-modal Grouping (MCG) blocks, each of CUG block and MCG block is ablated as a whole. What about the effects of the design choices in each block? E.g. the effect of concatenating class labels in Class-aware Unimodal Grouping (CUG) block?\n\n4. Only one dataset is experimented on.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IhT0cGJ_okD", "reviewer": "Reviewer_kAVj", "summary": "The authors present a new baseline architecture for video parsing. Using learnable categorial embedding tokens they propose class-aware unimodal grouping network in conjunction with a cross-modal grouping network to time-stamp audio, visual and audio-visual events using only video level labels. They show improved results compared to other baselines on LLP dataset. ", "questions": "- The model uses pre-trained ResNet and VGGish net for model input, both of which are very good feature extractors. What is the importance/requirement of using such feature extractors on the final result. More specifically, LLP dataset is derived from Audioset and VGGish net is trained in a supervised fashion using AudioSet, which makes it an ideal feature extractor. Can the author's comment if the unimodal grouping network would still be effective if not for such feature extractor.  \n- How does the model perform with event scaling ? (Any reason to limit the study to LLP and not use full AudioSet)?\n- Regarding implementation (line 188), How low can the time-resolution go? (ie., instead of 1s snippets, can 0.1s snippets be used?)\n- in Table2, why does using CUG only makes “Audio” result worse when compared to not using ?  (row1 vs row2)", "limitations": "see *Questions* section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The manuscript discusses the important problem of video parsing with very coarse labels. It is well written and easy to read through. The limited number of experiments presented are through and insightful. The proposed method if effective compared to other baselines as evident from the qualitative and quantitative results. However, it is heavily inspired from recently proposed GroupViT for image segmentation. Though the authors outline the differences between proposed approach and GroupViT in supplementary material, they are very nuanced and not significant.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept