Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a procedure for weakly supervised audio-visual semantic parsing, an interesting (relatively new) audio-visual task, where the goal is to predict and localize both audio and visual events, while leveraging only video level (ie non localized) annotations for training. 

The paper presents two main contributions. The first is the introduction of the so-called "cross-modality co-occurrence module" and the second is the observation that a cross-modal weakly-supervised contrastive loss enhances audio-visual parsing, by aligning video-level audio and visual representations of semantically similar samples. One of the main concerns about this paper is with respect to the motivation and experimental evaluation of the first contribution (see main review section).

The paper achieves good performance on the audio-visual parsing task, outperforming prior state-of-the-art by significant margins.

------

Post rebuttal:

After carefully considering the comments of all reviewers and the rebuttal, I will increase my rating to 7. The paper makes progress in an interesting and relatively recent audio-visual understanding problem. The proposed approach is sound and achieves significant gains over prior methods. The authors also addressed my main concerns in the rebuttal, and I strongly encourage the authors to add the new results to the paper, as well as the distinctions between the proposed method and [46].
main_review: While I'm overall positive about the paper (due to the reasons outlined above), I hope the authors can address the following concerns. I will reassess my recommendation based on the responses.

1) It was not clear to me why the module introduced in section 3.2 is called cross-modality co-occurrence (especially the co-occurrence part). From equations 4-5, it seems that this module computes "class-wise" features, and then performs aggregation over the class dimension using attention.

2) What are the parameter and FLOP count for the "cross-modality co-occurrence" (CM-Co) module? If this is significant, it may not be fair to compare prior work like MA or HAN to this method. In other words, it may not be possible to conclude that the reason why CM-Co works is that it provides a better inductive prior for the model to generalize faster, or is it only due to the higher parameter or FLOP counts. One possibility to control for this would be to implement HAN and MA with a feature aggregation module twice as deep.

3) What differences between the CM-Co and [46] should be highlighted in the paper. As is, it is unclear what are the differences, if any?
limitations_and_societal_impact: This paper aims to improve audio-visual semantic parsing (an established research area). As such, it does not introduce new negative societal impacts.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper describes a method for audiovisual event parsing. It proposes a method based on attention based features aggregation along with cross-modal co-occurrences. A contrastive learning based loss is also added to encourage features across modalities to be similar for the same event. Experiments are done on a single audiovisual dataset - LLP dataset -- which is an audiovisual dataset of primarily sound events.  
main_review: The paper proposes methods for audio-visual event recognition and detection. It is primarily an attention driven method in which attention mechanisms are used for feature aggregation as well as for learning audio-visual correlations.  

1. The method is primarily based on prior works [4] and [6]. Taking these two works into consideration, the method does not seem particularly novel and a minor addition over these works. The attention based feature aggregation is based on these prior works. The contrastive learning loss which is aimed at encouraging similarities between audio and visual features for the same event also seems to be based on a similar loss in [6]. So the main addition is cross-modality co-occurrence, in which the key differentiating factor is class-specific formulation. Otherwise, it is mere repetition of the feature aggregation process (another block of layers doing the same thing as feature aggregation). 


2. The class-specific attention in Sec 3.2 leads to learning  2xC (#num classes) matrices of size d x d_c. This can easily blow up as C increases, even if it reaches a few hundred ? The LLP dataset used in the experiments is derived from Audioset which itself has 500+ classes. I would like to hear authors comment on how it will scale as C becomes large. Also what if we dont do it in a class-specific manner ? How much worse that is compare to the current formulation ? 


3. How are the weights w’s in Eq 3 computed ? Its not clear from the text. 


4. There are plenty of works on audio-visual learning with different forms of attention - , temporal, self, cross-modal etc. R1 and R2 below to mention just a couple. There are likely more. The paper does not really put itself against these other formulations of audio-visual attention and does not explain why and how its different and perhaps better.  Authors should refer to these works and some sort of technical or empirical comparison is expected. R1 for example applies a relatively simpler attention mechanism for weakly supervised sound event recognition. It shows experimental results on full Audioset which is the source of the LLP dataset used here in experiments. 


5. The empirical evaluations and analyses are rather weak. Experimental results on only 1 dataset are shown. Not much insight is provided into the framework. Like when cross-modal is important when it is not. What type of weights do we learn for different classes - are they meaningful ? What if in Eq 5 only self-modal attention (without the visual  terms for audio and vice versa) ? The qualitative analysis is also not providing anything interesting. The ground truths for both classes are themselves spread over the whole video duration which is not really an interesting case to get an idea of localization capabilities. It would have been more interesting to see how well the method can localize when the events starts and end somewhere in the recording. 


[R1] Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data, IJCAI 2020. 

[R2] Cross-Attentional Audio-Visual Fusion for Weakly-Supervised Action Localization, ICLR 2021. 

limitations_and_societal_impact: The authors have not pointed out any specific limitations. Audiovisual learning in general can have negative societal impact when not carefully done. However, in the context of current paper there might not be obvious critical ones.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors address weakly-supervised audio-visual video parsing problem, which aims to temporally parse a video into audio, visual, and audio-visual events learning from video-level labels. To improve parsing performance, they propose to explore additional cross-video and cross-modality supervisory signals. Experiments on the LLP dataset can validate the effectiveness of the proposed method. 

**Post rebuttal:**

I've read comments from fellow reviewers and the rebuttal from the authors. 

In this paper, the authors propose an interesting method to address a new audio-visual video understanding problem: audio-visual video parsing. Two key technical contributions: co-attention with class-level representation and audio-visual contrastive objective are well motivated and technically sound. Experimental results show that it can outperform very recent state-of-the-art methods. 

I had some concerns about the limitations of the proposed loss term and missing experimental results. But, the rebuttal can successfully address my major concerns. Thus, I would like to upgrade my rating to 7.  
 
**To authors:** 

I do encourage the authors to add the discussion about the limitations of the proposed method and new results to the paper. 

main_review: Pros:

+ The proposed methods are well motivated and technically sound. With class-level audio and visual features, the co-attention mechanism can explicitly model the relationship between event categories cross different modalities. With the proposed contrastive objective, it will tend to encourage features across modalities with the same event category to be close and those with different events to be far away from each other. 
+ Extensive experiments on the LLP dataset validate that the proposed method is superior over recent state-of-the-art audio-visual video parsing approaches.

Cons:

-  In Eq. 8, the proposed contrastive objective encourages audio and visual features from two videos with at least one overlapping category to be close and those with non-overlapping categories to be far away from each other.  However, audio and visual tracks in a video can contain totally different events and the categories in the video-level labels could be from either single modality (audio or visual) or both. The loss will introduce learning noises when the overlapping categories only appear in the same single modality and two videos have no similar cross-modal content. For example,  given two videos A and B and their labels are {dog, speech} and {dog, guitar}, respectively, and dog is an audio event in A and B. Clearly, they have the same dog category. The loss term will push audio feature from A be close to visual feature from B. However, the dog event is an audio event in B. It means that the visual feature of B does not have any information about dog. Then, the loss term will introduce learning noises. 

- The MA method also used contrastive learning (CL) to improve audio-visual video parsing. The authors mentioned that the CL method in MA is based on a strong assumption that audio and visual signals are synchronized and the proposed CM-S will have no the problem. However, there are no stratforward experimental results to verify the superiority of the proposed CM-S over CL in MA. To validate the effectiveness of the proposed method: CM-S, the authors should compare CM-S with the CL method in MA using HAN and MA as baselines in ablation study. 

- Why HAN + Ours achieves even worse audio parsing results than HAN? It seems that the proposed method mainly improve visual parsing .  The authors should provide more analysis and explanation on the results. 

- The contrastive objective is computed by pairing all videos in training batches. Whether it will significantly increase training cost? 

Overall, I think the proposed method is technically sound. But, I still have some concerns about the method and experiments.  

limitations_and_societal_impact: It would be good if the authors can provide some failure examples in the paper and analyze the limitations of the proposed method.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors introduce an interesting audio-visual video parsing framework method in a weakly-supervised manner, which contains two part of *the shared cross-modality semantics module* and *the cross-modality co-occurrence module*. Specifically, the former leverages common and diverse event semantics across videos to learn robust cross-modality representations that facilitate models to identify differently modalities’ events. And the latter joint considers the relationship of event categories, which helps localize segments of target events and can exclude irrelevant ones by performing self-attention and cross modality co-attention on class-wise features. Additionally, the experiments conducted on the LLP dataset demonstrate the proposed model obtained the most advanced results in downstream classification with various settings.
main_review: Originality: This audio-visual video parsing(AVVP) task proposed by Tian et al[a] in 2020, which aims at providing a detailed analysis of audio, visual, and audio-visual events in videos without such alignment assumptions. The method in this paper is unique, but it is more like a combination of existing technologies[a][b]. Practically, relative to existing random sampling sound and images, the proposed method selects both related and irreverent videos to explore common semantics and discern dissimilar events. And the method can exploit videos in a training batch to extract their common semantics for a categorical event and discern unrelated clues. In addition, most of the related work are cited.

Quality: The submission sounds technically, and it's well supported by experimental results, and the methods used are appropriate. This is a complete piece of work，and the authors are careful and honest about evaluating both the strengths and weaknesses of their work

Clarity: The writing and organization of the submission is relatively clear, but some content and existing methods have too much overlap, mainly reflected in *Revisit of Weakly-Supervised Audio-Visual Video Parsing*.

Significance: The experimental results have a positive effect on related downstream tasks, and others researchers likely to use the ideas or build on them. The submission provide a new exploration for solving the audio-visual video parsing task, and to a certain extent solve the task better than the previous work. And the current model is the state-of-the-art by the experimental results, and some ablation experiments have been used to prove it.

[a] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly supervised audio-visual video parsing. In *ECCV*, 2020.
[b] Yu Wu and Yi Yang. Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In *CVPR*, 2021

Post Rebuttal:

The authors' response has partially addressed my previous concerns, such as the difference between this work and existing technologies [4, 6], as well as some method details. Overall, I think this is an incremental work on Weakly-Supervised Audio-Visual Video Parsing, where the main contribution is as what the authors emphasize, "the conventional contrastive learning is conducted over frame pairs of the same video, while contrastive learning in Eq. 8 of this work is applied to video pairs" in the weakly-supervised learning scheme. The proposed methods is somewhat similar to the previous ones, except that they utilize the video-pairs annotated by video-level annotations. Based on the above considerations, I decide to hold my previous rating of 6.
limitations_and_societal_impact: The paper still has some limitations, which are mainly reflected as follows:

- How to ensure that at least two of the n videos in each batch are related? Have you considered the extreme case where all videos in a batch are irrelevant?
- Since the video is weakly labeled, it is impossible to determine the specific moment when the labeled event occurs. At each time $t$ of the class-level operation, the corresponding label may not occur at that time. So, how to solve this problem?
- The Cross-modality Co-Occurrence module in Figure 1 has some minor problems, which is the output of the example is the probability of $\hat{p}^a_t$, but it is indeed $Query$ made by visual on the audio feature. I think this is the author accidentally wrote the opposite.
- There are too many similarities between the Preliminaries part of the article and the literature [c], and it is recommended to adjust.
- It is recommended to bold the best experimental results in Table 3.

[c] Yu Wu and Yi Yang. Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In *CVPR*, 2021.
review_title: null

Meta-review: The paper describes a method for audiovisual event parsing. It proposes a method based on attention based features aggregation along with cross-modal co-occurrences. A contrastive learning based loss is also added to encourage features across modalities to be similar for the same event. There was some disagreement between reviewer scores (three positives one mildly negative), although not for the reasons behind those scores, The reviewers agreed that the paper addresses a recent and interesting problem - weakly supervised audio-visual semantic parsing, where the goal is to predict and localize both audio and visual events, while leveraging only video level (ie non localized) annotations for training - , that the paper is technically sound, and that the approach produces good results. The main limitation was that there is limited technical novelty. While the solution is novel, it is obtained by combination of variants on previously used building blocks. After discussion, it was concluded that the paper addresses an interesting problem in an interesting way. Although somewhat incremental, the paper deserves publication.
Decision: Accept (Poster)