Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper tackles weakly-supervised audio-visual video parsing, and proposes a Multi-modal Grouping Network to explicitly group class-aware matching semantics with class-aware unimodal grouping module and modality-aware cross-modal grouping module. Experimental results are shown on the Look, Listen and Parse (LLP) Dataset with the generalizability to the audio-visual contrastive learning and label refinement.
strengths_and_weaknesses: - Strengths:
This paper tackles weakly-supervised audio-visual video parsing considering the asynchronous possibility of the two modalities.

- Weaknesses:
1. The introduction of this paper is not strongly motivated, but only with general and brief description. The introduction lists several related papers but the relations of these papers to the proposed model are not well illustrated.

2. L63: "The experiments can demonstrate the superiority of our MGN over state-of-the-art AVVP approaches and its generalizability to contrastive learning and label refinement."     When the authors fist mention "contrastive learning and label refinement" [4] in the introduction, these two concepts are not explained but directly appear with citation to another paper, which makes the paper reading confusing.

3. For the results in "Table 2: Ablation studies on Class-aware Unimodal Grouping (CUG) and Modality-aware Cross-modal Grouping (MCG) blocks, each of CUG block and MCG block is ablated as a whole. What about the effects of the design choices in each block? E.g. the effect of concatenating class labels in Class-aware Unimodal Grouping (CUG) block?

4. Only one dataset is experimented on.




questions: 1. L58: "We propose a new audio-visual video parsing baseline....."     Why is the proposed model called a "baseline"?
2.  For the results in Table 1, the best result for "Event-Level"-"A" is not highlighted. Why?
limitations: The authors list the model limitations of limited data, and worse performance with deeper network, and point out the possible solutions of semi-supervised training and incorporating more intermediate supervision. The authors also point out the case of rare events in real deployment, which is practical and has corresponding techniques focusing on the case.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors present a new baseline architecture for video parsing. Using learnable categorial embedding tokens they propose class-aware unimodal grouping network in conjunction with a cross-modal grouping network to time-stamp audio, visual and audio-visual events using only video level labels. They show improved results compared to other baselines on LLP dataset. 
strengths_and_weaknesses: The manuscript discusses the important problem of video parsing with very coarse labels. It is well written and easy to read through. The limited number of experiments presented are through and insightful. The proposed method if effective compared to other baselines as evident from the qualitative and quantitative results. However, it is heavily inspired from recently proposed GroupViT for image segmentation. Though the authors outline the differences between proposed approach and GroupViT in supplementary material, they are very nuanced and not significant.
questions: - The model uses pre-trained ResNet and VGGish net for model input, both of which are very good feature extractors. What is the importance/requirement of using such feature extractors on the final result. More specifically, LLP dataset is derived from Audioset and VGGish net is trained in a supervised fashion using AudioSet, which makes it an ideal feature extractor. Can the author's comment if the unimodal grouping network would still be effective if not for such feature extractor.  
- How does the model perform with event scaling ? (Any reason to limit the study to LLP and not use full AudioSet)?
- Regarding implementation (line 188), How low can the time-resolution go? (ie., instead of 1s snippets, can 0.1s snippets be used?)
- in Table2, why does using CUG only makes “Audio” result worse when compared to not using ?  (row1 vs row2)
limitations: see *Questions* section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses the problem of predicting event labels over time in audio visual data with weak labels.  A method is proposed that uses attention between audio visual features and learned class embeddings, and extracts class-specific embeddings for use in event detection.  The model is evaluated on the Look, LIsten, and Parse dataset, using 11K video clips with video labels for training, and around 2K video clips with both audio and video labels for evaluation.  
strengths_and_weaknesses: 
Strengths:
The proposed method is an interesting idea, and seems to help with the task. 
Small improvements are achieved on a challenging task evaluated on a sufficiently large test set. 

The ablations in Table 2 are informative.  

Weaknesses: 
The performance on audio events is not improved.

The analysis of false positives in Figure 3 is useful, but it seems like the models are just biased to different operating points, since their F-scores in table 1 are similar.  So the reduction of false positives in Figure 3 must come at the cost of a decrease in recall.   It would be better to show both numbers, or calibrate to the same operating point. 


Clarity: 

The paper is a bit difficult to understand due to notational issues

> These methods without explicit grouping suffer from false predictions due to the modality and temporal uncertainties.

It is not clear to me why that should be at this point in the paper.  Is this an empirical finding or a hypothesis?

Re: equation (6), cross entropy is usually from labels to estimates, so I would expect $CE(p,\\hat{p})$, but here it seems to be used the reverse way.  Also this looks like a vectorized version with binary probabilities in the elements.  It might be good to define it since technically cross entropy is defined over a distribution, rather than a vector of distributions.  That is, I think what is meant is something like $CE(\\mathbf{y}, \mathbf{p}) = \\sum_i CE(y_i, p_i)$.  If not please clarify. 

In the cross-modal attention $\\phi_{ca}(q,K,V)$, my understanding is that the key and value should be of the same modality, and the query of the other modality.  
So if I understand correctly, in equation (1)  $\\phi_{ca}(f^v_t, F^v, F^a)$  is probably supposed to be $\\phi_{ca}(f^a_t, F^v, F^v)$ , and in equation (2) $\\phi_{ca}(f^v_t, F^a, F^v)$  should be $\\phi_{ca}(f^v_t, F^a, F^a)$.

From equation (7) on, the notation for self-attention changes without warning from a three-argument function to a one argument function $\\phi_{sa}(X)$, which I assume in the notation of (3) would be $\\{\\phi_{sa}(x_i, X, X)\\}_i$, but it would be kinder to the reader to just define it. 

In equation (10), it is not clear why learned weights are needed to transform both the class tokens and the modality specific features.  Is it not equivalent to just transform the features?  That is $Ax \\cdot By = x^T A^T B y = B^T A x \\cdot  y = W x \\cdot y $, where $W = B^T A$.     

In equation (12), the notation is not clear.   The $\\mathbf{1}$ is not clear to me.  I was expecting to see a label for the presence of the class.  I'm probably misunderstanding something, but in any case, I think this needs to be better explained. Perhaps writing out the CE formula element by element would clarify.   Also, maybe (13) should come before (12) for clarity. 


Errata:

> In order to explicitly group~~ing~~ class-aware...



questions: See Weaknesses above for some questions. 
limitations: The authors acknowledge that performance on audio events is not improved.
review_title: null

Meta-review: metareview: The authors propose an approach for weakly supervised audio-visual parsing of videos. They propose using learnable categorical embedding to do class-aware unimodal grouping, combined with cross-modal grouping to time-stamp audio, visual and audio-visual events using only video level labels.

Based on the feedback provided by the reviewers, especially since Reviewer KNZ9 increased their score to Borderline Accept after the rebuttal period, we recommend this paper for publication at NeurIPS 2022. 

The reviewers had some concerns about the paper. Reviewer KNZ9 mentioned that the relations of the listed papers to the proposed model were not well explained -- they also had some concerns about the experimental results, and the fact that only one dataset was used in the evaluation. Reviewer kAVj had questions about model performance with event scaling, and time resolution lower bounds. Reviewer f8HF commented on the difficulty in following the notation in the paper, and pointed out the results on audio events is not improved.

We thank the authors for addressing the comments of the reviewers in their review during the author feedback period. The authors seem to have addressed some of the concerns/feedback from the reviewers with detailed discussions -- it would be good to include these discussions, as much as possible, in the updated paper or supplemental materials.
confidence: Certain
Decision: Accept