Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
Summary

The authors propose a novel approach to audio-visual action localization.  They introduce a multi-stage cross attention approach to fuse audio-visual features and a consistency loss to enforce temporal continuity.

Strengths

The approach is well motivated and the technical contribution is clear.  The paper is well written and the contributions are evaluated against a number of recent SOTA approaches on two datasets for action localization.

Weaknesses/Concerns

I would like to see a discussion on the drop in performance in the 3-stage model compared to the 2-stage and 1-stage models in Table 1.  This drop is not intuitive so the authors should provide some analysis here.

In section 4.3 and Table 2, it seems O-I, O-II and O-III are used before being defined.  Are these results with the 2-stage model?  I would like to see the effect of using the different stage models with the different versions of the loss function.  This will help understand the different contributions to the final performance.

The model introduces increased computation and additional parameters.  What is the increase in computation?  This could be important for detection systems that prioritize a trade-off of performance and efficiency.

Recommendation and reasoning

This is a good paper that could be improved with some additional experiments/analysis and improved clarity.  I recommend acceptance.
review_title: The authors propose a novel approach to audio-visual action localization.  They introduce a multi-stage cross attention approach to fuse audio-visual features and a consistency loss to enforce temporal continuity.  It is a good paper that could be improved with some additional experiments/analysis and improved clarity.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a method for predicting the class of an action in a video and localizing the range.
The focus is on using both video and audio modalities.
They use a cross modal "multi stage" attention mechanism, a background classifier, and losses designed to encourage temporal consistency in the output.

Reason for Score:

I would accept this paper.
The model shows good performance for weakly labelled, multi-model action recognition and localization.
The method is well described and would have immediate benefit in practical applications.

Pros:

Weakly supervised - avoiding the cost of detailed labelling is very important.
Multi Modal - Table 1 shows great advantage of using both modalities, and audio is nearly always preset in video. Table 2 shows ablations for the different parts of the loss.
The attention mechanism in the model is intuitive. (except the need for multiple stages - see cons).

Cons:

I is not clear multi-stage attention is necessary - one stage seems to do well. Did the authors try different hyperparameters (eg. units/dimensions of single attention block) for one-stage of attention. There may be enough room to get the extra 2 percent, and why we should need 2 stages of attention is not clear, what is the intended effect?

The related Work section is rather plain to read, as the differences with the current work are not explained. It is a long list, and interrupts the flow of the paper without showing how it relates. Also the first sentence in "Related Work" does not read well.

Second paragraph of introduction states that the work intends to "localize a wider range of actions". The AVE dataset does not do this - is this a reference to the ActivityNet1.2 dataset - does it contain noisier and "weaker" labels than AVE?

Section 3.2 "snippet-wisely" is understandable but strange. Also in Fig 2 text: "respectively without and with L_cont and L_pseu" is difficult to follow.

review_title: Weakly labelled multi modal action localization in video

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose to mix the independent embeddings of audio and visual data by a set of cross-attention layers to the task of audio-visual event localization. They weigh the features based on the correlation between the representations of both modalities to obtain the output representation. After several layers like these, combined with dense skip connections, the final output is a multimodal representation of the input where the features have been individually learned for each modality based on information from the other one. The final features are concatenated and passed through an open-max classifier. Finally, they design different losses to help enforce coherence and continuity among the proposed labels throughout the video.

In this respect, the only thing that is not clear enough to me is the novelty of the proposed cross-attention layer. While there are some mentions to previous works of multi-modal representation learning, there is no explicit assessment of the contributions of the proposed cross-attention mechanism with respect to previous attempts to build similar representations, or even previous uses of cross-attention for other tasks.

Regarding the experiments, on one hand they perform ablation studies that confirm the contribution of some of the proposed losses and the cross-attention mechanism for building the audio-visual representations, as well as the use of an open-max classifier instead of a soft-max. On the other hand, while an extensive experimental comparison with other recent state-of-the-art models has been performed, proving that the model can improve previous results, it is not clear to me how much of that improvement is due to the use of specific backbones. For instance, the authors use a ResNet152 to extract the visual features, which is a fairly big network. It is unclear to me how much of the improvement over previous state-of-the-art is due to the use of specific backbone networks. In other words, given that even the baseline without cross-attention layers performs close to second, how much is the number of parameters and specific architecture design in charge of building those initial embeddings actually responsible for the final results?

Pros of the paper:
Extensive experimental results.
Improves state of the art.
Clearly written.
Cons of the paper:
Novelty of some contributions unclear.

review_title: The authors propose solving the task of action localization by using a multi-stage cross attention to build a shared representation of audio and visual embeddings followed by an open-max classifier to properly handle no-action cases. They perform an extensive comparison with other recent state-of-the-art models and an ablation study, proving the validity of the proposed method.

final_score: rating: 7: Good paper, accept
review_content: The authors propose to learn richer audio-visual representations for weakly-supervised action localization. First, the authors propose a multi-stage cross-attention mechanism to collaboratively fuse audio and visual features. Extensive experiments on two publicly available video-datasets (AVE and ActivityNet1.2) show that the proposed method appears to be effective.
The authors did not give the explanation of the built network in more details. Moreover, the motivation of the skip connection between the cross attention modules is unclear.

review_title: The motivation of the skip connection between the cross attention modules is unclear.

Meta-review: The paper focuses on the task of weakly supervised activity detection (WSAD). The proposed method combines various ideas together: **(i)** a cross-attention module for audio-visual information fusion and better representation, **(ii)** an open-max classifier to treat the background as an open set, and **(iii)** loss terms to encourage temporal continuity of action predictions. The experimental results on well-known benchmark datasets are promising as they beat out many other methods in the literature.

Based on the reviewers' comments, it is clear that the reviewers unanimously see value in the proposed methodology and the competitive results. To strengthen the paper, the authors are encouraged to provide a stronger validation of the contributions being claimed for the specific task being addressed. This would more concretely position the claimed contributions in the WSAD literature. 
Decision: Accept (Poster)