Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a method to train self-supervised cross-modal embeddings that are strong in both global and local representations. This is done by optimizing two objectives -- spatially-local/temporally-global and spatially-global/temporally-local. The first is trained using contrastive learning by taking negative examples from different clips, whereas the second is trained by taking negative examples from different time in the same clip. The authors conduct a range of experiments lip reading, deepfake detection, sound classification, etc.
main_review: The paper is well motivated. The authors recognise an important problem and present an effective way of solving it. 

However, the key training method (e.g. sampling method for negatives) overlap strongly with Nagrani et al. [1] which is not cited in the paper. The authors should give a clear summary of the different between this paper and theirs. I may change my rating based on this.

The experiments are very thorough and convincing. The ablation studies are good. The paper is well written.

[1] https://ieeexplore.ieee.org/document/9054057
limitations_and_societal_impact: The comments on limitations and broader impact are adequate. There is no particular concern beyond generic risks in ML applications.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a novel method for self-supervised video from audio and video. The technique uses both the temporal and spatial dimension of the audio-visual input in order to build two contrastive losses: one capturing slow changing elements in the video present in the audio and the other one capturing temporally changing elements. After learning through these two losses, authors propose a detailed downstream evaluation consisting of task where spatio-temporal granularity is relevant (deepfake detection, lip reading) and other task about global information in the video (action classification).
main_review: Overall, I think the new ideas introduced in the paper are interesting and the evaluation section show that the resulting model has good performance for lip reading, detection and classification. I am missing some comparisons with the state-of-the-art, but overall I believe this is a good paper and should be accepted. 

Strengths: 
- The paper is clear and well written. The different parts of the model are well motivated and it is easy for follow and understand the reason behind the design decisions.

- The idea of introducing explicitly the notion of fine-grained spatiotemporal information in contrastive learning is interesting and novel. Authors execution of the idea is simple but effective, building the two losses consistently with their goals. In my opinion, new methods for self-supervised learning will start considering more granular information, following the direction of this paper. 

- Authors ablate all the relevant decisions, which to me is interesting and shows the importance and effect of the two losses used in the model. I like how looking at Table 5 you can extract the main conclusions in the paper.

- Authors also present qualitative results of the attention between the audio and the video modality. The results are consistent with the goal of the attention layer (highlight the areas in the video with relevant audio-visual correlation). 

- The model is evaluated for very different tasks (lip reading, deepfake detection and classification) which is good for the reader to understand better the properties of the model.

Weaknesses: 
- In terms of evaluation regimes, I am missing linear evaluation for some of the video classification tasks. It is hard to evaluate the quality of the features by only looking at the fine-tuning performance where all the network is updated on the downstream task.

- In terms of evaluation datasets, I am missing larger action classification datasets such as Kinetics-400 (where authors only train without labels). I believe having additional evaluation performances in this datasets would help the readers understand a bit better the strengths of the features.

- In terms of training datasets, I would be curious to know how this model would scale using larger scale dataset such as AudioSet. One question to answer would be if this performance improvement is only present for smaller datasets but goes away when using more data. I think it would be good if authors can answer this question.

- Although I understand that some papers cannot be directly comparable, I think it would be worth for the authors to at least report performances for some of the models trained on different architectures or datasets. I think it is important for the reader to put the quantitative performance on perspective to all the works on the topic. Some examples of this would be reporting XDC or GDT when trained in AudioSet or reporting performance of MMV. 

- Authors ablate very well all the design decisions. However, I am curious about whether it is necessary to have such a large negative pool for the S-Global / T-local loss. I wonder if authors can evaluate that by just sampling some of the negatives at each iteration. 
limitations_and_societal_impact: Yes, I think the authors have address the limitations of their work as well as their possible societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies self-supervised learning of audio-visual representations from video data using neural networks and a contrastive audio-visual correspondence objective.

The key novelty of this paper is that it asymmetrically applies pooling/attention to simultaneously learn two different representations: spatially-local/temporally-global, and spatially-global/temporally-local. Prior work has focused on learning only one of these two, or simply learned spatially-global/temporally-global representations. 

The model is pre-trained on both Kinetics-400 as well as a mixed dataset, using a 120k video subset of Kinetics-400 with a 120k video subset of the AVSpeech dataset. A single pre-trained model can then be fine-tuned on multiple downstream tasks, including action recognition on UCF101 and HMDB51, sound classification on ESC50, lip reading on LRW and LRS2, and deepfake detection on DFDC. Qualitative analysis of sounding object localization is also performed on Kinetics-Sounds. The quantitative results are strong, in most cases outperforming previous self-supervised approaches trained under similar conditions. 

main_review: Strengths:

The paper presents a novel and well-motivated approach to self-supervised contrastive learning from videos

The paper contains extensive experimental results on diverse downstream tasks that demonstrate the strong efficacy of the proposed approach

The paper is well-written and easy to understand

Weaknesses:

It would have been nice to evaluate on a downstream task that explicitly evaluates segmentation/localization, as opposed to tasks that only indirectly evaluate it

Suggestions:

Line 187: phoneme transcriptions are missing a slash, e.g. /p should be /p/

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an audio-visual self-supervised learning approach based on two cross-modal contrastive losses that learn audio-visual representations that can generalize to both the tasks which require global semantic information and localized spatio-temporal information. Extensive experiments on 4 downstream tasks demonstrate the usefulness of the learned representations, compared to their disjoint learned counterparts.
main_review: Strengths:

- The paper is generally nicely written. Existing works tend to learn either global representations or local representations, while this work aims at learning versatile representations that generalize well to both scenarios.

- The motivation/advantage of learning global and local representations jointly is well justified---learning global/local representations jointly improves the generalizability of the learned representations compared to the disjointly learned counterparts, especially when the downstream scenarios are unknown in advance. 

- Good observation and analysis to construct triplets from different spatial and temporal spans to capture either global information or local information. A tailored global-local contrastive learning network is designed to realize the proposed idea.

- Extensive experiments are performed on four downstream tasks across a number of datasets to demonstrate the generality of the learned representations. The proposed self-supervised audio-visual representation learning method compares favorably to other self-supervised learning methods and supervised learning methods.

- Very nice ablation studies in Table 5-7 demonstrate the effectiveness of the key components such as 1) impact of both the local and global features, 2) MIT for local contrast, and 3) attention mechanism.

Weakness / Detailed comments:

- The paper should discuss other prior work that learns global and local representations in a similar spirit, and explicitly discuss the main contribution and difference of the proposed method. E.g. [1][2]. Otherwise, the novelty of the proposed method is not salient.

- Justify the choice of downstream tasks to evaluate the learned global-local representations. Why these four tasks, and how do they represent other potential downstream tasks, since it's claimed that the proposed method learns very general representations? Or, more tasks are needed to justify that the learned representations can generalize to various tasks that leverage spatio-temporal representation.

[1] "Actbert: Learning global-local video-text representations." In CVPR, 2020.

 [2] "Contrastive learning of global and local features for medical image segmentation with limited annotations." In NeurIPS, 2020.


################################AFTER REBUTTAL###########################
Thank the authors for the response. My initial concern was that the idea of learning global and local information isn't entirely new and prior work is not acknowledged and discussed, and their choice of downstream tasks is not well justified. In their response, the authors have provided more discussions and explanations, which generally addressed my concerns. They have also provided interesting new results on action recognition (Kinetics-400) and audio-visual event localization (AVE). Therefore, I would be happy to see this paper getting accepted and encourage the authors to include the new discussions and results in the final version.



limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative social impact of the work.
review_title: null

Meta-review: All the reviewers appreciated the global and local message, the clear exposition, the ablations, and the additional experiments carried out
in response to the reviews.

Decision: Accept (Poster)