Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a cross modal (audio and video) contrastive learning scheme to pretrain on one of the modalities. They also propose a sampling scheme for the negative examples. 

The experiments and the corresponding ablations suggest that the proposed methods is helping overall. The table 1 suggests that adding the proposed components (cross-modal pretraining, active sampling for negative examples) seem to help overall. 

Also, the samples in the appendix seem to suggest that the proposed `'active negative sampling``'  scheme seems to provide more relevant negative samples. Figure 2 in the experiments also show that the proposed negative sampling mechanism seem to increase the diversity in the negative samples.  Figure 2, also showcases that the proposed negative sampling scheme works better than picking out examples by looking at feature embeddings. This suggests that the proposed negative sampling scheme based on looking at gradients is advantageous over a more standard feature-level approach. 

I am not entirely sure if table 2 is exhaustive on the state-of-the-art (SOTA), but from what I gather from the narrative of the paper, they seem to improve the SOTA numbers on three different pretraining dataset size regimes for action recognition datasets. They also provide a similar experiment for audio event classification in table 3, and claim SOTA results. (where they pretrain on different datasets, and test on another) 

I think the clarity of the manuscript could be improved, especially in Section 3.3. I think that perhaps you could replace the low level algorithm with a higher-level pseudocode or a diagram and take the algorithm to the supplementary materials? 

Also, my understanding of the proposed dictionary building / sampling approach for negative sampling is that you look at the magnitude of the gradients with respect to the contrastive learning objective, and pick the examples which give a large gradient magnitude. I think that this could be made clearer for the reader, perhaps by including a toy example, or including a figure in the appendix which showcases the gradient magnitude vs. the chosen negative examples. 
review_title: Good results, but the manuscript is difficult to get into 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The goal of the paper is audio-visual self-supervised learning using an active sampling technique to mine hard negatives during training. 

Strengths 
- The paper is well written and clearly explained. 
- The strategy for selecting negatives based on diversity seems well reasoned and experimentally outperforms random sampling and OHEM. I like the study showing empirically more categories covered in active sampling vs random sampling. 

Weaknesses: 
- It would be nice to see some discussion of the other self-supervised contrastive works that also focus on the optimal selection of “negatives”: eg. Korbar et al 2018 (where they use a curriculum based on time distance of negatives from the positives), Iscen et al. 2018: https://arxiv.org/abs/1803.11095, Cao et al.2020: https://arxiv.org/abs/2006.14618, Wu et al 2020: https://arxiv.org/abs/2005.13149 (variational extension to InfoNCE with modified strategies for negative sampling). In a similar vein, the comparison to OHEM in the supplementary is quite nice and I believe should be in the main paper. 
- Why is the ablation showing the benefits of Active Sampling only on Kinetics-Sound? Would the same trends shown in Table 1 hold on a random sampled proportion of Kinetics of the same size, or even on the whole dataset?
- The audio-visual contrastive method has been proposed in numerous papers before, so the novelty of this paper lies solely in the active sampling technique that increases diversity using K-means clustering 
- (Minor) The footnote on page 1 is a bit confusing, it’s hard to see the probability drop from just Fig. 2 since the division needs to be done, probably better to plot the probabilities directly somewhere or remove this. 
- (Minor) It would be interesting to also test the audio representations for classification on VGG-Sound (http://www.robots.ox.ac.uk/~vgg/data/vggsound/) 
review_title: Nice idea and decent results, lacks discussion of other existing hard negative mining strategies 

final_score: rating: 7: Good paper, accept
review_content: In addition to my review - directly next, I also give this paper to a junior PhD student who I believe did a very good job in the review. There's no disagreement between both reviews, and I have checked both for correctness, so I provide both of them below: [so it's a 2-in-1 review option.]

================

The paper is straight-forward, easy to read and clearly written. It builds on the Momentum Contrast (MoCo) paper (2020), which aims for a moving average dictionary per modality along with cross-modal contrastive learning for self-supervised tasks, evaluated on audio-visual datasets. The paper takes this a step further by arguing for a more active way to select negative samples that form the dictionary (or negative memory). While dictionaries are efficient ways to perform contrastive learning, this paper showcases that the random selection of negative samples can have a substantive impact on the performance.
To resolve this, the paper proposes to select negatives based on two criteria: informativeness measured by the gradients of the sample and diversity (or representativeness of the potential classes) through a greedy (or online) K-means approach. When both are utilised, the result showcases state-of-the-art on downstream recognition tasks on 3 standard datasets. Results demonstrate the effectiveness of the approach.

The paper is scientifically correct and the proposal is explained both theoretically, with equations, with an algorithm and supported by results. 

My only concern with the paper is that by the end of the introduction, the proposal and concept are already well-stated. The rest of the paper keeps repeating the same argument. Apart from the introduction, only 3.3 offers more details. Related work, sections 3.1 and 3.2, figure and algorithm all repeat the same thing in various forms. While emphasis is good, I found the repetitiveness to affect my interest in reading the paper. I recommend that the authors revisit their manuscript to remove the repetitive argument (e.g. the notion that you need both informative and diverse negative samples is made in 4 different locations). 
Instead, the authors can provide more insight into the negative dictionary for the variety of models and datasets, showcasing the impact on certain classes. They can also discuss failure cases and whether they relate to classes that are still under-represented in active negative sampling. Comparing an online to offline diversity options (to know the upper limit) would also be interesting.

Additionally, the need for fully connected layers (described in page 6) is only showcased for XMoCo, and not for the final proposal. An ablation of its effect on the final proposal would be important to include.

==============

Summary

This paper proposes an approach to actively sample negative examples for audio-visual representation learning and combines this with an audio-visual approach for contrastive learning. The paper suggests that an active sampling approach that samples datapoints that are are both diverse and uncertain would result in a much more informative dictionary of negative samples, and uses gradient information and the k-MEANS++ algorithm to create such a dictionary of negatives. They then perform a cross-modal dictionary lookup where the queries and keys for each modality are each represented by a learnt encoder where the parameters of the query encoders are learnt directly via backpropogation and the key encoders' parameters are defined as a moving average of the query encoders' parameters. A contrastive loss is then used to learn a representation that is effective for fine-tuning to down-stream tasks, achieving state-of-the-art results for both action recognition and audio event classification.

Strengths

The paper presents novel, well motivated results in the field of cross-modal representation learning. It includes detailed comparison to the state-of-the-art across a wide selelection of papers and datasets. The proposed method shows state-of-the art results across small, medium and large pre-trained datasets for both action recognition and audio even classification.

Weaknesses

Whilst there are detailed comparisons between related works the paper does not discuss many of these related works in the related work section. It would be nice to have some more information on the related works, how they relate, how they differ and the gap that this work fits into.
Additionaly, some information on the models used for audio and visual encoders would help with understanding. They are described in the appendix but there is no reference to them in the main text. The paper adopts a style wherein the entire process is shown as a single algorithm, and the text describes the algorithm step-by-step. This is, in general, quite clear, however continually changing pages to check the algorithm can decrease readability. Perhaps putting the full algorithm in the appendices and providing a simplified version in the main body would be clearer, or alternatively, splitting the algorithm in to multiple, smaller components.
The comparisons to related work are quite detailed, however, in some cases the proposed method offers only a marginal improvement (particularly for the medium and large datasets). Running some experiments multiple times and including error bars would strengthen the state-of-the-art claims. The comparisons could also be improved by consistently comparing across the same architectures. For example, the proposed method is only trained with R(2+1)D-18 for the large dataset, despite the architecture being widely used within the medium dataset comparisons. Perhaps two or three architectures across each dataset size would further strengthen comparisons.

Further points:
*) typo: "meduim scale"
*) Random overleaf link in the appendix
*) references to relevant sections in the appendix would be nice

review_title: Nice trick, well-motivated and supported by results. Clear improvement over prior work. Text is quite repetitive though making it a boring read unfortunately

final_score: rating: 7: Good paper, accept
review_content: #### Summary
In this paper, the authors propose a cross-modal (audio-video) self-supervised representation learning method with a contrastive learning framework. To overcome the high redundancy in the negative samples, they propose an active negative sampling method. They use a gradient with respect to the pseudo label to measure the uncertainty of a negative sample. They use K-means clustering to maximize the negative sample diversity when constructing a new negative set for queueing. They show their method's efficacy on the public benchmarks: Kinetics, AudioSet for retraining, and UCF-101, HMDB-51, ESC-50 for downstream tasks. 

#### Strengths
* The problem is interesting and relevant to the community. Audio-visual self-supervised learning could be interesting for many researchers in the computer vision community.
* Design choice looks reasonable and effective. Cross-modal contrastive learning gives a significant boost to performance. Active sampling gives a significant boost. 
* Writing is straightforward and easy to follow.

#### Weaknesses
However, I have a few concerns about this work.
* To show the effectiveness of the proposed gradient-based active sampling method compared to the embedding-based active sampling, the authors plot the number of action categories vs. the number of training iterations in Figure 2. However, in my opinion, more action categories might not mean high diversity in a negative set. For example, an action could be compositional. "Drinking coffee" action consists of "picking up object", "approach an object", "bring an object to the actor". Since the proposed method uses 16 frames, very short, feature encoding, the embeddings will be more towards these subactions, not the entire actions. Therefore, more action categories do not directly mean more diverse negative examples. So I would suggest the authors directly compare the downstream task performance of gradient-based sampling and embedding-based sampling.
* The comparison of the proposed method with the fully supervised method is unfair. The fully supervised method has VGG-M-2048 as a backbone and pretrained on ImageNet, while the proposed methods have 3D-ResNet-18 or R(2+1)D-18 as a backbone and pertained on Kinetics or AudioSet. It is also unclear which row of "ours" the authors compare to the fully supervised method. I cannot find a 7.2p improvement on the UCF-101 and the 14.1p improvement on the HMDB-51 compared to the fully supervised method. It is okay that the proposed self-supervised method is weaker than the fully supervised method. However, the comparison should be fair, in my opinion.
 
I will increase my rating if the authors adequately address my concerns.

#### Minor comments:
* 6. Conclusion, fourth row: "When pertained …" -> "When pre-trained …"?


review_title: Interesting multi-modal contrastive learning method, but needs some more empirical validation and fair comparison

Meta-review: The paper focuses on the task of learning audiovisual representations through contrastive learning on unlabelled videos. This work is another addition to the ever-growing literature on self-supervised learning (SSL) with emphasis on video and multi-modal data. The main contribution of this work is the manner in which it tackles a well-known drawback of contrastive learning, namely the strategy used to sample negatives in the contrastive pipeline. The authors propose an active sampling strategy that adaptively chooses negative samples that are informative and diverse. This active selection technique is similar in spirit to many selector functions proposed in the active learning literature. It seems to be the first time it is used for contrastive SSL. 

Based on all the reviews and the subsequent discussions, it seems that the reviewers' comments were addressed. The authors are commended on integrating the reviewers' suggestions and making the necessary edits to the paper in a timely manner. 
Decision: Accept (Poster)