Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper works on self-supervised audio-visual representation learning. It shows uncurated movies and TV shows are good sources for learning audio-visual representation. The proposed model targets on short video clips like 3-second clips, and learned representation is evaluated on action classification and audio classification. 
main_review: The paper trains models on a large-scale TV and movie dataset for representation learning, and it shows uncurated TVs and movies are good for audio-visual representation learning with a simple contrastive method. This message is valuable because there are not many large-scale TV/movie datasets available for representation learning. But the technical contribution of the paper is limited. The key method is a basic contrastive loss, and additionally, the paper explores sampling strategy, which is not very new (see limitation). 

The paper provides detailed experimental studies on sampling strategy, color jittering and the effect of genre. The quality of submission is good.

Overall, the paper is clearly written. But figure 1 is hard to read. I suggest giving short names for different KL divergence values and adding more captions. 

The key message from the paper is encouraging, showing that uncurated TV shows and movies are a good data source for audio-visual representation learning. However, public access to these copy-right datasets is a long-standing problem. The result of the paper will be more significant if these datasets can be publicly released in some way.  

limitations_and_societal_impact: Given the fact that the TV and movie datasets cannot be released and only the pretrained models will be publicly shared, I think the representation quality is not strong enough (measured by downstream performance). If the TV and movie dataset can be released it would be an impactful dataset for the community, but without that the technical contribution of the paper is limited.

There are some properties of the TV and movie datasets that are neglected. I think the audio accompanying TVs and movies could be background music (not much relevant to visual information), human’s speech (may correspond to the human's face/body), and the sound of objects/scenes that the model aims to learn. In other words, some portion of the audio is noise for audio-visual representation learning. But the proposed basic contrastive loss does not explicitly handle this.

The tech details about sampling policy (e.g. sampling clips from the same source video as hard negative, and the sampling windows affects difficulties) was studied in previous audio-visual paper [1] and video paper [2]. Thus the novelty is limited. 

Besides, I think the experiments do not match the claimed contribution very well. It feels the claimed main contribution is the power of large-scale TV and movie datasets, then maybe the paper can consider comparing the same method trained on other TV/movie datasets (like [3]). 

* [1] Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization, Korbar et al. 
* [2] Video Representation Learning by Dense Predictive Coding, Han et al.  
* [3] Condensed Movies: Story Based Retrieval with Contextual Embeddings, Bain et al.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes a method to learn self-supervised audio-visual representations from long (order of half to couple hours) uncurated videos namely from movies and TV shows. A new sampling approach is presented to ensure batches are created while taking into consideration the effect of re-occurring audio-visual patterns (main character or background scenes). Picking multiple clips from the same long video that are spaced by some minimum distance is found to be effective as compared against random sampling. Pretrained models are evaluated on standard action recognition tasks: UCF101 and HMDB, and audio classification on ESC50.
main_review: *Strengths:*
1. The paper makes a strong case for using uncurated datasets used for pretraining.
2. Using KL divergence to measure the quality of batches created for contrastive learning is interesting. In particular, desirable features are observed by the proposed sampling scheme - a low KLD between negative clips from the same movie ($S^-$) and other movies ($S^\neq$), and a small difference between $S^\neq$ and $S^+$ and $S^-$ and $S^+$ (see Fig. 1).

*Weaknesses:*
1. Learning from uncurated data. Firstly, I'm not convinced that movies and TV shows are a uncurated dataset. In fact, as compared to most user generated content on Youtube, one could argue that they are highly curated with a large amount of editing and production value. Please also see point 2, which clarifies how movie data is not uncurated with respect to downstream tasks either.

*Discussion for point 1, not influencing the rating*. I think the paper raises some interesting questions on whether it is possible to truly acquire random uncurated audio-visual data. Randomly picking Instagram/Youtube ids too has a bias of the video being on Instagram/Youtube in the first place, so it's hard to say that IG-Random [3] is in fact uncurated. In fact, I'm not even convinced that we need to train machines on uncurated videos - humans don't simply listen and watch *everything* to learn about the world, we go study at schools and universities and experience curated learning.

2. One of the downstream evaluation datasets, HMDB51, has a majority of clips sourced from movies. HMDB51 also tends to show highest performance improvements for the video representation in the experiments when training with movies (+2.6% in Table 1 over k=1), while UCF101 which features the sports categories, not covered commonly in movies and TV shows, does not improve much (+0.6% in Table 1). This raises the question of whether we should be training on uncurated data. More importantly it goes against one of the claims on lines 72-73: that there is a "meaningful domain gap between our pretraining data and space of downstream tasks". In fact, ESC50 categories may also benefit more by sound effects inserted in TV/movies as compared to speech or background-music heavy vlogs, messages, or instructional videos found on youtube/instagram, but this would require further investigation.

3. While the core idea presented here has merit - how to sample when given long videos - I don't see anything special catered towards movies/TV, for example, by actually analyzing their content. I see two options:    
a) Either position the paper on how to create batches from long videos and also demonstrate the effectiveness of the sampling algorithm by pretraining on other datasets with long(ish) videos such as HowTo100M (million videos of 6 minutes average duration, admittedly not hours); or  
b) it would be useful to incorporate some movie/TV content analysis if the core message is about the benefits of training on such data.

4. (not influencing rating) The argument about such data democratizing research for smaller groups with limited compute resources should be removed. Firstly, such data is not commonly available: 3600 films and 9200 TV episodes. Secondly, training required almost 2 days with 8 A100 GPUs - 320 GB memory could often be 50-100% of the GPU memory of an academic lab, 32 GPUs x 10GB. In such a case, even creating just the final version of Fig. 1 (20+ experiments), will require all GPUs used for 40 days!

5. Minor:
a) Fig. 1b, k is said to be 16. Similarly, what is w for Fig. 1a?  
b) Is the batch size 64 as indicated on L260?  
c) Making highest performance numbers bold in the tables would improve readability.  
d) While Fig. 1 is very interesting, it would also be useful to include the actual values of KL(S^- || S^+) as an indicator of training performance, since the desired low scores can be achieved for the other metrics with a failure mode in training where all S+, S-, S^\neq lead to similar scores.


limitations_and_societal_impact: Some worry of using the models trained on movies for downstream tasks as movies may often be racially insensitive and surface social problems. This is however indicated in the paper.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The task is to learn audio-visual representations from uncurated data such as movies and TV videos. Unlike previous works that show learning from uncurated data is ineffective, the authors show good performance from learning from these videos. They propose hierachical sampling policy, with negatives being sampled from both within and across different sources. The model is trained on the dataset that the authors collected, which consists of thousands of hours. The performance on various downstream tasks (action recognition, sound classification) is on par compared to using much larger uncurated datasets such as IG-Random.
main_review: The authors tackle an important problem of learning uncurated datasets. The effective use of uncurated data is an important question and a future direction for the ML community. 

The dataset used in the paper is of significant size -- 1.4 years (12k hours). The sources are from TV/ movies, which is already somewhat refined and curated compared to user-created content such as IG-Random. This might explain the performance advantage over IG-Random. The data has not been released due to copyright.

The method used is fairly standard contrastive learning. The authors propose sampling strategies to adjust the difficulty of negatives from same and different source videos. The taking of negatives from same/different videos has been shown in previous works, but the adjustment of temporal windows based on the intuition on formulation on scenes is original. However, the effect of sampling strategies shown in Table 1 is marginal, which leads me to think that the good performance comes from the quality of inputs rather than the training strategy.

Overall, it is notable that the authors highlighted the TV/ movies data as a good source of self-supervised learning, but the technical contributions are not sufficient to merit presentation at the conference.

The writing is clear.
limitations_and_societal_impact: The explanation in Sec 6 is adequate.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper performs self-supervised, audio-visual pretraining of deep CNNs on movies and TV shows. The downstream tasks investigated are action classification in videos (HMDB51 and UCF101 datasets) and environmental audio classification (ESC50 dataset). The paper shows that pre-training on 0.7-1.4 years worth of TV and movies results in downstream task performance that is competitive with, but lags slightly behind, the current state-of-the-art approaches that pre-train on datasets like HowTo100M, AudioSet, and IG-Kinetics65M. 
main_review: Strengths:

- The paper studies a timely problem that is of broad and growing interest to the community

- The paper is clear and easy to read

- The paper demonstrates that Hollywood Movies and TV shows are an effective data source for learning audio-visual representations


Weaknesses:

- Aside from the novel dataset used, the paper offers little in the way of modeling or algorithmic novelty over the existing state of the art. 

- The downstream results slightly lag current SotA approaches trained on other widely-used datasets

- It would have been better to also run baseline models on the movie dataset, as current experiments tangle up architecture with training data

- One of the paper's main arguments is that uncurated data is preferable over curated datasets, but the licensing and distribution restrictions surrounding the data used here (movies, TV shows) make it even more difficult for the research community to use than existing datasets based on e.g. YouTube. It’s also not clear to me why a collection of Hollywood movies and TV shows that are categorized by genre qualifies as “uncurated” when a dataset of how-to videos scraped from youtube (e.g. HowTo100M) does not.


Suggestions to authors:

It seems to me that the experiments here show that long-form content like movies offer no advantage over scraped youtube videos when incorporated into architectures that make i.i.d. assumptions about the relationship of clips drawn from the same video. It seems like a key difference between long-form content and youtube content is the presence of an extensive overarching narrative structure, which I can see being useful in future work but wasn’t leveraged here.

I would have liked to seen a comparison between the TV+Movie dataset and a subset of HT100M or IG-Kinetics65M that was sized to match the duration of the TV+Movie dataset; the current results don’t establish why pre-training on TV+Movies lags behind the other datasets. Is it simply because the other datasets are larger, or is one video domain naturally more informative than others? 

Following from the previous point, it would have been even more informative if the paper included experiments that held the modeling architecture and learning algorithm fixed, and varied the dataset used for pre-training.

-----Post rebuttal update:
After reading the authors' feedback and the thoughts of the other reviewers, my assessment of the paper remains unchanged. 

limitations_and_societal_impact: The authors do acknowledge the copyright restrictions which would make their dataset generally inaccessible to most of the research community.
review_title: null

Meta-review: The paper has a good message about using uncurated data for audio-visual self-supervised learning. However, the reviewers were not convinced that this paper carried through on the message, since TV material and movies are curated to an extent (though not class balanced). 

The reviewers identified a number of weaknesses, including:

- insufficient novelty

- algorithm not applied to other (less curated) datasets

The rebuttal did not change their opinion. 

The authors should take note of the suggestions for how to improve the paper,  particularly from Reviewer cgAj

Decision: Reject