Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper incorporate the popular contrastive with unsupervised learning from video. Specifically, multiple frames from the same video is used as positive pairs and frames from different videos is viewed as negative pair.  The author also proposed a simple and effective ways to collect class-balanced and diverse video frame dataset from Youtube.  The author conducted extensive evaluation experiments on both video recognition and image recognition downstream tasks. Extensive ablation experiments demonstrated the effectiveness  of utilizing multiple frames and the balanced data collection algorithms. 

My concerns:
1.  Lack of novelty.  The idea of mapping different frames in one video closer,  predicting other frames in one videos, or maximizing mutual informations of embedding of different frames in one video is widely explored.  And adopting contrastive learning in video unsupervised learning scenario has been done before. (Mentioned in the related work section of your paper too)
2. No comparison against other video based unsupervised learning algorithms.  From my viewpoint, improve over single framed based contrastive learning only proves that your algorithm successfully utilized temporal information encoded in the data and provide limited insights for exploiting more useful information from videos.  
3. If you can demonstrate your way of incorporating contrastive learning into video based unsupervised learning offers a non-trivial improvement or have a significant difference with other video based unsupervised learning, the impact of your work will be larger. 

Overall, I think this paper is interesting but its contribution is limited. 
review_title: Initial review from R3

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The idea of learning representations from video rather than single images is an appealing one with many favorable properties to allow a system to get direct signal on appearance of objects under various natural transformations (occlusion, lighting, etc). Combining instance discrimination ideas of loss based on unlabelled images for which it is known whether they are similar or not, with the idea of curating the images from video is hypothesized to yield learned representations that capture properties enabling improved performance across a variety of single image tasks. The authors create a dataset based on video with positive pairs for noise contrastive estimation, conduct fairly comprehensive experiments and promise to make their newly constructed dataset available. The experiments showcase this type of learned representation outperform alternatives not based on videos on a variety of tasks. 

Quality : this seems like a solid paper offering a good intuitive idea with well supported experimental section to show case its relevance. 

Clarity : the paper is quite clearly written for the most part. The dataset section 3.1 seems to have an omitted paragraph, please see point 1 below. Section 3.2 is quite lean and does not stand on its own, but relies heavily on previous work omitting much of the essence. I would recommend spending a bit more time on ensuring it is more rigorously written. See for example my comment 2 below. 

Originality : the paper is modestly original. It combines two existing ideas - that of using discrimination loss for unsupervised learning of image features, and that of using video based data to allow for rich example of the same data that takes into account real world type transformations. Thus, it is hard to claim more than moderate originality. However,  

Significance : the improvements over existing baselines are solid, though I would not categorize them as dramatic. Given the originality is also solid, the overall significance is moderate. 

Comments:
1. The dataset generation section is strange, did you omit too much? "We use the following fast and automated procedure to generate the images in our dataset. Using this procedure,..." it almost seems like a few sentences were dropped between the first and second sentences. While the information exists in the appendix, a sentence or two seem to be warranted in the main test. 
2. "Gradients flow through the positive pairs" - at this point in the text you have only introduced a loss. The sentence warrants the question of gradients with respect to what? for rigor, and clarity of exposition, this intuition related statement should come after you talk of what is the parameterized aspect of eq (1) wrt which gradients are taken (so after eq 2 is introduced and the idea that the feature representations are captured through learned method, and in particular some reference to the NN you are using. This makes it a bit more complete as a description of the method and presented in a more methodical order. 
3. Why would you not remove the option of choosing an anchor and positive as the same image? seems easy to avoid
4. how do you know if the video doesn't contain a shift to a different scene, with different content, thus making the positive actually very different? 
5. Since the representation you provide is supposed to learn representations that are somehow more informed about natural transformations (occlusion, lighting) - is there an experiment you can conceive of to test this specific hypothesis? i think it would both be interesting, and also give insight on whether this is indeed what is being learned. This might also make this representation useful for other types of tasks that are not looked at in the paper that I would encourage the authors to explore. 
review_title: Learning representations from video sequences, using the Noise Contrastive Estimation proves to yield useful representations

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an idea to use NCE for videos where positive/negative training pairs are created by temporally sampling different frames in the video.

The idea is fairly straightforward, there is not anything particularly novel about the approach, other than how the training pairs are created. I think it would have been more interesting and insightful if more ways to generate these pairs were proposed and evaluated. For example, video has many modalities (e.g., RGB, audio, optical flow, etc) in addition to the spatial and temporal dimensions. Exploring ways of creating pairs using multi-modal data would have been more interesting.

Experimentally, there aren't really any comparisons to previous self-supervised learning methods. This is a pretty major weakness as it makes it difficult to understand how well this task is doing. Methods like SimCLR provide >70% accuracy on ImageNet and others do well on video tasks (see missing related works below). Currently, I'm not convinced by the experiments.

The studies comparing different pretraining data and multi-frame vs. single frame are interesting and show the potential of the approach. 

There are some missing related works, for example:
- "Cooperative learning of audio and video models from self-supervised synchronization", NeurIPS'18
- "Audio-visual scene analysis with self-supervised multisensory features", ECCV'18
- "Evolving Losses for Unsupervised Video Representation Learning", CVPR'20 

These works all provide strong performance on unsupervised video representation learning, yet are not mentioned or compared to.

Overall, I think the proposed idea is not especially novel and the experiments aren't strong enough to show that the simple idea is good. I think there is some potential in the paper, but needs more to be convincing.
review_title: Review

Meta-review: This paper was a difficult decision. Overall it seems to be a quality paper, well written and with many experiments, in particular evaluating learned representations across various tasks and datasets. The authors were also quite courteous in their replies which is appreciated. I really like the point the paper makes about video as a natural augmentation and I find that novel amid the recent NCE surge, where most papers rely critically on augmentation. R4 was also very positive about the paper overall concept.

In terms of paper weaknesses two of the reviewers voted for rejection because the paper ignores existing work on contrastive learning from videos. The authors rebuttal is that they are the first evaluating on images, not on videos. All reviewers also point out limited technical novelty, which the authors acknowledge. Finally, R1 is not very confident about the experiments.

Overall, and after calibration, the appropriate recommendation seems to be rejection.


Decision: Reject