Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a self-supervised learning (SSL) method for learning strong visual representations based on compressed videos. Compared with prior work, which sees videos in RGB only, the proposed method is more efficient and leverages the motion information that's already accessible in compressed videos. The authors propose to use a contrastive learning method to supervise on kind of modality with correspondence of another modality. On multiple datasets (mainly UCF and HMDB), good performance is achieved, with both linear probing and transfer learning settings.

main_review: [originality: medium-high]
 - The idea of using compressed videos for self-supervised learning is novel/original. Most prior work that operates on compressed videos focues on supervised learning setting only. The implementation of this idea is standard/expected (this is not a drawback).

[quality: medium-high]
 - The paper covers useful ablation study that I'd expect. In particular, the authors demonstrate the usefulness of the compressed-video modality (MV) compared to RGB only (Table 3). Compared to prior work, the performance is also strong while being significantly more efficient. The ablation study that compares with optical flow and low-res optical flow is also interesting (Table 2).

[clarity: high]
 - I think this paper is overall clearly writen. One question about implementation details: How is the similarity calculated in the hard-positive mining procedure?

[significance: medium-high]
 - The area of SSL draws a lot of attention in our community and is of great potential. I think this work extends our knowledge from a new dimension with the use of compressed videos. I think this work can lead to promising new directions in SSL research. 



limitations_and_societal_impact: 1. In the paper, the authors perform experiments/evaluation solely on videos. I was wondering if the method learns good representations for "image" tasks as well. Namely, leveraging MV can potentially give us better RGB representations for images, too. I wonder what the authors think about this.
2. If the hard-positive mining is based on the learned features, would there be any "chichen-and-egg" problem introduced by the proposed procedure?
3. I like the comparison to MV_{CoViAR}, but I wonder how each specific compression setting affects the results. For example, how does GOP size, the use of B-frame vs P-frame, etc. affect results.

Regarding societal impact, I don't see anything particularly concerning (or more concerning) than other SSL work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a self-supervised method for compressed video representation learning, which helps save both storage and computation. 
main_review: Strength:

- The proposed method directly uses RGB and motion vectors in the compressed video for efficient representation learning.
- Experimental results show that the proposed method reduces the total (inference) time and storage, and achieves marginal improvement compared to state-of-the-art methods.

Weakness:

- Adding temporal learning objectives such as changing the temporal order of RGB frames or motion vectors within a clip ( as in IMRNet ), which is missing from the proposed method, would greatly help learning video representation.
- Modeling of the proposed method mostly applies the InfoNCE loss, only selecting of positive and negative samples are updated for RGB and motion vectors.
- As shown in table 1, the computation overhead largely comes from the preprocessing time. Though not an apple-to-apple comparison, parallel re-encoding for IMRNet may largely mitigate the overhead?
limitations_and_societal_impact: The reviewer would encourage the authors to discuss limitations and impacts of the proposed method, as mentioned in the 'Main Review'.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper presents a self-supervised algorithm for representation learning of videos. The authors propose using RGB clips and Motion Vectors (MV) available in compressed videos to reduce the computational burden of processing. The two streams are seen as views in the contrastive setting: positive pairs are composed of RGB and MV clips of the same video, negative ones have clips belonging to different samples. Moreover, hard positive mining extends the set of positive pairs by seeking other videos that are similar in both views. The approach compares favorably on action datasets UC101 and HMDB51 for both action recognition and video retrieval tasks.
main_review: ## Strenghts
* The contribution of this paper is timely: the computational requirements of contrastive learning hinders its applicability on video representation learning. Compressed videos enables the shrinkage of required time and storage.
* The work proves how effective use of the less informative MV is on par with the computationally demanding optical flow
* Extensive experiments on action recognition and retrieval prove that the proposed gets competitive results with state of the art methods: linear evaluation and full fine-tuning accuracy for action recognition and R@k metrics for retrieval.

## Weaknesses
* The method builds on [1] by extending the single-view contrastive approach to learn the correspondence between two views and changing the hard-positive condition. However, it is well known that cross-view contrastive is beneficial [2, 3, 4]. Yet, the showed technique for harvesting hard positives is an incremental contribution to the literature.
* How cross guidance and hard positive mining contribute to the final performance is not well disentangled and it's not easy to understand what their impact is. What's the performance loss without hard-positives? What is the effect of using hard positives in the initialization phase? What is the performance of the ensemble of RGB and MV model right after initialization?

### Minor
* Table 2: I find somewhat surprising that the same technique applied to OF gets lower performances, could the authors provide some explanations/insights?
* Line 258: concerning the claim "MVCGC is robust to motion vectors under different encoding configurations": is not really clear how this is proved. In table 2 MV_CoViAr is used for initialization only.
* Line 303: I think that it is "by retrieval with only" RGB "are even higher"
* Appendix, fig 1: the underlying assumptions is that semantically similar videos are close in both RGB and MV space. How do the authors think the model would handle failures of such assumption, e.g., mopping a gymnastic floor, biking in a volleyball arena, etc.?
--------------------
* **Correctness:** The proposed method and the experimental evaluation are sound.
* **Clarity:** The paper is well written and easy to follow. The content is well organized: contribution is expressed concisely and the experimental part is well explained.
* **Relation to prior work:** Authors relates their work with previous literature while concisely highlighting their contribution.
* **Reproducibility:** Architecture, method and optimization hyperparameters are available for implementation, missing augmentation parameters (random cropping, blurring, jittering) and missing code limits the reproducibility of the work though.
* **Originality**: The contribution of this paper is incremental with respect to cited papers, the real advantages of the technique are not thoroughly highlighted in the ablation part.
------------------------
[1] Han, Tengda, Weidi Xie, and Andrew Zisserman. "Self-supervised Co-Training for Video Representation Learning." Advances in Neural Information Processing Systems 33 (2020): 5679-5690.

[2] Tian, Yonglong, Dilip Krishnan, and Phillip Isola. "Contrastive multiview coding." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16. Springer International Publishing, 2020.

[3] Bachman, Philip, R. Devon Hjelm, and William Buchwalter. "Learning Representations by Maximizing Mutual Information Across Views." Advances in Neural Information Processing Systems 32 (2019): 15535-15545.

[4] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

limitations_and_societal_impact: N/A. However, I would argue that no specific discussion is needed on potential negative societal impact for a contrastive learning method.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper tackles the problem of self-supervised video representation learning. The authors propose a compressed video based self-supervised learning approach, where the model is forced to learn matching representations between raw RGB clip and its corresponding motion vector clip. This is achieved using a relatively standard contrastive learning formulation. The authors then demonstrate that their model achieves state-of-the-art performance on a few downstream tasks (i.e., action recognition, and video retrieval) while also being significantly more efficient (particularly in terms of the video processing time).
main_review: Strengths:
- The paper is written clearly, and it addresses a well motivated and an important research problem.
- A nice idea of using compressed video for self-supervised feature learning.
- The proposed approach delivers performance that is comparable/better to previous methods (e.g. CoCLR [Han et al., 2020b]) while being much more efficient.
- Thorough ablation studies.

Weaknesses:
- The major weakness of this paper is its limited technical novelty. While the idea of using compressed videos in the context of self-supervised learning is new and interesting, overall the proposed approach is largely built on the ideas from CoCLR [Han et al., 2020b]. Having said this, the proposed approach seems to be both more effective and more efficient than CoCLR. Therefore, I think this would be a good "poster" paper.
- The proposed approach performs quite a bit worse than the recent CVPR 2021 methods such as CVRL, and especially the method of Feichtenhofer et al. "A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning" (CVPR 2021). Having said this, given a very large computational cost of both of these prior methods (i.e. 800 epoch training), I think that it would be unfair to compare the proposed MVCGC approach directly with these methods. The authors should still add the second reference to their related work section, and also to their quantitative comparisons. A discussion on why the direct comparison between these methods is unfair would also be useful.

Questions / Minor Details:
- In L85-88, the authors mention that unlike CoCLR, they construct sample pairs between different views of different clip (i.e., RGB frames of one clip and motion vectors of another clip from the same video). However, the caption of Fig.3 says that the motion vector clip corresponds to the raw RGB clip. Therefore, I was confused, which one is it? Also, if it's the former, could the authors also ablate the latter?
- Typo in Table 3 (it should be Retrieval instead of Retrival). 
- In the third part of Table 4, the authors highlight/bold the numbers corresponding to their approach even though CVRL achieves higher accuracy.
- If the authors could include the linear evaluation on Kinetics-400 (similar to "A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning") that would be great. I think the evaluation on UCF and HMDB is a bit saturated and outdated. Therefore, it would be useful to have additional numbers on Kinetics, which is a bit more challenging dataset. 
limitations_and_societal_impact: Limitations were not discussed.
review_title: null

Meta-review: The paper received the following final ratings: 6, 6, 7, 4.

Reviewer rcKJ raised concerns about the close technical reliance on CoCLR [Tengda et al., NeurIPS 2020] and the lack of understanding about the impact of cross guidance on performance. The authors rebutted these points in their response. Unfortunately, Reviewer rcKJ failed to provide post-rebuttal comments, but the ACs found the points made by the authors convincing.

The other reviewers appreciated the responses provided by the authors, especially the clarification about the similarity calculation, the differences wrt CoCLR, and the inclusion of the linear evaluation on Kinetics-400. As noted by Reviewer Rmsi, an ablation on how the different compression settings affect performance would further strengthen this work. The ACs agree with the majority recommendation of acceptance expressed by the reviewers.
Decision: Accept (Poster)