Title: Compressed Video Contrastive Learning
Authors: Yuqi Huo
Mingyu Ding
Haoyu Lu
Nanyi Fei
Zhiwu Lu
Ji-Rong Wen
Ping Luo
Venue: NeurIPS 2021
Keywords: compressed video
self-supervised learning
motion vector
contrastive learning

Abstract: This work concerns self-supervised video representation learning (SSVRL), one topic that has received much attention recently. Since videos are storage-intensive and contain a rich source of visual content, models designed for SSVRL are expected to be storage- and computation-efficient, as well as effective. However, most existing methods only focus on one of the two objectives, failing to consider both at the same time. In this work, for the first time, the seemingly contradictory goals are simultaneously achieved by exploiting compressed videos and capturing mutual information between two input streams. Specifically, a novel Motion Vector based Cross Guidance Contrastive learning approach (MVCGC) is proposed. For storage and computation efficiency, we choose to directly decode RGB frames and motion vectors (that resemble low-resolution optical flows) from compressed videos on-the-fly. To enhance the representation ability of the motion vectors, hence the effectiveness of our method, we design a cross guidance contrastive learning algorithm based on multi-instance InfoNCE loss, where motion vectors can take supervision signals from RGB frames and vice versa. Comprehensive experiments on two downstream tasks show that our MVCGC yields new state-of-the-art while being significantly more efficient than its competitors.

TL;DR: We propose an efficient and effective framework for self-supervised representation learning from compressed videos (without decompressing off-the-fly).

Review discussion:
It is not hard to tell that this work is based on a previous NeurIPs'20 paper CoCLR, with only two differences: 1. The proposed MVCGC uses motion vectors extracted from compressed videos as the second view of a video clip while CoCLR uses unsupervised optical flows; 2. MVCGC tries to embed two views into the same latent space while CoCLR embeds RGB and optical flow into separate spaces.
Other than these, MVCGC shares the contrastive learning approach  and the hard positive mining from the second view with CoCLR. Given the high similarity with CoCLR, this work is not sufficiently novel for publication in NeurIPs.
- Although motion vectors are directly extracted from compressed videos, obtaining the RGB frames still needs a complete decoding. The only merit left is to avoid the additional computation of optical flows.
But one could argue that moderate-quality optical flows are not very expensive to compute or store after pre-processing.
- It is debatable whether RGB and motion should be embedded into the same latent space. It is not fair to say that "CoCLR fails to learn the correspondence between two views" (P5L184). The cross-view hard positive mining proposed in CoCLR (which is also adopted in MVCGC) builds a strong bond between two views although they are not embedded into the same latent space.
In fact, the performance of MVCGC is only slightly better than that of CoCLR.
---- after rebuttal ----
Thanks the reviewer for the response, which did clear up a few issues.
However, the concern on limited novelty remains.
Therefore, this reviewer will keep the original rating.
No
Q1. Distinct contribution compared to COCLR.\
A1. Thanks for pointing this out. Our MVCGC is different from CoCLR in two aspects: (1) CoCLR decodes compressed videos off-the-fly and computes optical flows (OF) with RGB frames, which is storage- and computation-intensive.
In contrast, our MVCGC decodes RGB frames and motion vectors (MV) directly from compressed videos on-the-fly with high efficiency. (2) There is no information exchange in the pre-training stage of CoCLR, while MVCGC leverages cross guidance to capture mutual information between two streams. Note that our work is the first to exploit contrastive loss in compressed video SSL by leveraging RGB and MV directly from compressed videos. We show that when these two streams are seamlessly integrated by our MVCGC, they carry enough information for video understanding and their fusion is even more powerful than OF+RGB.
A2. One could argue that moderate-quality optical flows are not very expensive to compute or store after pre-processing.\
Q2. Thanks. Table 2 shows the comparison between MV and low resolution OF. We find that by only reducing the resolution of the raw OF to a half (i.e., the same value in each 2x2 macroblock), it performs even worse than the motion vector, indicating that although MV is commonly regarded as OF with lower resolutions, it is still able to well represent the action motions in a somewhat distinct way.
Q3. It is debatable whether RGB and motion should be embedded into the same latent space.\
A3. Good suggestion! We demonstrate the advantages of our MVCGC over CoCLR in the upper part of Table 3. We find that our MVCGC indeed helps the RGB network to learn more discriminative representations than CoCLR when the extra information of the motion vector view (or the optical flow view) is leveraged. Especially, MVCGC outperforms CoCLR in utilizing MV (73.1|60.8 vs. 67.4|53.8), which is not "only slightly better". This shows the effectiveness of our end-to-end cross guidance contrastive learning stage.
## Pros
The computational and storage efficiency for video methods are crucial topics to be studied in the future. An increasing amount of visual information is shared in video formats on the web, but there is relatively less work on processing videos in an efficient fashion. It is also great that the paper is pointing to the limitations of prior approaches that seemingly have not thought deeply about computational efficiency - e.g. redundant motion computation in IMRNet. The resulting reduction in computational time (Figure 1, Table 1) and codec-agnocity (L51) are great, though I doubt if the paper itself can be credited for these advances (see cons).
The paper also follows a commendable practice of showing the settings like available test-time data, architecture, resolution, and the training datasets in Tables 4. Oftentimes, such tables lack those details, making it practically impossible to discern performance improvements from the proposed method from other technological advances.
The experimental analysis in Table 3 looks great. Compared to the closes counterpart, CoCLR, MVCGC improves the performance greatly by tuning the representation with the proposed cross-view info-NCE (and the hard-positive mining scheme).
## Cons
Exactly how does MVCGC improve efficiency? The MVCGC-led improvement in efficiency is because of the replacement of expensive data preprocessing steps, such as optical flow estimation and residual computation, in previous approaches (IIC, MemDPC, CoCLR, IMRNet). It is great that MVCGC has opted for motion vectors that seem to be readily available through the decoding, which is anyways needed to obtain the RGB frames.
However, the paper does not go into full detail about how the decoding works and how fast it is to compute the motion vectors. The method seems to use the "pyav" library (L202), but the paper should elaborate much further on the decoding bit if it wants to claim the contribution here. For now, it is difficult to fully acknowledge its efficiency-related contribution here.
Related to this, the authors argue that MVCGC is efficient and effective and that it is the first to exploit contrastive loss in compressed video self-supervised learning (L59-61).
However, for example, CoCLR has also used the info-NCE loss on RGB + flows, just like MVCGC. If the core contribution is on the "compressed video" bit, I am lost. MVCGC also operates on the compressed RGB + flows, though it uses a cheaper algorithm to compute the flows. It would be great if the authors can clearly state this contribution again and provide supporting evidence.
In fact, when I first heard the term "compressed videos" (title and many other places), I thought the inputs to the model were encoded byte streams.
However, MVCGC still requires decoded frames and motion vector fields. It is a bit misleading for me to say that the model operates on "compressed videos".
Please move the following bit to the method section: "Our MVCGC proceeds in two stages ... to minimize the loss in Eq. (5)" (L206). Before reading this, I have been wondering why MVCGC would not exploit the single-view contrastive learning.
## Conclusion
The paper achieves great improvements in representation qualities and computational efficiency.
However, I am still hesitating to recommend an acceptance because it is unclear how much the paper could claim its contribution regarding the computation efficiency. Please provide the full technical details around the decoding step for me to evaluate the contribution more precisely.
## Post rebuttal
Thank you for the response. I would also like to thank other reviewers for their expert reviews. I still believe the paper either lacks contribution or does not present its contributions clearly. I would recommend another peer-review round to enhance the impactfulness for this paper. My final recommendation is a rejection for this round.
No
Q1. Exactly how does MVCGC improve efficiency?\
A1. Thanks. We implement our method using "pyav" library. During the decoding step, given various codecs of video, "pyav" will capture motion vectors for each frame from the bitstream and convert it to tensors. The motion vector's computing time is far less than the network forward time, and these two stages are count together as "inference time" as shown in Table 1.
Q2. Contribution clarification.\
A2. Our MVCGC is different from CoCLR in two aspects.
First, CoCLR decodes compressed videos off-the-fly and computes optical flows (OF) with RGB frames, which is storage- and computation-intensive.
In contrast, our MVCGC decodes RGB frames and motion vectors (MV) directly from compressed videos on-the-fly with high efficiency.
Second, there is no information exchange in the pre-training stage of CoCLR, while MVCGC leverages cross guidance to capture mutual information between two streams. Our contribution is that, for the first time, we exploit how to solve self-supervised video representation learning in both storage-efficient as well as effective ways. We achieve this goal by leveraging the MV stream in compressed videos as an extra stream in our proposed contrastive learning algorithm. Extensive experiments by applying the learned representations on two different downstream tasks demonstrate that our MVCGC achieves state-of-the-art performance while being more efficient than its counterparts.
Q3. It is a bit misleading for me to say that the model operates on "compressed videos".\
A3. Sorry for the confusion. We follow the term "compressed videos" by CoViAR [1], and the precise meaning of decoding compressed videos is decoding videos on-the-fly, instead of the previous off-the-fly counterparts. We will clarify in the revised version.\
[1] Wu, Chao-Yuan, et al. "Compressed video action recognition." CVPR 2018, pp. 6026-6035.
Q4. Sentence issue.\
A4. Thanks for pointing this out. We will do it in the revision.
Thank you for the response. I would also like to thank other reviewers for their expert reviews.
I share the novelty concerns with other reviewers (3btZ, UKWL). I tend to echo the comment "I recommend another round of revision to really clear up exactly what the novelty of the paper is if it isn't coming across clearly" by UKWL. I am struggling even after reading the rebuttal. I am unsure if using modules in "pyav" can really be considered a sizable contribution. The same goes for the use of MV on top of OF. I do not mean the paper should have introduced a new fancy module, but just that the paper can be presented better to make the actual contribution stand out.
For example, I am having a hard time recognising the statement "[o]ur contribution is that, for the first time, we exploit how to solve self-supervised video representation learning in both storage-efficient as well as effective ways" as a contribution. It sounds too blurry to count as a concrete contribution. The next sentence "We achieve this goal by leveraging the MV stream in compressed videos as an extra stream in our proposed contrastive learning algorithm." is also very generic. I would also recommend another peer-review round to enhance the impactfulness.
I'm very on the fence about this paper and I am open to upgrading my review provided sufficient feedback from the authors.
On the positive side, I note that this method appears to have good performance on the self-supervised tasks and the addition of a contrastive loss on bitstream data is novel if incremental. Beyond the contrastive loss however, I struggle to find additional novelty.
The paper seems quite fixated on speed in the sense that the method does not depend on the "coviar method" of computing error residuals, choosing to discard them and instead use only I-frame pixels and P-frame motion vectors. I did not find this argument convincing. Retrieval of true residuals from an encoded bitstream vs. computing them from motion frames is an implementation detail and it would be a system level task to correct this. Any "real world" use of coviar-like technologies would be re-implemented using a proper bitstream analysis library or would modify the codec directly to produce error residuals with no computation, a task which is time consuming but not overly difficult (see [1] for example).
This same argument extends to coviar (and the follow on works) use of MPEG-4 part 2: this codec was chosen mainly for ease of implementation of research code and would be easily corrected in production to support modern codecs which would alleviate any re-encoding concerns. Nothing about coviar mandates MPEG-4 part 2.
Finally, the paper indicates that residual streams are "proved to only supply a marginal performance gain" (ln103) without evidence of such proof. I disagree with this characterization as the residual data can provide additional information about scene elements that may be missing from I-frames (see [2, 3], if P-frame residuals were not important, multi-frame restoration would have no advantage).
I'll also note that the closest prior work to this paper, IMRNet, uses a 3D resnet backbone whereas the proposed method uses an S3D backbone which may not be a fair comparison of the methods unless IMRNet was re-implemented with the matching backbone [2].
Finally, the bolding on the tables (specifically Tab. 4, 5) was somewhat confusing as the proposed method is always shown in bold but is not always the best performing method.
Given the stated flaws and the marginal novelty of the method, I am inclined to reject the paper but I am open to discussion with the authors on all points.
1. https://vqeg.github.io/software-tools/encoding/modified-avc-codec/
2. Wang, Xintao, et al. "Edvr: Video restoration with enhanced deformable convolutional networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2019.
3. Xue, Tianfan, et al. "Video enhancement with task-oriented flow." International Journal of Computer Vision 127.8 (2019): 1106-1125.
4. Musgrave, Kevin, Serge Belongie, and Ser-Nam Lim. "A metric learning reality check." European Conference on Computer Vision. Springer, Cham, 2020.
I have read the authors' rebuttals as well as the other reviews.
No
Q1. Contribution clarification.\
A1. Thanks. Our contribution is that, for the first time, we exploit how to solve self-supervised video representation learning in both storage-efficient as well as effective ways. We achieve this goal by leveraging the MV stream in compressed videos as an extra stream in our proposed contrastive learning algorithm. Extensive experiments by applying the learned representations on two different downstream tasks demonstrate that our MVCGC achieves state-of-the-art performance while being more efficient than its counterparts.
Q2. The paper fixated on speed in the sense that the method does not depend on the "CoViAR method" of computing error residuals, which is not convincing.\
A2. Thank for the suggestion! This work concerns self-supervised video representation learning, which aims to find an efficient way to learn from compressed videos. We claim that "residual proved to only supply a marginal performance gain" from Tables 1 and 2 in CoViAR [1], where the reported results show that in the action recognition task, residuals indeed only supply a marginal performance gain. We will do a thorough survey in the revised version.\
[1] Wu, Chao-Yuan, et al. "Compressed video action recognition." CVPR 2018, pp. 6026-6035.
Q3. IMR-Net unfair comparison in Table 4.\
A3. Thanks for pointing this out. Note that we try to compare all approaches as fairly as we can, although variations are unavoidable in terms of architecture, training data, and resolution. Compared to IMRNet, MVCGC is trained using a shallower architecture (S3D-23 vs. R3D-50) with fewer parameters (7.9M vs. 33.1M) and smaller resolution (128 vs. 224). Under such setting, the effectiveness of MVCGC can even be better shown.
Q4. Bold number in Table 4.\
A4. Sorry for the confusion. We will clarify.
Firstly I want to thank the authors for their rebuttal.
While it cleared up some of my issues I still find the novelty quite limited and I recommend another round of revision to really clear up exactly what the novelty of the paper is if it isn't coming across clearly. As stated in my initial review, the criticism around certain systems level design choices of prior works should probably be removed as that argument is not compelling.
This papers main strength is a slight improvement on some benchmark accuracies coupled with a speed increase but I'm not convinced that it outweighs the novelty concerns w.r.t IMRNet.
Pros:
1. The proposed idea is straightforward and the writing is easy to follow.
2. The accuracy seems superior compared to previous compressed video action recognition models.
3. It is interesting that the authors bring up the discussion about the actual decoding process of compressed video action recognition models.
Cons:
1. The main concern is that the authors did not justify why the proposed method is strongly entangled with compressed videos. The proposed method is motivated by learning more effective representations from compressed videos. The proposed cross-guidance could be applied two jointly trained RGB feature extractors without using any cross- stream information. Without this kind of justification, it is not very convincing that the proposed method specifically resolves an issue in compressed video action recognition or especially leveraged certain property of compressed videos.
Moreover, the experimental results are somewhat puzzling: considering that downsampled optical flow should be highly similar to motion vector sampled following CoViar, it is hard to come up with any possible reason that leads to the huge performance gap as shown in Table 2.
2. The second concern is on the unfair comparison of efficiency with previous methods. The inefficiency of computing residue in  previous methods is really an implementation issue rather than a methodology issue.
Moreover, the assumption on IP encoding actually holds in many real-world use cases, .e.g, streaming.
Therefore, it is not perfectly fair to compare with previous methods directly counting these steps in terms of inefficient implementation or different use scenario.
Moreover, it is not clear what the batchsize is when conducting this comparison. This could lead to completely different conclusion in terms of inference speed.
3. In Table 4, IMR-Net actually uses I frames rather than RGB stream but the author still used RGB to represent IMR-Net's test videos, which actually contradicts with Fig 2 that says IMR-Net discards RGB stream.
Although they are in both RGB format, I frames are much sparser compared to RGB stream and thus should be considered separately.
After rebuttal: The reviewer thank the authors for reponse.
However, the concerns are not well addressed.
The justification on the comparison with ensemble of two RGB models is not convincing. The features of two RGB models trained from different random initialization also has dense correspondence. The authors should show that the improvement didn't come from a simple ensemble of two RGB models. The response to the justification between MV and OF is also vague.
Unless the authors could show major difference between MV and downsampled OF, it cannot explain the bad performance of OF+RGB.
No
Q1. Why the proposed method is strongly entangled with compressed videos? Considering that the proposed cross-guidance could be applied to two jointly trained RGB feature extractors without using any cross-stream information.
Moreover, it is hard to come up with any possible reason that leads to the huge performance gap as shown in Table 2.\
A1. Thanks for pointing this out.
But we disagree with this comment. (1) The cross-guidance stage could be conducted only if there exists a correspondence relationship, i.e., RGB and MV for the same clip. (2) Although OF is better than MV when they are used alone, MV can be combined with RGB seamlessly, and thus MV+RGB performs even better than OF+RGB (see Table 2). This is a little counter-intuitive but can be explained. Note that MV and RGB are two original components of video codecs and MV+RGB carries more raw video information than OF+RGB. This enables our MVCGC to better learn the correspondence within the two streams (i.e., MV and RGB), leading to higher accuracy and efficiency.
Q2. Unfair comparison of efficiency with previous methods.\
A2. Thanks. (1) As shown in Figure 2, IMRNet leverages an additional residual stream, while MVCGC decodes motion vectors from compressed videos directly, which alleviates computation burdens from the methodology’s perspective. (2) We claim that the drawback of IMRNet (or CoViAR) is not that they can only handle P-frames, but that they can only handle videos with the outdated MPEG-4 Part 2 codec. Any type of videos need to be re-encoded by IMRNet.
In comparison, our MVCGC can decode any type of compressed videos directly, including IP streaming codecs, indicating the generalizability of MVCGC. (3) All experiments in Table 1 set batch size as 1 for fair comparison.
Q3. IMR-Net unfair comparison in Table 4.\
A3. Sorry for the confusion. We will clarify. The views of IMRNet should be "I+MV+Res". Note that we try to compare all approaches as fairly as we can, although variations are unavoidable in terms of architecture, training data, and resolution. Compared to IMRNet, MVCGC is trained using a shallower architecture (S3D-23 vs. R3D-50) with fewer parameters (7.9M vs. 33.1M) and smaller resolution (128 vs. 224). Under such setting, the effectiveness of MVCGC can even be better shown.

Meta-review:
role: PC
data: comment: The reviewers acknowledged that this paper tackles an important problem of making video pipelines compute/storage-efficient. However, they raised concerns over the limited novelty, especially compared to CoCLR and IMRNet. They also commented that the arguments around codecs and decoding mechanisms used in prior approaches are system-level design choices rather than a scientific challenge, and questioned whether replacing coviar-like video processing with the `pyav` python library can be considered a novel technical contribution. The rebuttal addressed some of the questions but unfortunately did not perfectly clear up the major concerns. Based on this, I am recommending a rejection at this time, but would invite the authors to improve their paper based on the reviewers' suggestions.

Decision: Accept (Poster)