Compressed Video Contrastive Learning
Authors: Yuqi Huo
Mingyu Ding
Haoyu Lu
Nanyi Fei
Zhiwu Lu
Ji-Rong Wen
Ping Luo
Venue: NeurIPS 2021 Poster (2021)
Keywords: compressed video
self-supervised learning
motion vector
contrastive learning

Abstract: This work concerns self-supervised video representation learning (SSVRL), one topic that has received much attention recently. Since videos are storage-intensive and contain a rich source of visual content, models designed for SSVRL are expected to be storage- and computation-efficient, as well as effective. However, most existing methods only focus on one of the two objectives, failing to consider both at the same time. In this work, for the first time, the seemingly contradictory goals are simultaneously achieved by exploiting compressed videos and capturing mutual information between two input streams. Specifically, a novel Motion Vector based Cross Guidance Contrastive learning approach (MVCGC) is proposed. For storage and computation efficiency, we choose to directly decode RGB frames and motion vectors (that resemble low-resolution optical flows) from compressed videos on-the-fly. To enhance the representation ability of the motion vectors, hence the effectiveness of our method, we design a cross guidance contrastive learning algorithm based on multi-instance InfoNCE loss, where motion vectors can take supervision signals from RGB frames and vice versa. Comprehensive experiments on two downstream tasks show that our MVCGC yields new state-of-the-art while being significantly more efficient than its competitors.

Compressed Video Contrastive Learning
Yugi Huo 2,3 Mingyu Ding Haoyu Lu Nanyi Fei2.3
22 Ji-Rong Wen Ping Luo Gaaoling School of Artificial Intelligence, Remmin University of China. Beijing, China Beijing Key Laboratory of Big Data Management and Analysis Methods
Zhiwu Lu L
3 School of Information, Renmin University of China, Beijing, China "The University of Hong Kong, Pokfulam, Hong Kong, China
(bnhony, luzhiwu] Gruc. edu. cn Equal contribution Corresponding author
Abstract
This work concerns self-supervise video representation learning (SSVRL), one topic that has received much attention recently. Since videos are dtorage-intensive and contain rich source of visual content, models designed for SSVRL are expected to be storage- and computation-effiiinn most existing methods only focus one of the two objectives, failing to consider both at the same time. In this work, for the first time, the seemingly contradictory goals simultaneously achieved by exploiting compressed videos and capturing mutual information between two input streams. Specifically, novel Motion Vector based Cross Guidance Contrastive learning approach (MVCGC) is proposed. For storage and computation efficiency, we choose to directly decode RGB frames and motion vectors (that resemble low-resolution optical flows) from compressed videos on-the-fly fly. To enhance the representation ability of the motion vectors, hence the effectiveness of method, we design cross guidance contrastive learning algorithm based on multi-instance InfoNCE loss, where motion vectors can take supervision signals from RGB frames and vice versa. Comprehensive experiments on two downstream tasks show that our MVCGC yields new state-of-the-arr while being significantly more efficient than its competitors.
well effective. However,
Introduction
Recent self-supervised image representation learning approaches [He et al.) 2020 Chen et al. 2020
have been reported to outperform supervised ones a wide range of downstream tasks by (I) leveraging a large amount of unlabeled data available online for pre-training, and (2) designing
powerful algorithm to discriminate unlabeled samples with different semantic meanings. However, the situation for self-supervised video representation learning (SSVRL) is somewhat different: self-supervised methods still perform worse than supervised ones, since videos extremely storage- intensive (scalability thus becomes a serious issue) and have richer source visual content (SSVRL thus becomes very difficult). Therefore, designing storage-and-oe aandd-oomputtioo--fffiiieee effective models for SSVRL remains a challenging problem that has not been well studied.
well as
Existing state-of- offte-of-the-rrr aht methods [Han et al., 2020b] Tao et al. 2020 Huo et al. [222 mainly focus on designing effective algorithms without considering the storage and computation costs during large-scale video self-supervised training. Particularly, in order to leverage both appearance and temporal information in the video, they exploit the optical flow as an extra view to complement RGB stream. However, this procedure is both storage- and cemputatioo-intensin because of storing the decoded frames and computing optical flows, respectively (e.g., it costs more than 100GB for
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

88
MVCGC (ours)
COCLR
85
Top-1 Accuracy (%)
MemDPC
82
79
IMRNet
76
73
10"2
10
10a
Processing Time Per-video (second)
Processing time per-video (i.e., data pre-processing and inference) and the accuracies for different methods UCF101. Each method is represented by circle, whose size represents the storage size occupied by the test data. MVCGC achieves the highest accuracy under a significantly reduced storage budget compared to its counterparts. More details can be found in Table !
Figure I:
frame storage and a number of days for computing optical flows (Han et al. 2020b on the UCF101 dataset (Soomro ef al. 2012].. This clearly hinders large-scale video self-supervised training.
Considering that videos are already stored in compressed formats to reduce storage requirements, natural choice is to resort to on-the-fly decoding, i.e., decoding frames from compressed videos during storage. Moreover, since motion vectors encoded in
training/inference without introducing any compressed videos resemble low-resolution optical flows in describing local motions, they can be used avoid the costly optical flow computation. Consequently, compressed videos inherently contain both static and motion information that is suitable for video representation learning. Recently, pretext task based method called IMRNet (Yu et al.) [02T] leverages the standard process in CoVVAR [Wu al. 2018 to decode compressed videos, but inevitably encounters the inefficiency and ineffectiveness problems: (1) The outdated CoViAR is not truly on-the-fly, i.e., it re-encodes videos and stores them before decoding, which is storage and computation inefficient. (2) Motion vectors are less discriminative and thus weaker than optical flows, resulting in sub-optimal performance compared with state-of-the-art methods using optical flows, as shown in Figure I As result, exploiting the compressed videos efficiently, followed by learning discriminative representations, is still challenging.
Motivated by the above observations, we propose Motion Vector based Cross Guidance Contrastive learning approach (MVCGC), which has several appealing benefits: (1) It able decode RGB frames and motion vectors from compressed videos with various codecs. The storage and computation budgets are alleviated since videos can be decoded without re-encoding. (2) Our MVCGC can learn discriminative features from both RGB and motion vector streams. This is achieved by designing cross guidance contrastive learning algorithm based on multi intttiiittancee InfoNCE loss: each sample pair is constructed by one RGB clip and one motion vector clip, and multiple positive samples can be mined by calculating the similarity two views. Therefore, the two views can take supervision signals from each other to improve the representation quality of both streams, especially for the motion vector stream (whose learned representations are even comparable to optical flow features).
Our main contributions are three-fold: (1) We propose an efficient and effective framework called knowledge. we are the first to exploit contrastive loss in compressed video self-supervised learning. (2) We design novel contrastive learning algorithm capture mutual information between RGB frames and motion vectors from compressed videos. Importantly, we even make the learned features of motion vectors as representative as those optical flows. (3) Extensive experiments by applying the learned video representations two downstream tasks (i.e., action recognition and action retrieval) across different benchmarks demonstrate that our MVCGC achieves ofae performance while being more efficient than its counterparts (see Figure
MVCGC that can learn representations from compressed videos directly. To the best

---

2 Related Work
Self-Supervised Video Representation Learning. Existing self-supervised methods for video
representation learning can be divided into two categories: (1) One common way to learn good
representations is to define and solve pretext tasks, such as ordering frames Misra et al. [016 Lee er
al.| [017;; Xu et al. [2099] Luo et al. [020], predicting the speed of videos Yao et al. [2220] Benaim
et al. 2020; Wang et al.) 2020]. and solving space-time cubic puzzles [Kim et al. 2019]. (2) Another
more recent way focuses instance discrimination, which leverages distance- based contrastive loss
and distinguishes the positive samples from a group of negative ones [Oord el al.) 2018 He et al.
2020 Chen et al. 2020]. Recently, many efforts based contrastive learning have demonstrated
promising results. They either focus only on RGB frames (Han et al. 2019) Zhuang et al. [020]]
Wang et al.) 2020 Kong al. 2020 Wang et al. 2021] or introduce an additional optical flow view
in order to achieve state-of-the-arr accuracy [Han et al. 2220aa Tian et al.) 2020) Tao et al. 2020
Han et al. 2020b]. Considering the heavy computational and stta-st burdens for computing the
optical flow, our proposed MVCGC replaces it by introducing the motion vector from the compressed
video as a new view, which significantly cuts down the computational and storage costs. Moreover,
MVCGC is different from the previous approaches [Han et al. [2020a; Tian er al. [2020; Tao et al.
2020, Han ef al.) [220bb] in how contrastive loss and sample pairs are defined. CMC [Tian al. 2020) and TIC TTo et | 2020 use the single-instance InfoNCE loss, resulting in neglect of hard positives.
MemDPC |Han et al.) [020a] and COCLR (Han et al.) 2020b] construct sample pairs of contrastive loss in RGB frames and optical flows separately, ignoring the correspondence between different views. In contrast, our MVCGC constructs sample pairs between different views of different clips (i.., RGB frames of one clip and motion vectors of another clip from the same video) and applies the multi-instance InfoNCE loss. We use two views as cross guidance to train the representations of two streams simultancously and learn mutual information between RGB frames and motion vectors.
Compressed Video Representation Learning. Earlier approaches have explored the compressed video for supervised recognition (Zhang et al. [018]] Wu et al. 2018:; Shou et al.) 2019], in which large-scale supervised datasets (such as ImageNet [Russakovsky et al. 2015]) are used for pre- training. In contrast, we focus on self-supervised learning, i.e., only unlabeled data is used in the pre-training stage. After the first exploration of using compressed video to learn "the arrow of time"(Wes et al. 2018). a related work, IMRNet (Yu et al. 2021]. also explores compressed video self-supervised learning besides supervised learning. Our proposed MVCGC is fundamentally different from IMRNet in three key aspects: (1) Instead of focusing on designing pretext tasks as in IMRNet, our MVCGC leverages contrastive loss. (2) With the outdated MPEG-4 Part 2 codec [Huo et al. 2020 IMRNet has to re-encode and store videos for data loading, whereas our MVCGC can decode compressed videos on-the-fly fly and save storage spaces. (3) Besides the decoded RGB and motion vector streams, IMRNet leverages an additional residual stream computed from the two streams, which has already been proved only supply a marginal performance gain. In comparison, our MVCGC utilizes the first two streams and thus alleviates computation burdens.
3 Methodology
This work aims to learn robust video representations efficiently from compressed videos by designing a contrastive learning algorithm. this section, we first review the basics of compressed video and discuss the advantages of our decoding process compared to existing counterparts. Afterward, we detail our MVCGC on how learns effective feature representations and captures mutual information between RGB frames and motion vectors from compressed videos.
3.1 Basics of Compressed Video
Considering the redundancy in consecutive frames, codecs are proposed to compress videos for efficient storage, i.e., compress one frame by reusing contents from another frame (termed reference frame) and only store the change. According to the choice of reference frame, frames are categorized into three types: I-frame (Intra-coded frame), P-frame (Predictive frame), and B-frame (Bi-predictive frame). I-frame has no reference frame and is directly stored in image format, while P- and B-frame take reference frames forwardly and bi-directionally, respectively, and store the change in the format of the motion vector and residual: A codec first divides frame into macroblocks of size such
3

---

Comp Video
Comp. Video
Re enode
Comp, Video
Comp. Video
Decdde
Discarded
Decode
Compute
Deooe
Compute
RGB
(a)
(b)
(c)
Figure 2: Usage of compressed videos (Comp. Video) for SSVRL: (a) Traditional methods decode videos off-the-fly; (b) IMRNet re-encodes videos. followed by the additional residuals computation with RGB frames being discarded afterward; (c) MVCGC decodes RGB frames and motion vectors on-the-fly. "RGB", "MV", "OF", and "Res" denote RGB frames, motion vectors, optical flows, and residuals, respectively. Procedures colored with red computation expensive, and their backgrounds which are colored with orange or blue, indicate the happening place of them, Le., videos in the orange background are stored the disk while those the blue are decoded in cache memory.
as 16x16 and then searches the most similar image patch in the reference frame for each of these macroblocks. Lastly, the motion vector is represented by the displacements between macroblocks and the searched image patches, and the residual is computed as the difference between the target frame and its reference frame warped by the motion vector.
Discussion As shown in Figure 2 existing SSVRL methods can be categorized into three groups according to how they exploit videos for SSVRL. (1) Traditional methods off-the-fly decode all RGB frames from compressed videos and compute optical flows with RGB frames, which is storage- and computation-intessit leveraging the process in CoViAR [Wu et al. 2018). which only supports the outdated MPEG-4 Part codec (Sikoral 1997] and lacks generalizability In this way. videos with modern codecs (e.g., H.264/AVC [Wiegand et al.) 2003] and HEVC (Sullivan et al. 2012)) in current application scenarios need to be re-encoded and re-stored, downgrading the computational efficiency. IMRNet residuals of P-frames as third view: since residuals post-computed by two decoded views, iee,, RGB frames and motion vectors, which introduces extra computation requirements. Moreover, in IMRNet, RGB frames decoded only for computing residuals, while being discarded afterward, hindering the data usage effectiveness. (3) Our proposed MVCGC can decode RGB frames and motion vectors directly from various video codecs without the aforementioned costly off-the-fly frame extracting, video re-encoding, or residual computing, and is thus high high-efficienn efficiiefcyy and application -iexibiiiii..iiitt
(2) The latest IMRNet (Yu et al. [02T] decodes compressed videos by
not be extracted in practice, it is
3.2 Motion Vector based Cross Guidance Contrastive Learning
The goal of SSVRL is to train an encoder that can embed video samples effectively for various downstream tasks (e.g., action recognition and retrieval). Towards this end, method is based on contrastive learning with InfoNCE loss [Oord et al., 2018). as shown in Figure Next we start by reviewing base single-vies view model with InfoNCE loss, and then extend the base model using cross guidance contrastive learning as our MVCGC.
Base Single- Kingle-View View Model with InfoNCE Loss Formally, let D 121, , ZNN } denotes dataset of N compressed video samples, where each clip z; (i 1,2,.... ,N) consists of T frames. In the single-view scenario, these clips are in the same view, i.e., D is dataset of only RGB clips or motion vectors. The objective of learning an effective encoding function f(-) for the single stream is thus achieved by discriminating one positive sample of each video clip from its negatives with InfoNCE loss [Oord et al.) [2018].
Let ((: 0)) be an augmentation function applied to D, where 0 is sampled from set of transformations O. For each clip x, D (i- = 1.2...., N), we denote the encoded feature vector of n; as 2; = f(x)). the positive sample set as P. [f(t(z;:)) 0 θ), ), and the negative set as N₁ ((((z)(0)))

---

N
Cross Guidance
Hard Postiive Mining
Lavenn
86-81
PN
K
Tops Striletty
Trpa Smrfantt
Guidanoe
N
NN
NNN
(a]
(b)
Figure 3: Overview of the proposed MVCGC architecture. (a) From raw RGB clip and its corresponding motion vector clip, MVCGC computes the anchor and the positive features, while other samples in the dataset are used to construct the negative set. (b) The complete design of the hard positive mining stage.
i, 0 O). where [P.] and |
N
The InfoNCE loss is then defined as:
exp (24 2p/T)
LIntaNCB
Σ log
cxp (z: zp/T) + Ex. ENG exp (2; =n/T)²
where 2p E P; is the only positive sample of Z4, 2, 2p (or 2; =n, refers to the dot product between two vectors, and 7 > 0 is temperature parameter. This instance discrimination loss forces the base model to learn a higher similarity score between a given sample 2; and its augmented/target sample Zp while a lower one between 2; and other instances 2n E N,
Cross Guidance Contrastive Learning with Multiple Views While the InfoNCE loss can learn discriminative representations in a single view well, it faces two key challenges when being extended to multiple views: (1) For multiple sources of information (i.e. RGB frames, motion vectors, and optical flows), simply applying an InfoNCE loss for each stream ignores the corresponding mutual information across these streams. (2) aims only to discriminate the augmented sample from all other instances, with the assumption that the target sample is semantically different from all the others. This is counter-intuitive, since different video clips may also have similar semantic content, i.e., hard positives are neglected by InfoNCE. We thus propose MVCGC to address these two problems.
Concretely, MVCGC decodes an RGB stream (RGB) r3 E RTxxxWx3 and a motion vector stream (MV) m4 E R x2 as two input views for each video clip r; E D (i 1,2, .N. MVCGC aims to learn two encoding functions f (-)) f(m) (-) for the two streams, respectively. To model the interaction between two streams, we extend the single-view InfoNCE loss in Eq. (1) by choosing positive and negative samples from the other view for each stream. Specifically, given the query of an RGB clip =("), the objective is compare its similarity among the features of motion vector clips, iee. emitting higher similarity between the positive sample zip .P E P. than with those of other negative instances _(m)) n Nr. The positive sets (i.e., p(") and p((m)) of each r; or m, are also expanded for the hard positive mining by adding the top-k similar clips:
p(r) ={5(m) ([(m))((mm; 0))l S,, ~ -..
p(m) = (r(r) (t( ry; S,, -.)
(3)
where S. -ilj e (top-k(2))) 2/(11 n top-k(2 (m)) 2) ((m)
)).j (m))
(1,2, ,N) denotes the index
set of positive samples, 2(t) f(r) (r:) and 2 (m) f(m) (m²). Our learning objective is thus to use the features in the two views as the cross guidance of each other:
Σ,(om exp(2 (xp)(2())))))))) (m)))))/// //) (m)
Livvae - llg
Σ Σn(m) exp( (2)
/T) + Σ (m) exple (r) (m) (T) 2p
Σ En (m) (mmeep(?? (m)
(m)) (1)
-log
En exp(2
(m)
/T) Σ (0) exp(> (m) 1 /T)
zp
5

---

The final loss is computed over where (mm) ppf ((m) EN (r) E p!!m! and an E e,
all training samples:
LMvCGG E Livcoc
(5)
Discussion Note that our MVCGC is different from other methods with discriminative InfoNCE loss in two aspects. First, differing from CMC [Tian et al. 2020] and IIC (Tao et al. 2020] which only mine single positive and regard the hard positives negative samples, MVCGC incorporates the learning from hard positives. Second, the co-training CoCLR (Han et al. 2020b] fails to learn the correspondence between two views (i.e., there is information exchange in the pre-learning stage), but our MVCGC leverages cross guidance to capture mutual information between two streams. We provide the algorithm in the supplementary material.
3.3 MVCGC Algorithm
The detailed procedure of our MVCGC is summarized in Algorithm
Algorithm Motion Vector based Cross Guidance Contrastive Learning (MVCGC)
Require:
Compressed video dataset D.
An RGB clip r; E RTTxxxW** x3 and motion vector clip m, E RTxxxx for x; E D
(i = 1,2, ................................ N).
Encoders f (.) (.) for RGB and motion vector streams, respectively.
The augmentation set t(:; 0). 0 ~ O
Temperature т, top-k hard positive mining parameter k.
1: for epoch = 1 to #epochs: do
2: Compute = 23 .2j 2,
(m) (r)
23 ViE NJ.J #
3: Compute S;;
4: Obtain ((r) ((m) Aorr) Nr(m)
5: Compute cross-entropy loss Lmvcaei
6: Update model parameters;
7: end for
8: return Optimized encoders (-) and (-)
Experiments
4.1 Datasets
In this paper, use UCF101 (Soomro et al. 2012 and Kinetics-400 (K400) Kay al. [017] for self-supervised pre-training. UCFI01 contains 13,320 videos with 101 action classes and has three standard training/test splits. While K400 is larger dataset consisting of 400 human action classes and has 230k/20k clips for training/validation videos in the first training set of UCF101 and the training split of K400 used without labels. Following the common practice [Han et all 2020b), we benchmark downstream evaluation tasks on the first test set of UCF101, and the test split I of HMDB51 [Kuehne al. 2011]. relatively small action dataset containing 6,766 videos with 51 categories.
respectively. In the pre-training stage, compressed
4.2 Implementation details
We follow the state-of-the-art method Haan al. 2020b] for adopting S3D Xie et al, 2018 architecture as the backbone feature extractor for all experiments and apply the momentum momentum-updatter updated history queue in MoCo [He et al.) 2020) for the framework of contrastive learning to leverage a larger number of negative samples. Datasets used contain videos with various codecs, which are directly decoded into RGB frames and motion vectors. We implement MVCGC based on "pyav", Pythonic binding for the FFmpeg libraries used for decoding RGB frames and motion vectors on-the-fly. fly. non-linear projection head is attached above each encoder during the pre-training stage and is removed for downstream evaluations.

---

Table 1: Results of processing speeds, storages and accuracies on UCFI01, The underline represents the second-best result.
Method
IIC
MemDPC
CoCLR
IMRNet Yu er el a 2021
MVCGC
Tao et al. / 2020
Han et al. 2020al
Han et al. 22220b
(ours)
Pre- pre-procssiin Time (s)
12.7
12.7
13.2
0.128
0.008
Inference Time (ms)
69.3
253.1
248.7
227.2
70.3
Total Time (s) I
12.8
13.0
13.4
0.355
0.078
Storage (GB)
38.0
38.0
35.4
6.1
1.9
Top-I Acc (9))
72.7
84.3
87.3
76.8
87.4
Our MVCGC proceeds in two stages: the initialization stage and the cross guidance stage. In the first stage, the encoder of each view (i.e., RGB or MV) is initialized with Eq. (1) independently. After initializatinn, these two encoders large number of features, we adopt a momentum momentum-updatte history queue as in MoCo [e et al. 2020 0.07.
cross coos-traiine together minimize the loss in Eq. (5). To cache
which is used in both two pre-training stages. For the pre- pre-training UCF101, temperature
momentum m = 0.999 and queue size 2048 are used, while queue size is set to 16384 on K400. When pre-training on UCF101, the initialization stage lasts 300 epochs for each stream, and we then continually train the cross guidance for another 200 epochs. On K400, we train 200 epochs for each stream in the initialization stage and 50 epochs for cross guidance contrastive learning. 100 and 500 epochs are used for linear and fully fine-tuning, respectively. We use the Adam optimizer with a le-4 learning rate and le-5 weight decay for pre-training and the SGD optimizer with le-1 learning rate and 1e-3 weight decay for fine-tumie validation loss plateaus. The tuning.nnggg............... The learning rate is decayed down by 1/10 twice when the in MVCGC is set as 5 according to the ablation study, All experiments are trained on 4 TiTan RTX GPUs, with batch size of 32 samples per GPU,
We make evaluation action classification task, two evaluation settings are used: (1) linear probing, where parameters of the learned encoder are frozen and only a single linear layer is fine-tuned; (2) fully fine-tuning, where the encoder is fine-tuned together. At the inference stage in either of these two settings, we follow the same procedure as in the previous work (Han el al. 2020b: for each video, we spatially apply ten-crops and temporally take clips with moving windows, and then average the predicted probabilities. In the video retrieval task, we adopt widely-used evaluation metries (Han el al. 2020b) Tao et al. 2020). i.e., we leverage the extracted features for nearest-neighben (NN) retrieval without fine-tuning. More details can be found in the supplementary material.
downstream tasks: action classification and video retrieval. In the
4.3 Efficiency Results
Table presents the efficiency results on UCF101 compared with four state-of-the-art self-supervised methods: IIC [Tao al. 2020]. MemDPC Han el al. 2020a), CoCLR (Han al. 2020b), and IMRNet (Yu el al.) 2021]. We use per-video processing time, storage spaces, and top- accuracies as evaluation metries under the fully fine-tuning setting. All methods are measured in exactly the same environment: Intel Xeon 51 18 CPUs and a Titan RTX GPU, The total processing time consists of two parts: data pre-processeeeee processccsing and inference time. In the pre-processing stage, IIC, MemDPC, and CoCLR decode RGB frames and calculate optical flows with the TV-L1 algorithm, while IMRNet re-encodes the raw compressed videos. We can observe that: (1) our MVCGC has the fastest speed and reduces the storage budgets by 95% and 70% compared with traditional methods and IMRNet, respectively. (2) For inference, the data is loaded from disks and is used for network forwarding. Although the time costs may differ because of the backbone architectures used by different methods, MVCGC still gets a comparable result to IIC and is faster than other three approaches. Nevertheless, the majority of the total process is dominated by data pre- pe-processing. where MVCGC significantly outperforms competitors. (3) Among fully fine-tuning methods, MVCGC outperforms those using either residuals or optical flows, showing the effectiveness of our proposed algorithm.
4.4 Ablation Study
We first show how our proposed MVCGC benefits the learned representations of motion vectors in Table 2, where the top-1 accuracies of action classification and retrieval performance are used as evaluation metrics since their evaluation fast. We can make the following four observations: (1) The upper part of the table shows the results of different views after initialization, where there is clearly
7

---

Table 2: Evaluation of motion vectors (MV) and optical flows (OF) on downstream action classifica- tion and retrieval on UCF101.
Method
Pre-training
Evaluation
Linear Probe
Retrieval
Init.
MV
MV
65.0
39.9
Init.
OF
OF
66.8
45.2
Init.
OF
OF
51.3
30.4
Init.
MVCOVAA
MVCAVAR
63.4
39.2
MVCGC
RGB+OF
OF
73.1 (+6.3)
60.6 (+15.4)
MVCGC
RGB+MV
MV
73.8 (+8.8)
60.8 (+20.9)
MV
MVMvVOC
OAvCCC
MV Init.
MV
FavcCC
pplying Lipstick
Shooting A row
Biking
On Balancing Beam
Figure 4: Visualization of the last feature maps from the models linear probed on UCF101. Each row shows the attention visualization of motion vectors before and after MVCGC pre-training and the final result of optical flows, respectively. Attention map is generated with 32-frames clip inputs and applied the middle frame in the video clips.
performance gap between the motion vector and the optical flow. To verify whether this gap originates from the lower resolution of the motion vector (i.e., same value in each macroblock), we also compare with the lower resolution of the optical flow (i.e., OF We find that by only reducing the resolution of the raw optical flow to a half (i.e., same value in each 2 x macroblock), it performs even worse than the motion vector. This comparison demonstrates that the motion vector, although having lower resolutions than the optical flow, is also able to well represent the action motions. (2) Since motion vectors may appear differently based on codec settings (i.e., different codecs and coding qualities), we also conduct experiment with the motion vectors extracted by the previous procedure (Yu et al. 2021; Wu al.) 2018 (i.e., M.CoVAA). where they are all decoded from P-frames. We find that our MVCGC is robust to motion vectors under different encoding configurations. (3) The motion vector view (or the optical flow view) can significantly benefit from the RGB view by our MVCGC. validating the effectiveness of our proposed cross guidance pre-training used in our MVCGC. (4) The performance of our MVCGC using motion vector RGB comparable to (and even slightly better than) that of our MVCGC using optical flow RGB. This finding is also supported by the visualization results shown in Figure
We then demonstrate the advantages of our proposed MVCGC over the closely-related method CoCLR [Han et al. (2200b] and also our choices for the hyper-parameters of MVCGC in Table 3 where we focus on the performance of the RGB network during evaluation. First, from the upper part of Table 3 we find that our MVCGC indeed helps the RGB network learn more discriminative representations than CoCLR, when the extra information of the motion vector view (or the optical flow view) leveraged. This shows the effectiveness of our cross guidance contrastive learning. Second, according to the lower part of Table 3 we select & = 5 as the optimal setting among l 1/5/10/20. Particularly, MVCGC with 'w/o init.' means that there no initialization stage before the cross guidance stage, which performs the worse. Moreover, MVCGC with "union' means that the index set S, generated by the union of positive candidates two streams, which performs worse than the intersection used in our MVCGC.
8

---

Table 3: Evaluation of different pre-training settings (i.e., algorithms and hyper-parameters. on downstream action classification and video retrieval UCF101. All results are evaluated using only the RGB view. The underline represents the second-best result.
Method
Pre-training
Evaluation
Linear Probe
Retrieval
Init.
RGB
RGB
52.3
33.1
CoCLR
RGB+OF
RGB
70.2(+17.9)
51.8(+18.7)
CoCLR
RGB+MV
RGB
67.4(+15.1)
53.8(+20.7)
MVCGC
RGB+OF
RGB
70.6(+18.3)
58.6(+25.5)
MVCGC (k=5)
RGB+MV
RGB
73.1(+20.8)
60.8(+27.7)
MVCGC (k=5, init.
RGB+MV
RGB
65.0
48.4
MVCGC (k-1)
RGB+MV
RGB
71.5
60.0
MVCGC (k=10)
RGB+MV
RGB
72.7
61.4
MVCGC
RGB+MV
RGB
71.5
59.7
MVCGC (k=5, union)
RGB+MV
RGB
71.3
57.7
Table 4: The full table of comparison with state-of-the-ar approaches linear probing (Lin.) and fully fine-tuning (Full.) on UCF101 and HMDB51 benchmarks. Top-1 accuracies are reported. "Dataset" denotes the pre-training dataset and "Duration" is the total length of videos. Methods marked with use multiple views for pre-training. "Test View(s)' denotes views used in evaluation,
i.e. RGB, motion vector(MV), optical flow(OF), residual(Res), Audio, and Text.
Method
Test View(s)
Arch.
Res.
Datasct(Duration)
UCF UCF HMDB HMDB
Lin. Full.
Lin.
Full.
VCP
Luo
2020
RGB
C3D
16*
UCF(Ld)
68.5
32.5
PRP
Yao ef di 2000
RGB
R(2+1)D-18
16*112
UCF(ld)
72.1
35.0
Pace
Wang el at.
2020
RGB
R(2+1)D-18
16*112
UCF(ld)
75.9
DSM
Wang
202
RGB
C3D
16*112
UCF(d))
70.3
40.5
CMC
Tall m.
2120
RGB
CaffeNet
25*256
UCF(Id)
59.1
26.7
CoCLR
Han ef al. 2020b
RGB
S3D
32*128
UCF(ld)
70,2
81.4
39.1
52.1
MVCGC (ourS)
RGB
S3D
32$128
UCF(1d)
73.1
82.0
41.1
58.4
IIC
Tao 2020
RGB+OF
R3D-18
16*112
UCF(1d)
72.7
36.8
McmDPC
Hum et dr 2020a
RGB+OF
R2D3D-34
40º128
UCF(ld)
84.3
CoCLR
Ham
RGB+OF
S3D
32*
UCF(l))
72.1
87.3
58.7
MVCGC (ours)
RGB+MV
S3D
32*128
UCF(ld)
77.2
87.4
41.0
59.7
CCL
Kong et al. 2020
RGB
R(2+1)D-18
8-112
K400X28d)
52.1
69.4
27.8
37.8
Hun df
2019
R2D3D-34
40*224
K400228)
75.7
Pace
Wang
20E0)
RGB
R(2+1)D-18
16*112
K400(284)
77.1
36.6
SpeedNer
Bonaam ol.
2020
RGB
S3D-G
16*224
K400(28d)
81.1
48.8
DSM
Wing et dl. 202
RGB
R3D-34
16*224
K400(28d)
78.2
VIE
Zhuann
RGB
SlowFast-18
16*112
K400/28d)
80.4
52.5
CoCLK
Han of di. 2020b
RGB
S3D
32*128
K400228)
74.5
87.9
46.1
54.6
MVCGC
(ours)
RGB
32*128
K400X280)
AVTS
Korbur
2018
RGB+Audio
R(++))--18
25*224
K400(28))
86.2
52.3
ELO""
Piergiovannt of 2020
RGB+Aoddoo=FF
R(2+1)D-50
32+224
YouTTuhe800M
(00.00MM(1.9 .9y)
93.8
64.5
67.4
XDC
Alwassel el
RGB+Audio
R(2+1)D-18
32*224
K400(28d)
86.8
GDT
Palnck
RGB+Audio
R(2+1)D-18
32*224
K400(284)
89.3
60.0
CBT
Sun cl o/. 2019
RGB+Text
S3D
16*112
K400(28d)
$4.0
79.5
29.5
44.6
MIL-NCE
Miech
2020
RGB+Text
32*200
Howto (55)
82.7
91.3
IMRNet
Yu
RGB-MV+Res
R3D-50
30+224
K400(28))
77.3
McmDPC
Hil el 2020a
RGB+0F
R2D3D-34
40*224
K400(28d)
54.1
86.1
30.5
54.5
CoCLR"
Ham al. 20206
RGB+OF
S3D
324128
K400(284)
77.8
90.6
52.4
62.9
MVCGC (ours)
RGB+MV
32#128
K400(28d)
78.0
4.5 Comparison with State-of-the-AAt
In this section, compare our MVCGC with the state-of-the-art self-supervised SSVRL approaches on the action classification and video retrieval tasks. For the action classification task, we provide the comparative results in Table by pre-training on UCF101 and Kinetics-400 and then evaluating with either linear probing or fully fine-tuning on UCF101 and HMDB51. We list recent approaches evaluated on the same benchmark, trying to compare with them as fairly as we can, although variations are unavoidable in terms of architecture, training data, and resolution. The "Test View(s)* column refers the view(s) used for test/evaluation, e.g., the result of MVCGC in RRMV views obtained by averaging the predictions from RGB and motion vector networks. We can observe that: (1)

---

Table 5: Comparison with state-of-the-ee art methods in nearest-neight neighbur video retrieval on UCF101 and HMDB51. Videos in the test set are used to retrieve the videos in the training set, and R@k is reported with & E {1, 5, 10, 20, 50). All compared methods pre-trained on UCF101 except SpeedNet pre-trained on larger Kinetics-400. Methods marked with use audio for pre-training.
UCF
HMDB
Method
R@l
R@5
e@ 10
R@
R@ 50
R@
R@s
R@
R@20
R 50
Retrieval with RGB only:
VCP [Luo el al. 2020
18.6
53.5
68.1
7.6
24.4
36.3
53.6
76.4
SpeedNet Benam ef dt. 2020
13.0
28.1
37.5
49.5
65.0
DSM
Warg 20211
16.8
33.4
43.4
54.6
70.7
8.2
25.9
38.1
52.0
75.0
PRP
Yao a). 2020
23.2
38.1
46.0
55.7
68.4
10.5
27.2
40.4
56.2
75.9
Mem
[aan
2020a
20.2
40.4
52.4
64.7
7.7
25.7
40.6
57.7
Pace
Wang et at. 2020
31.9
49.7
59.2
68.9
80.2
12.5
32.2
45.4
61.0
80.7
CCL
Kong cf al.
2020
32.7
42.5
50.8
61.2
68.9
CoCLK Han er al.
2020b
53.3
69.4
76.6
82.0
23.2
43.2
53.5
GDT
Patrick
2020
73.4
88.1
92.9
25.4
51.4
63.9
75.0
87.8
MVCGC (ours)
60.8
74.1
79.8
85.8
92.6
24.1
49.7
61.3
73.3
87.5
Retrieval with two views:
IIC Tao er al. 2020
42.4
60.9
69.2
77.1
86.5
19.7
42.9
57.1
70.6
85.9
CoCLR Han et al. 2020b
76.9
82.5
26.1
45.8
57.9
69.7
MVCGC (ours)
66.1
79.1
84.0
89.0
93.7
25.6
49.2
60.7
73.3
86.5
Our MVCGC outperforms most of latest methods under both linear probing and fully fine-tuning settings on both benchmarks.
(2) Our MVCGC using motion vectors from compressed videos clearly beats CMC (Tian et al.) 2020] and CoCLR [Han et al. (2220b] which exploit optical flows for pre-training. (3) Compared to IMRNet (Yu et al.) [021] that also utilizes compressed videos for self-supervised learning, our method achieves significantly better results, demonstrating both efficiency and effectiveness of MVCGC. (4) Our MVCGC outperforms its competitors under the pre-training Kinetics-400 and fine-tuning on UCF101/HMDB55 setting, indicating the robustness of MVCGC to process motion vectors that are obtained with different codecs. (5) It can be observed that we outperform some approaches that exploit the correspondence of visual information with text or audio. This is impressive since we only focus on visual representations and only use visual-related views (i.e., RGB, motion vectors, optical flows).
In Table we evaluate our MVCGC in video retrieval on both UCF101 and HMDB51 We can see that our MVCGC significantly outperforms all compared methods in terms of all five metrics UCF101 and four the five metrics on HMDB51, achieving new state-of state-of-theeart. Particularly, our obtained retrieval results by retrieval with only motion vectors latest competitor CoCLR [Han et al.) 2020b) which uses optical flows for retrieval, indicating the effectiveness of our MVCGC. Furthermore, obtain comparable results with methods using extra audio modalities Patrick el al. 2020). indicating the effectiveness of MVCGC
higher than those of the
Conclusion
We have proposed a novel self-supervised video representation learning method named Motion Vector based Cross Guidance Contrastive Learning (MVCGC). By introducing on-the-fly decoded motion vectors and cross guidance contrastive learning, our proposed MVCGC has for the first time leveraged contrastive loss in compressed video self supersupeeeieed learning. Extensive experiments carried out to validate the efficiency and effectiveness of MVCGC. Importantly, our approach achieves new state-of-the aftthe-rrt on two downstream tasks across different benchmarks.
Acknowledgments and Disclosure of Funding
This work was supported in part by National Natural Science Foundation of China (61976220 and 61832017), Beijing Outstanding Young Scientist Program (BIJWZYJHOI H0AHHHHH111100022000 2019100020099) and Open Project Program Foundation of Key Laboratory of Opto-Electrooics Information Processing, Chinese Academy of Sciences (OEIP-O-202006) Ping Luo was supported by the General Research Fund of Hong Kong No. No.27208720.
10

---

References
Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. In NeurlPS, 2020.
Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T. Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. Speednet: Learning the speediness in videos. In CVPR, pages 9922-9931, 2020.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In ICML, pages 10709-10719, 2020.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In ICCVW, pages 1483-1492, 2019.
Tengda Han, Weidi Xie, and Andrew Zisserman. Memory-augmented dense predictive coding for video representation learning. In ECCV, pages 312 329, 2020.
Tengda Han, Weidi Xie, and Andrew Zisserman. Self-supervised co-training for video representation learning. In NewrIPS, 2020.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9729-9738, 2020.
Yuqi Huo, Xiaoli Xu, Yao Lu, Yulei Niu, Mingyu Ding, Zhiwu Lu, Tao Xiang, and Ji-Rong Wen. Lightweight action recognition in compressed videos. In ECCVW, 2020.
Yuqi Huo, Mingyu Ding, Haoyu Lu, Ziyuan Huang, Minggian Tang, Zhiwu Lu, and Tao Xiang. Self-supervised video representation learning with constrained spatiotemporal jigsaw. In ICCAI, 2021.
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. arXiv preprint arXiv: 06950, 2017.
Dahun Kim, Donghyeon Cho, and In So Kweon. Self-supervised video representation learning with space-tima cubic puzzles. In AAAI, pages 8545 8552, 2019.
Quan Kong, Wenpeng Wei, Ziwei Deng, Tomoaki Yoshinaga, and Tomokazu Murakami. Cycle-contrast for self-supervised vidco representation learning. In NeurlPS, 2020.
Bruno Korbar, Du Tran, and Lorenzo Torresani, Cooperative learning of audio and video models from self- supervised synchronization. In NeurlPS, pages 7763-7774, 2018.
Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: A large video database for human motion recognition. In ICCV, pages 2556-2563, 2011.
Hsin-Ying Lee, Jia-Bin Huang. Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representation learning by sorting sequences. In ICCV, pages 667-676, 2017.
Dezhao Luo, Chang Liu, Yu Zhou, Dongbao Yang, Can Ma. Qixiang Ye, and Weiping Wang. Video cloze procedure for self-supervised spatio-temporal learning. In AAAI, 2020.
Antoine Micch, Jean-Bape bean-Baptiste Alayrac, Lueas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End- to-end learning of visual representations from uncurated instructional vidcos. In CVPR, pages 9879-9889, 2020.
Ishan Misra, C. Lawrence Zitnick, and Martial Hebert. Shuffic and learn: Unsupervised learning using temporal order verification. In ECCV, pages 527-544, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: /877 2018
Mandela Patrick, Yuki M. Asano, Ruth Fong, João F. Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi- modal self-supervision from generalized data transformations. arXiv preprint MXXi:2003300229,, 2020.
AJ Piergiovanni, Anelia Angelova, and Michael S. Ryoo. Evolving losses for unsupervised video representation learning. In CVPR, pages 133-142, 2020.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV 115(3):211 2015.
11

---

Zheng Shou, Xudong Lin, Yannis Kalantidis, Laura Sevilla-Lara, Marcus Rohrbach, Shih-Fu Chang, and Zhicheng Yan. Dme-net: Generating discriminative motion cues for fast compressed video action recognition. In CVPR. pages 1268-1277, 2019.
Thomas Sikora. The mpeg-4 video standard verification model. JEEE TCSVT, 7(1):19-1, 1997.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:/212.04022 2012.
Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (heve) standard. JEEE TCSVT, 22(12):1649-1668 2012.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations using contrastive bidirectional transformer. arXiv preprint arXiv: 1906.05743, 2019.
Li Tao, Xucting Wang, and Toshihiko Yamasaki. Self-supervised video representation learning using inter-intra contrastive framework. In ACMMM, pages 2193-2201, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, pages 776-794, 2020.
Jiangliu Wang, Jianbo Jiao, and Yun-Ho Liu. Self-supervised video representation learning by pace prediction. In ECCV, pages 504-521, 2020.
Jinpeng Wang, Yuting Gao, Ke Li, Xinyang Jiang, Xiaowei Guo, Rongrong Ji, and Xing Sun. Enhancing unsupervised video representation learning by decoupling the scene and the motion. In AAAI, 2021.
Donglai Wei, Joseph Lim, Andrew Zisserman, and William T. Freeman. Learning and using the arrow of time. In CVPR, pages 8052-8060, 2018.
Thomas Wiegand, Gary J. Sullivan, Gislc Bjontegaard, and Ajay Luthra. Overview of the h. 264/avc vidco coding standard. JEÉE TCSVT, 13(7):560- 576, 2003.
Chao-Yune Yuan Wu, Manzil Zaheer, Hexiang Hu, R. Manmatha, Alexander J. Smola, and Philipp Krähenbühl. Compressed video action recognition. In CVPR, pages 6026-6035, 2018.
Saining Xie, Chen Sun, Jonathan Huang. Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: accuracy trade-offs in video classification. In ECCV, pages 318 335, 2018.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In CVPR, pages 10334-10343, 2019.
Yuan Yao, Chang Liu, Dezhao Luo. Yu Zhou, and Qixiang Ye. Video playback rate perception for self- self-supervised teatio-temporal representation learning. In CVPR, pages
2020.
Youngjac Yu, Sangho Lec, Gunhee Kim, and Yale Song. Self-supervised learning of compressed vidco representations. In ICLR, 2021.
Bowen Zhang, Limin Wang, Zhe Wang. Yu Qiao, and Hanli Wang. Real-time action recognition with deeply transferred motion vector cans. TIP, 27(5):2326-22 2018.
Chengxu Zhuang, Tianwei She, Alex Andonian, Max Sobol Mark, and Daniel Yamins. Unsupervised learning from video with deep neural embeddings. In CVPR, pages 9563 9972,,97, 2020.
12


Reviews:
[{"review_id": "ce1r5E_38Y", "reviewer": "Reviewer_ZRPJ", "summary": "The paper introduces a self-supervised learning method, MVCGC, on unlabelled video streams based on the RGB frames and the motion vectors (thus \"two views\"). The availability of two views enables the expansion of the positive pairs during the info-NCE based training based on cross-view similarities in the feature space. The MVCGC is argued to be faster than previous approaches using some form of motion (motion vector or optical flow) because it uses the motion vectors readily available through video decoding. In experiments, MVCGC is trained on UCF and Kinetics datasets with the S3D architecture and is tested on UCF and HMDB benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n\nThe computational and storage efficiency for video methods are crucial topics to be studied in the future. An increasing amount of visual information is shared in video formats on the web, but there is relatively less work on processing videos in an efficient fashion. It is also great that the paper is pointing to the limitations of prior approaches that seemingly have not thought deeply about computational efficiency - e.g. redundant motion computation in IMRNet. The resulting reduction in computational time (Figure 1, Table 1) and codec-agnocity (L51) are great, though I doubt if the paper itself can be credited for these advances (see cons). \n\nThe paper also follows a commendable practice of showing the settings like available test-time data, architecture, resolution, and the training datasets in Tables 4. Oftentimes, such tables lack those details, making it practically impossible to discern performance improvements from the proposed method from other technological advances.\n\nThe experimental analysis in Table 3 looks great. Compared to the closes counterpart, CoCLR, MVCGC improves the performance greatly by tuning the representation with the proposed cross-view info-NCE (and the hard-positive mining scheme).\n\n## Cons\n\nExactly how does MVCGC improve efficiency? The MVCGC-led improvement in efficiency is because of the replacement of expensive data preprocessing steps, such as optical flow estimation and residual computation, in previous approaches (IIC, MemDPC, CoCLR, IMRNet). It is great that MVCGC has opted for motion vectors that seem to be readily available through the decoding, which is anyways needed to obtain the RGB frames. However, the paper does not go into full detail about how the decoding works and how fast it is to compute the motion vectors. The method seems to use the \"pyav\" library (L202), but the paper should elaborate much further on the decoding bit if it wants to claim the contribution here. For now, it is difficult to fully acknowledge its efficiency-related contribution here.\n\nRelated to this, the authors argue that MVCGC is efficient and effective and that it is the first to exploit contrastive loss in compressed video self-supervised learning (L59-61). However, for example, CoCLR has also used the info-NCE loss on RGB + flows, just like MVCGC. If the core contribution is on the \"compressed video\" bit, I am lost. MVCGC also operates on the compressed RGB + flows, though it uses a cheaper algorithm to compute the flows. It would be great if the authors can clearly state this contribution again and provide supporting evidence.\n\nIn fact, when I first heard the term \"compressed videos\" (title and many other places), I thought the inputs to the model were encoded byte streams. However, MVCGC still requires decoded frames and motion vector fields. It is a bit misleading for me to say that the model operates on \"compressed videos\".\n\nPlease move the following bit to the method section: \"Our MVCGC proceeds in two stages ... to minimize the loss in Eq. (5)\" (L206). Before reading this, I have been wondering why MVCGC would not exploit the single-view contrastive learning.\n\n\n## Conclusion\n\nThe paper achieves great improvements in representation qualities and computational efficiency. However, I am still hesitating to recommend an acceptance because it is unclear how much the paper could claim its contribution regarding the computation efficiency. Please provide the full technical details around the decoding step for me to evaluate the contribution more precisely.\n\n## Post rebuttal\n\nThank you for the response. I would also like to thank other reviewers for their expert reviews. I still believe the paper either lacks contribution or does not present its contributions clearly. I would recommend another peer-review round to enhance the impactfulness for this paper. My final recommendation is a rejection for this round.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not discussed the limitations and societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "VjmsqRECKcv", "reviewer": "Reviewer_iwkT", "summary": "The authors proposed to leverage cross guidance between decoded RGB frames and Motion Vectors from compressed codecs to learn an efficient and effective action recognition model in a self-supervised manner. Experimental results on two datasets demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The proposed idea is straightforward and the writing is easy to follow.\n2. The accuracy seems superior compared to previous compressed video action recognition models.\n3. It is interesting that the authors bring up the discussion about the actual decoding process of compressed video action recognition models.\n\nCons:\n1. The main concern is that the authors did not justify why the proposed method is strongly entangled with compressed videos. The proposed method is motivated by learning more effective representations from compressed videos. The proposed cross-guidance could be applied two jointly trained RGB feature extractors without using any cross- stream information. Without this kind of justification, it is not very convincing that the proposed method specifically resolves an issue in compressed video action recognition or especially leveraged certain property of compressed videos. Moreover, the experimental results are somewhat puzzling: considering that downsampled optical flow should be highly similar to motion vector sampled following CoViar, it is hard to come up with any possible reason that leads to the huge performance gap as shown in Table 2.\n2. The second concern is on the unfair comparison of efficiency with previous methods. The inefficiency of computing residue in  previous methods is really an implementation issue rather than a methodology issue. Moreover, the assumption on IP encoding actually holds in many real-world use cases, .e.g, streaming. Therefore, it is not perfectly fair to compare with previous methods directly counting these steps in terms of inefficient implementation or different use scenario. Moreover, it is not clear what the batchsize is when conducting this comparison. This could lead to completely different conclusion in terms of inference speed.\n3. In Table 4, IMR-Net actually uses I frames rather than RGB stream but the author still used RGB to represent IMR-Net's test videos, which actually contradicts with Fig 2 that says IMR-Net discards RGB stream. Although they are in both RGB format, I frames are much sparser compared to RGB stream and thus should be considered separately.\n\nAfter rebuttal: The reviewer thank the authors for reponse. However, the concerns are not well addressed. \nThe justification on the comparison with ensemble of two RGB models is not convincing. The features of two RGB models trained from different random initialization also has dense correspondence. The authors should show that the improvement didn't come from a simple ensemble of two RGB models. The response to the justification between MV and OF is also vague. Unless the authors could show major difference between MV and downsampled OF, it cannot explain the bad performance of OF+RGB.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to the suggested discussion as listed in the cons.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "MwJNJoUGek", "reviewer": "Reviewer_3btZ", "summary": "This paper presents a self-supervised video representation learning scheme based on the cross guidance between RGB and motion vectors. The paper is titled compressed video contrastive learning because the motion vectors are directly obtained from compressed video streams. By mining hard positives from the other view, the proposed MVCGC method uses contrastive learning and multi-instance InfoNCE loss to train a DNN model for video representation. The method is evaluated on UCF101, HMDB51, and K400 benchmark datasets using standard setting. It achieves similar results as CoCLR. ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is not hard to tell that this work is based on a previous NeurIPs'20 paper CoCLR, with only two differences: 1. The proposed MVCGC uses motion vectors extracted from compressed videos as the second view of a video clip while CoCLR uses unsupervised optical flows; 2. MVCGC tries to embed two views into the same latent space while CoCLR embeds RGB and optical flow into separate spaces. Other than these, MVCGC shares the contrastive learning approach  and the hard positive mining from the second view with CoCLR. Given the high similarity with CoCLR, this work is not sufficiently novel for publication in NeurIPs. \n\n- Although motion vectors are directly extracted from compressed videos, obtaining the RGB frames still needs a complete decoding. The only merit left is to avoid the additional computation of optical flows. But one could argue that moderate-quality optical flows are not very expensive to compute or store after pre-processing. \n- It is debatable whether RGB and motion should be embedded into the same latent space. It is not fair to say that \"CoCLR fails to learn the correspondence between two views\" (P5L184). The cross-view hard positive mining proposed in CoCLR (which is also adopted in MVCGC) builds a strong bond between two views although they are not embedded into the same latent space. In fact, the performance of MVCGC is only slightly better than that of CoCLR. \n\n---- after rebuttal ----\nThanks the reviewer for the response, which did clear up a few issues. However, the concern on limited novelty remains. Therefore, this reviewer will keep the original rating. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This reviewer did not see possible negative societal impact of the work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "81xzM_lRbz7", "reviewer": "Reviewer_UKWL", "summary": "The paper presents a method for self-supervised learning that leverages video bitstream data. The method includes a contrastive loss function and leverages pixel data from I-frames and motion data from P-frame motion vectors. The paper reports good results on a variety of experiments and with a faster execution time.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I'm very on the fence about this paper and I am open to upgrading my review provided sufficient feedback from the authors.\n\nOn the positive side, I note that this method appears to have good performance on the self-supervised tasks and the addition of a contrastive loss on bitstream data is novel if incremental. Beyond the contrastive loss however, I struggle to find additional novelty.\n\nThe paper seems quite fixated on speed in the sense that the method does not depend on the \"coviar method\" of computing error residuals, choosing to discard them and instead use only I-frame pixels and P-frame motion vectors. I did not find this argument convincing. Retrieval of true residuals from an encoded bitstream vs. computing them from motion frames is an implementation detail and it would be a system level task to correct this. Any \"real world\" use of coviar-like technologies would be re-implemented using a proper bitstream analysis library or would modify the codec directly to produce error residuals with no computation, a task which is time consuming but not overly difficult (see [1] for example). This same argument extends to coviar (and the follow on works) use of MPEG-4 part 2: this codec was chosen mainly for ease of implementation of research code and would be easily corrected in production to support modern codecs which would alleviate any re-encoding concerns. Nothing about coviar mandates MPEG-4 part 2. Finally, the paper indicates that residual streams are \"proved to only supply a marginal performance gain\" (ln103) without evidence of such proof. I disagree with this characterization as the residual data can provide additional information about scene elements that may be missing from I-frames (see [2, 3], if P-frame residuals were not important, multi-frame restoration would have no advantage).\n\nI'll also note that the closest prior work to this paper, IMRNet, uses a 3D resnet backbone whereas the proposed method uses an S3D backbone which may not be a fair comparison of the methods unless IMRNet was re-implemented with the matching backbone [2].\n\nFinally, the bolding on the tables (specifically Tab. 4, 5) was somewhat confusing as the proposed method is always shown in bold but is not always the best performing method.\n\nGiven the stated flaws and the marginal novelty of the method, I am inclined to reject the paper but I am open to discussion with the authors on all points.\n\n1. https://vqeg.github.io/software-tools/encoding/modified-avc-codec/\n2. Wang, Xintao, et al. \"Edvr: Video restoration with enhanced deformable convolutional networks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2019.\n3. Xue, Tianfan, et al. \"Video enhancement with task-oriented flow.\" International Journal of Computer Vision 127.8 (2019): 1106-1125.\n4. Musgrave, Kevin, Serge Belongie, and Ser-Nam Lim. \"A metric learning reality check.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\nI have read the authors' rebuttals as well as the other reviews.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers acknowledged that this paper tackles an important problem of making video pipelines compute/storage-efficient. However, they raised concerns over the limited novelty, especially compared to CoCLR and IMRNet. They also commented that the arguments around codecs and decoding mechanisms used in prior approaches are system-level design choices rather than a scientific challenge, and questioned whether replacing coviar-like video processing with the `pyav` python library can be considered a novel technical contribution. The rebuttal addressed some of the questions but unfortunately did not perfectly clear up the major concerns. Based on this, I am recommending a rejection at this time, but would invite the authors to improve their paper based on the reviewers' suggestions.
Decision: Accept (Poster)