Compressed Video Contrastive Learning
Authors: Yuqi Huo
Mingyu Ding
Haoyu Lu
Nanyi Fei
Zhiwu Lu
Ji-Rong Wen
Ping Luo
Venue: NeurIPS 2021 Poster (2021)
Keywords: compressed video
self-supervised learning
motion vector
contrastive learning

Abstract: This work concerns self-supervised video representation learning (SSVRL), one topic that has received much attention recently. Since videos are storage-intensive and contain a rich source of visual content, models designed for SSVRL are expected to be storage- and computation-efficient, as well as effective. However, most existing methods only focus on one of the two objectives, failing to consider both at the same time. In this work, for the first time, the seemingly contradictory goals are simultaneously achieved by exploiting compressed videos and capturing mutual information between two input streams. Specifically, a novel Motion Vector based Cross Guidance Contrastive learning approach (MVCGC) is proposed. For storage and computation efficiency, we choose to directly decode RGB frames and motion vectors (that resemble low-resolution optical flows) from compressed videos on-the-fly. To enhance the representation ability of the motion vectors, hence the effectiveness of our method, we design a cross guidance contrastive learning algorithm based on multi-instance InfoNCE loss, where motion vectors can take supervision signals from RGB frames and vice versa. Comprehensive experiments on two downstream tasks show that our MVCGC yields new state-of-the-art while being significantly more efficient than its competitors.

Compressed Video Contrastive Learning
Yugi Huo 2,3 Mingyu Ding Haoyu Lu
Nanyi Fei2.3
22 Ji-Rong Wen Ping Luo Gaalling School of Artificial Intelligence, Remmin University of China. Beijing, China Beijing Key Laboratory of Big Data Management and Analysis Methods
Zhiwu Lu L
3 School of Information, Renmin University of China, Beijing, China "The University of Hong Kong, Pokfulam, Hong Kong, China
(bnhony, luzhiwu] Gruc. edu. cn Equal contribution Corresponding author
Abstract
This work concerns self-supervise video representation learning (SSVRL), one topic that has received much attention recently. Since videos are dtorage-intensive and contain rich source of visual content, models designed for SSVRL are expected to be storage- and computation-effiiinn most existing methods only focus one of the two objectives, failing to consider both at the same time. In this work, for the first time, the seemingly contradictory goals simultaneously achieved by exploiting compressed videos and capturing mutual information between two input streams. Specifically, novel Motion Vector based Cross Guidance Contrastive learning approach (MVCGC) is proposed. For storage and computation efficiency, we choose to directly decode RGB frames and motion vectors (that resemble low-resolution optical flows) from compressed videos on-the-fly fly. To enhance the representation ability of the motion vectors, hence the effectiveness of method, we design cross guidance contrastive learning algorithm based on multi-instance InfoNCE loss, where motion vectors can take supervision signals from RGB frames and vice versa. Comprehensive experiments on two downstream tasks show that our MVCGC yields new state-of-the-arr while being significantly more efficient than its competitors.
well effective. However,
Introduction
Recent self-supervised image representation learning approaches [He et al.) 2020 Chen et al. 2020
have been reported to outperform supervised ones a wide range of downstream tasks by (I) leveraging a large amount of unlabeled data available online for pre-training, and (2) designing
powerful algorithm to discriminate unlabeled samples with different semantic meanings. However, the situation for self-supervised video representation learning (SSVRL) is somewhat different: self-supervised methods still perform worse than supervised ones, since videos extremely storage- intensive (scalability thus becomes a serious issue) and have richer source visual content (SSVRL thus becomes very difficult). Therefore, designing storage-and-oe aandd-oomputtioon-fffiiieee effective models for SSVRL remains a challenging problem that has not been well studied.
well as
Existing state-of- offte-of-the-rrr aht methods [Han et al., 2020b] Tao et al. 2020 Huo et al. [222 mainly focus on designing effective algorithms without considering the storage and computation costs during large-scale video self-supervised training. Particularly, in order to leverage both appearance and temporal information in the video, they exploit the optical flow as an extra view to complement RGB stream. However, this procedure is both storage- and cemputatioo-intensin because of storing the decoded frames and computing optical flows, respectively (e.g., it costs more than 100GB for
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

88
MVCGC (ours)
COCLR
85
Top-1 Accuracy (%)
MemDPC
82
79
IMRNet
76
73
10"2
10
10a
Processing Time Per-video (second)
Processing time per-video (i.e., data pre-processing and inference) and the accuracies for different methods UCF101. Each method is represented by circle, whose size represents the storage size occupied by the test data. MVCGC achieves the highest accuracy under a significantly reduced storage budget compared to its counterparts. More details can be found in Table !
Figure I:
frame storage and a number of days for computing optical flows (Han et al. 2020b on the UCF101 dataset (Soomro ef al. 2012].. This clearly hinders large-scale video self-supervised training.
Considering that videos are already stored in compressed formats to reduce storage requirements, natural choice is to resort to on-the-fly decoding, i.e., decoding frames from compressed videos during storage. Moreover, since motion vectors encoded in
training/inference without introducing any compressed videos resemble low-resolution optical flows in describing local motions, they can be used avoid the costly optical flow computation. Consequently, compressed videos inherently contain both static and motion information that is suitable for video representation learning. Recently, pretext task based method called IMRNet (Yu et al.) [02T] leverages the standard process in CoVVAR [Wu al. 2018 to decode compressed videos, but inevitably encounters the inefficiency and ineffectiveness problems: (1) The outdated CoViAR is not truly on-the-fly, i.e., it re-encodes videos and stores them before decoding, which is storage and computation inefficient. (2) Motion vectors are less discriminative and thus weaker than optical flows, resulting in sub-optimal performance compared with state-of-the-art methods using optical flows, as shown in Figure I As result, exploiting the compressed videos efficiently, followed by learning discriminative representations, is still challenging.
Motivated by the above observations, we propose Motion Vector based Cross Guidance Contrastive learning approach (MVCGC), which has several appealing benefits: (1) It able decode RGB frames and motion vectors from compressed videos with various codecs. The storage and computation budgets are alleviated since videos can be decoded without re-encoding. (2) Our MVCGC can learn discriminative features from both RGB and motion vector streams. This is achieved by designing cross guidance contrastive learning algorithm based on multi intttiiittancee InfoNCE loss: each sample pair is constructed by one RGB clip and one motion vector clip, and multiple positive samples can be mined by calculating the similarity two views. Therefore, the two views can take supervision signals from each other to improve the representation quality of both streams, especially for the motion vector stream (whose learned representations are even comparable to optical flow features).
Our main contributions are three-fold: (1) We propose an efficient and effective framework called knowledge. we are the first to exploit contrastive loss in compressed video self-supervised learning. (2) We design novel contrastive learning algorithm capture mutual information between RGB frames and motion vectors from compressed videos. Importantly, we even make the learned features of motion vectors as representative as those optical flows. (3) Extensive experiments by applying the learned video representations two downstream tasks (i.e., action recognition and action retrieval) across different benchmarks demonstrate that our MVCGC achieves ofaee- performance while being more efficient than its counterparts (see Figure
MVCGC that can learn representations from compressed videos directly. To the best

---

2 Related Work
Self-Supervised Video Representation Learning. Existing self-supervised methods for video
representation learning can be divided into two categories: (1) One common way to learn good
representations is to define and solve pretext tasks, such as ordering frames Misra et al. [016 Lee er
al.| [017;; Xu er al. [2099] Luo et al. [020], predicting the speed of videos Yao et al. [2220] Benaim
et al. 2020; Wang et al.) 2020]. and solving space-time cubic puzzles [Kim et al. 2019]. (2) Another
more recent way focuses instance discrimination, which leverages distance- based contrastive loss
and distinguishes the positive samples from a group of negative ones [Oord el al.) 2018 He et al.
2020 Chen et al. 2020]. Recently, many efforts based contrastive learning have demonstrated
promising results. They either focus only on RGB frames (Han et al. 2019) Zhuang et al. [020]]
Wang et al.) 2020 Kong al. 2020 Wang et al. 2021] or introduce an additional optical flow view
in order to achieve state-of-the-arr accuracy [Han et al. 2220aa Tian et al.) 2020) Tao et al. 2020
Han et al. 2020b]. Considering the heavy computational and stta-st burdens for computing the
optical flow, our proposed MVCGC replaces it by introducing the motion vector from the compressed
video as a new view, which significantly cuts down the computational and storage costs. Moreover,
MVCGC is different from the previous approaches [Han et al. [2020a; Tian er al. [2020; Tao et al.
2020, Han ef al.) [220bb] in how contrastive loss and sample pairs are defined. CMC [Tian al. 2020) and TIC TTo et | 2020 use the single-instance InfoNCE loss, resulting in neglect of hard positives.
MemDPC |Han et al.) [020a] and COCLR (Han et al.) 2020b] construct sample pairs of contrastive loss in RGB frames and optical flows separately, ignoring the correspondence between different views. In contrast, our MVCGC constructs sample pairs between different views of different clips (i.., RGB frames of one clip and motion vectors of another clip from the same video) and applies the multi-instance InfoNCE loss. We use two views as cross guidance to train the representations of two streams simultancously and learn mutual information between RGB frames and motion vectors.
Compressed Video Representation Learning. Earlier approaches have explored the compressed video for supervised recognition (Zhang et al. [018]] Wu et al. 2018:; Shou et al.) 2019], in which large-scale supervised datasets (such as ImageNet [Russakovsky et al. 2015]) are used for pre- training. In contrast, we focus on self-supervised learning, i.e., only unlabeled data is used in the pre-training stage. After the first exploration of using compressed video to learn "the arrow of time"(Wes et al. 2018). a related work, IMRNet (Yu et al. 2021]. also explores compressed video self-supervised learning besides supervised learning. Our proposed MVCGC is fundamentally different from IMRNet in three key aspects: (1) Instead of focusing on designing pretext tasks as in IMRNet, our MVCGC leverages contrastive loss. (2) With the outdated MPEG-4 Part 2 codec [Huo et al. 2020 IMRNet has to re-encode and store videos for data loading, whereas our MVCGC can decode compressed videos on-the-fly fly and save storage spaces. (3) Besides the decoded RGB and motion vector streams, IMRNet leverages an additional residual stream computed from the two streams, which has already been proved only supply a marginal performance gain. In comparison, our MVCGC utilizes the first two streams and thus alleviates computation burdens.
3 Methodology
This work aims to learn robust video representations efficiently from compressed videos by designing a contrastive learning algorithm. this section, we first review the basics of compressed video and discuss the advantages of our decoding process compared to existing counterparts. Afterward, we detail our MVCGC on how learns effective feature representations and captures mutual information between RGB frames and motion vectors from compressed videos.
3.1 Basics of Compressed Video
Considering the redundancy in consecutive frames, codecs are proposed to compress videos for efficient storage, i.e., compress one frame by reusing contents from another frame (termed reference frame) and only store the change. According to the choice of reference frame, frames are categorized into three types: I-frame (Intra-coded frame), P-frame (Predictive frame), and B-frame (Bi-predictive frame). I-frame has no reference frame and is directly stored in image format, while P- and B-frame take reference frames forwardly and bi-directionally, respectively, and store the change in the format of the motion vector and residual: A codec first divides frame into macroblocks of size such
3

---

Comp Video
Comp. Video
Re enode
Comp, Video
Comp. Video
Decdde
Discarded
Decode
Compute
Deooe
Compute
RGB
(a)
(b)
(c)
Figure 2: Usage of compressed videos (Comp. Video) for SSVRL: (a) Traditional methods decode videos off-the-fly; (b) IMRNet re-encodes videos. followed by the additional residuals computation with RGB frames being discarded afterward; (c) MVCGC decodes RGB frames and motion vectors on-the-fly. "RGB", "MV", "OF", and "Res" denote RGB frames, motion vectors, optical flows, and residuals, respectively. Procedures colored with red computation expensive, and their backgrounds which are colored with orange or blue, indicate the happening place of them, Le., videos in the orange background are stored the disk while those the blue are decoded in cache memory.
as 16x16 and then searches the most similar image patch in the reference frame for each of these macroblocks. Lastly, the motion vector is represented by the displacements between macroblocks and the searched image patches, and the residual is computed as the difference between the target frame and its reference frame warped by the motion vector.
Discussion As shown in Figure 2 existing SSVRL methods can be categorized into three groups according to how they exploit videos for SSVRL. (1) Traditional methods off-the-fly decode all RGB frames from compressed videos and compute optical flows with RGB frames, which is storage- and computation-intessit leveraging the process in CoViAR [Wu et al. 2018). which only supports the outdated MPEG-4 Part codec (Sikoral 1997] and lacks generalizability In this way. videos with modern codecs (e.g., H.264/AVC [Wiegand et al.) 2003] and HEVC (Sullivan et al. 2012)) in current application scenarios need to be re-encoded and re-stored, downgrading the computational efficiency. IMRNet residuals of P-frames as third view: since residuals post-computed by two decoded views, iee,, RGB frames and motion vectors, which introduces extra computation requirements. Moreover, in IMRNet, RGB frames decoded only for computing residuals, while being discarded afterward, hindering the data usage effectiveness. (3) Our proposed MVCGC can decode RGB frames and motion vectors directly from various video codecs without the aforementioned costly off-the-fly frame extracting, video re-encoding, or residual computing, and is thus high high-efficienn efficiiefcyy and application -iexiiiiiii..iiiit
(2) The latest IMRNet (Yu et al. [02T] decodes compressed videos by
not be extracted in practice, it is
3.2 Motion Vector based Cross Guidance Contrastive Learning
The goal of SSVRL is to train an encoder that can embed video samples effectively for various downstream tasks (e.g., action recognition and retrieval). Towards this end, method is based on contrastive learning with InfoNCE loss [Oord et al., 2018). as shown in Figure Next we start by reviewing base single-vies view model with InfoNCE loss, and then extend the base model using cross guidance contrastive learning as our MVCGC.
Base Single- Vingle-View View Model with InfoNCE Loss Formally, let D 121, , ZNN } denotes dataset of N compressed video samples, where each clip z; (i 1,2,.... ,N) consists of T frames. In the single-view scenario, these clips are in the same view, i.e., D is dataset of only RGB clips or motion vectors. The objective of learning an effective encoding function f(-) for the single stream is thus achieved by discriminating one positive sample of each video clip from its negatives with InfoNCE loss [Oord et al.) [2018].
Let ((: 0)) be an augmentation function applied to D, where 0 is sampled from set of transformations O. For each clip x, D (i- = 1.2...., N), we denote the encoded feature vector of n; as 2; = f(x)). the positive sample set as P. [f(t(z;:)) 0 θ), ), and the negative set as N₁ ((((z)(0)))

---

N
Cross Guidance
Hard Postiive Mining
Lavenn
86-81
PN
K
Tops Striletty
Trpa Smrfantt
Guidanoe
N
NN
NNN
(a]
(b)
Figure 3: Overview of the proposed MVCGC architecture. (a) From raw RGB clip and its corresponding motion vector clip, MVCGC computes the anchor and the positive features, while other samples in the dataset are used to construct the negative set. (b) The complete design of the hard positive mining stage.
i, 0 O). where [P.] and |
N
The InfoNCE loss is then defined as:
exp (24 2p/T)
LIntaNCB
Σ log
cxp (z: zp/T) + Ex. ENG exp (2; =n/T)²
where 2p E P; is the only positive sample of Z4, 2, 2p (or 2; =n, refers to the dot product between two vectors, and 7 > 0 is temperature parameter. This instance discrimination loss forces the base model to learn a higher similarity score between a given sample 2; and its augmented/target sample Zp while a lower one between 2; and other instances 2n E N,
Cross Guidance Contrastive Learning with Multiple Views While the InfoNCE loss can learn discriminative representations in a single view well, it faces two key challenges when being extended to multiple views: (1) For multiple sources of information (i.e. RGB frames, motion vectors, and optical flows), simply applying an InfoNCE loss for each stream ignores the corresponding mutual information across these streams. (2) aims only to discriminate the augmented sample from all other instances, with the assumption that the target sample is semantically different from all the others. This is counter-intuitive, since different video clips may also have similar semantic content, i.e., hard positives are neglected by InfoNCE. We thus propose MVCGC to address these two problems.
Concretely, MVCGC decodes an RGB stream (RGB) r3 E RTxxxWx3 and a motion vector stream (MV) m4 E R x2 as two input views for each video clip r; E D (i 1,2, .N. MVCGC aims to learn two encoding functions f (-)) f(m) (-) for the two streams, respectively. To model the interaction between two streams, we extend the single-view InfoNCE loss in Eq. (1) by choosing positive and negative samples from the other view for each stream. Specifically, given the query of an RGB clip =("), the objective is compare its similarity among the features of motion vector clips, iee. emitting higher similarity between the positive sample zip .P E P. than with those of other negative instances _(m)) n Nr. The positive sets (i.e., p(") and p((m)) of each r; or m, are also expanded for the hard positive mining by adding the top-k similar clips:
p(r) ={5(m) ([(m))((mm; 0))l S,, ~ -..
p(m) = (r(r) (t( ry; S,, -.)
(3)
where S. -ilj e (top-k(2))) 2/(11 n top-k(2 (m)) 2) ((m)
)).j (m))
(1,2, ,N) denotes the index
set of positive samples, 2(t) f(r) (r:) and 2 (m) f(m) (m²). Our learning objective is thus to use the features in the two views as the cross guidance of each other:
Σ,(om exp(2 (xp)(2())))))))) (m)))))/// //) (m)
Livvae - llg
Σ Σn(m) exp( (2)
/T) + Σ (m) exple (r) (m) (T) 2p
Σ En (m) (mmeep(?? (m)
(m)) (1)
-log
E, exp(2
(m)
/T) Σ (0) exp(> (m) 1 /T)
zp
5

---

The final loss is computed over where (mm) ppfr) ((m) EN (r) E p!!m! and an E e,
all training samples:
LMvCGG E Livcoc
(5)
Discussion Note that our MVCGC is different from other methods with discriminative InfoNCE loss in two aspects. First, differing from CMC [Tian et al. 2020] and IIC (Tao et al. 2020] which only mine single positive and regard the hard positives negative samples, MVCGC incorporates the learning from hard positives. Second, the co-training CoCLR (Han et al. 2020b] fails to learn the correspondence between two views (i.e., there is information exchange in the pre-learning stage), but our MVCGC leverages cross guidance to capture mutual information between two streams. We provide the algorithm in the supplementary material.
3.3 MVCGC Algorithm
The detailed procedure of our MVCGC is summarized in Algorithm
Algorithm Motion Vector based Cross Guidance Contrastive Learning (MVCGC)
Require:
Compressed video dataset D.
An RGB clip r; E RTTxxxW** x3 and motion vector clip m, E RTxxxx for x; E D
(i = 1,2, ................................ N).
Encoders f (.) (.) for RGB and motion vector streams, respectively.
The augmentation set t(:; 0). 0 ~ O
Temperature т, top-k hard positive mining parameter k.
1: for epoch = 1 to #epochs: do
2: Compute = 23 .2j 2,
(m) (r)
23 ViE NJ.J #
3: Compute S;;
4: Obtain ((r) ((m) Aorr) Nr(m)
5: Compute cross-entropy loss Lmvcaei
6: Update model parameters;
7: end for
8: return Optimized encoders (-) and (-)
Experiments
4.1 Datasets
In this paper, use UCF101 (Soomro et al. 2012 and Kinetics-400 (K400) Kay al. [017] for self-supervised pre-training. UCFI01 contains 13,320 videos with 101 action classes and has three standard training/test splits. While K400 is larger dataset consisting of 400 human action classes and has 230k/20k clips for training/validation videos in the first training set of UCF101 and the training split of K400 used without labels. Following the common practice [Han et all 2020b), we benchmark downstream evaluation tasks on the first test set of UCF101, and the test split I of HMDB51 [Kuehne al. 2011]. relatively small action dataset containing 6,766 videos with 51 categories.
respectively. In the pre-training stage, compressed
4.2 Implementation details
We follow the state-of-the-art method Haan al. 2020b] for adopting S3D Xie et al, 2018 architecture as the backbone feature extractor for all experiments and apply the momentum momentum-updatter updated history queue in MoCo [He et al. 2020) for the framework of contrastive learning to leverage a larger number of negative samples. Datasets used contain videos with various codecs, which are directly decoded into RGB frames and motion vectors. We implement MVCGC based on "pyav", Pythonic binding for the FFmpeg libraries used for decoding RGB frames and motion vectors on-the-fly. fly. non-linear projection head is attached above each encoder during the pre-training stage and is removed for downstream evaluations.

---

Table 1: Results of processing speeds, storages and accuracies on UCFI01, The underline represents the second-best result.
Method
IIC
MemDPC
CoCLR
IMRNet Yu er el a 2021
MVCGC
Tao et al. / 2020
Han et al. 2020al
Han et al. 22220b
(ours)
Pre- pre-procssiin Time (s)
12.7
12.7
13.2
0.128
0.008
Inference Time (ms)
69.3
253.1
248.7
227.2
70.3
Total Time (s) I
12.8
13.0
13.4
0.355
0.078
Storage (GB)
38.0
38.0
35.4
6.1
1.9
Top-I Acc (9))
72.7
84.3
87.3
76.8
87.4
Our MVCGC proceeds in two stages: the initialization stage and the cross guidance stage. In the first stage, the encoder of each view (i.e., RGB or MV) is initialized with Eq. (1) independently. After initializatinn, these two encoders large number of features, we adopt a momentum momentum-updatee history queue as in MoCo [ee et al. 2020 0.07.
cross coos-traiine together minimize the loss in Eq. (5). To cache
which is used in both two pre-training stages. For the pre- pre-training UCF101, temperature
momentum m = 0.999 and queue size 2048 are used, while queue size is set to 16384 on K400. When pre-training on UCF101, the initialization stage lasts 300 epochs for each stream, and we then continually train the cross guidance for another 200 epochs. On K400, we train 200 epochs for each stream in the initialization stage and 50 epochs for cross guidance contrastive learning. 100 and 500 epochs are used for linear and fully fine-tuning, respectively. We use the Adam optimizer with a le-4 learning rate and le-5 weight decay for pre-training and the SGD optimizer with le-1 learning rate and 1e-3 weight decay for fine-tumie validation loss plateaus. The tuning.nnggg............... The learning rate is decayed down by 1/10 twice when the in MVCGC is set as 5 according to the ablation study, All experiments are trained on 4 TiTan RTX GPUs, with batch size of 32 samples per GPU,
We make evaluation action classification task, two evaluation settings are used: (1) linear probing, where parameters of the learned encoder are frozen and only a single linear layer is fine-tuned; (2) fully fine-tuning, where the encoder is fine-tuned together. At the inference stage in either of these two settings, we follow the same procedure as in the previous work (Han el al. 2020b: for each video, we spatially apply ten-crops and temporally take clips with moving windows, and then average the predicted probabilities. In the video retrieval task, we adopt widely-used evaluation metries (Han el al. 2020b) Tao et al. 2020). i.e., we leverage the extracted features for nearest-neighben (NN) retrieval without fine-tuning. More details can be found in the supplementary material.
downstream tasks: action classification and video retrieval. In the
4.3 Efficiency Results
Table presents the efficiency results on UCF101 compared with four state-of-the-art self-supervised methods: IIC [Tao al. 2020]. MemDPC Han el al. 2020a), CoCLR (Han al. 2020b), and IMRNet (Yu el al.) 2021]. We use per-video processing time, storage spaces, and top- accuracies as evaluation metries under the fully fine-tuning setting. All methods are measured in exactly the same environment: Intel Xeon 51 18 CPUs and a Titan RTX GPU, The total processing time consists of two parts: data pre-proces processccsing and inference time. In the pre-processing stage, IIC, MemDPC, and CoCLR decode RGB frames and calculate optical flows with the TV-L1 algorithm, while IMRNet re-encodes the raw compressed videos. We can observe that: (1) our MVCGC has the fastest speed and reduces the storage budgets by 95% and 70% compared with traditional methods and IMRNet, respectively. (2) For inference, the data is loaded from disks and is used for network forwarding. Although the time costs may differ because of the backbone architectures used by different methods, MVCGC still gets a comparable result to IIC and is faster than other three approaches. Nevertheless, the majority of the total process is dominated by data pre- pe-processingg where MVCGC significantly outperforms competitors. (3) Among fully fine-tuning methods, MVCGC outperforms those using either residuals or optical flows, showing the effectiveness of our proposed algorithm.
4.4 Ablation Study
We first show how our proposed MVCGC benefits the learned representations of motion vectors in Table 2, where the top-1 accuracies of action classification and retrieval performance are used as evaluation metrics since their evaluation fast. We can make the following four observations: (1) The upper part of the table shows the results of different views after initialization, where there is clearly
7

---

Table 2: Evaluation of motion vectors (MV) and optical flows (OF) on downstream action classifica- tion and retrieval on UCF101.
Method
Pre-training
Evaluation
Linear Probe
Retrieval
Init.
MV
MV
65.0
39.9
Init.
OF
OF
66.8
45.2
Init.
OF
OF
51.3
30.4
Init.
MVCOVAA
MVCAVAR
63.4
39.2
MVCGC
RGB+OF
OF
73.1 (+6.3)
60.6 (+15.4)
MVCGC
RGB+MV
MV
73.8 (+8.8)
60.8 (+20.9)
MV
MVMvVOC
OAvCCC
MV Init.
MV
Favcco
pplying Lipstick
Shooting A row
Biking
On Balancing Beam
Figure 4: Visualization of the last feature maps from the models linear probed on UCF101. Each row shows the attention visualization of motion vectors before and after MVCGC pre-training and the final result of optical flows, respectively. Attention map is generated with 32-frames clip inputs and applied the middle frame in the video clips.
performance gap between the motion vector and the optical flow. To verify whether this gap originates from the lower resolution of the motion vector (i.e., same value in each macroblock), we also compare with the lower resolution of the optical flow (i.e., OF We find that by only reducing the resolution of the raw optical flow to a half (i.e., same value in each 2 x macroblock), it performs even worse than the motion vector. This comparison demonstrates that the motion vector, although having lower resolutions than the optical flow, is also able to well represent the action motions. (2) Since motion vectors may appear differently based on codec settings (i.e., different codecs and coding qualities), we also conduct experiment with the motion vectors extracted by the previous procedure (Yu et al. 2021; Wu al.) 2018 (i.e., M.CoVAA). where they are all decoded from P-frames. We find that our MVCGC is robust to motion vectors under different encoding configurations. (3) The motion vector view (or the optical flow view) can significantly benefit from the RGB view by our MVCGC. validating the effectiveness of our proposed cross guidance pre-training used in our MVCGC. (4) The performance of our MVCGC using motion vector RGB comparable to (and even slightly better than) that of our MVCGC using optical flow RGB. This finding is also supported by the visualization results shown in Figure
We then demonstrate the advantages of our proposed MVCGC over the closely-related method CoCLR [Han et al. (2200b] and also our choices for the hyper-parameters of MVCGC in Table 3 where we focus on the performance of the RGB network during evaluation. First, from the upper part of Table 3 we find that our MVCGC indeed helps the RGB network learn more discriminative representations than CoCLR, when the extra information of the motion vector view (or the optical flow view) leveraged. This shows the effectiveness of our cross guidance contrastive learning. Second, according to the lower part of Table 3 we select & = 5 as the optimal setting among l 1/5/10/20. Particularly, MVCGC with 'w/o init.' means that there no initialization stage before the cross guidance stage, which performs the worse. Moreover, MVCGC with "union' means that the index set S, generated by the union of positive candidates two streams, which performs worse than the intersection used in our MVCGC.
8

---

Table 3: Evaluation of different pre-training settings (i.e., algorithms and hyper-parameters. on downstream action classification and video retrieval UCF101. All results are evaluated using only the RGB view. The underline represents the second-best result.
Method
Pre-training
Evaluation
Linear Probe
Retrieval
Init.
RGB
RGB
52.3
33.1
CoCLR
RGB+OF
RGB
70.2(+17.9)
51.8(+18.7)
CoCLR
RGB+MV
RGB
67.4(+15.1)
53.8(+20.7)
MVCGC
RGB+OF
RGB
70.6(+18.3)
58.6(+25.5)
MVCGC (k=5)
RGB+MV
RGB
73.1(+20.8)
60.8(+27.7)
MVCGC (k=5, init.
RGB+MV
RGB
65.0
48.4
MVCGC (k-1)
RGB+MV
RGB
71.5
60.0
MVCGC (k=10)
RGB+MV
RGB
72.7
61.4
MVCGC
RGB+MV
RGB
71.5
59.7
MVCGC (k=5, union)
RGB+MV
RGB
71.3
57.7
Table 4: The full table of comparison with state-of-the-ar approaches linear probing (Lin.) and fully fine-tuning (Full.) on UCF101 and HMDB51 benchmarks. Top-1 accuracies are reported. "Dataset" denotes the pre-training dataset and "Duration" is the total length of videos. Methods marked with use multiple views for pre-training. "Test View(s)' denotes views used in evaluation,
i.e. RGB, motion vector(MV), optical flow(OF), residual(Res), Audio, and Text.
Method
Test View(s)
Arch.
Res.
Datasct(Duration)
UCF UCF HMDB HMDB
Lin. Full.
Lin.
Full.
VCP
Luo
2020
RGB
C3D
16*
UCF(Ld)
68.5
32.5
PRP
Yao ef di 2000
RGB
R(2+1)D-18
16*112
UCF(ld)
72.1
35.0
Pace
Wang el at.
2020
RGB
R(2+1)D-18
16*112
UCF(ld)
75.9
DSM
Wang
202
RGB
C3D
16*112
UCF(d))
70.3
40.5
CMC
Tall m.
2120
RGB
CaffeNet
25*256
UCF(Id)
59.1
26.7
CoCLR
Han ef al.
2020b
RGB
S3D
32*128
UCF(ld)
70,2
81.4
39.1
52.1
MVCGC (ourS)
RGB
S3D
32$128
UCF(1d)
73.1
82.0
41.1
58.4
IIC
Tao 2020
RGB+OF
R3D-18
16*112
UCF(1d)
72.7
36.8
McmDPC
Hum et dr 2020a
RGB+OF
R2D3D-34
40*128
UCF(ld)
84.3
CoCLR
Ham
RGB+OF
S3D
32*
UCF(l))
72.1
87.3
58.7
MVCGC (ours)
RGB+MV
S3D
32*128
UCF(ld)
77.2
87.4
41.0
59.7
CCL
Kong et al. 2020
RGB
R(2+1)D-18
8-112
K400X28d)
52.1
69.4
27.8
37.8
Hun df
2019
R2D3D-34
40*224
K400228)
75.7
Pace
Wang
20E0)
RGB
R(2+1)D-18
16*112
K400(284)
77.1
36.6
SpeedNer
Bonaam ol.
2020
RGB
S3D-G
16*224
K400(28d)
81.1
48.8
DSM
Wing et dl. 202
RGB
R3D-34
16*224
K400(28d)
78.2
VIE
Zhuann
RGB
SlowFast-18
16*112
K400/28d)
80.4
52.5
CoCLK
Han of di. 2020b
RGB
S3D
32*128
K400228)
74.5
87.9
46.1
54.6
MVCGC
(ours)
RGB
32*128
K400X280)
AVTS
Korbur
2018
RGB+Audio
R(++))--18
25*224
K400(28))
86.2
52.3
ELO""
Piergiovannt of 2020
RGB+Aoddoo=OF
R(2+1)D-50
32+224
YouTTuhe800M
(00.00MM(1.9 .9y)
93.8
64.5
67.4
XDC
Alwassel el
RGB+Audio
R(2+1)D-18
32*224
K400(28d)
86.8
GDT
Palnck
RGB+Audio
R(2+1)D-18
32*224
K400(284)
89.3
60.0
CBT
Sun cl o/. 2019
RGB+Text
S3D
16*112
K400(28d)
$4.0
79.5
29.5
44.6
MIL-NCE
Miech
2020
RGB+Text
32*200
Howto (55)
82.7
91.3
IMRNet
Yu
RGB-MV+Res
R3D-50
30+224
K400(28))
77.3
McmDPC
Hil el 2020a
RGB+0F
R2D3D-34
40*224
K400(28d)
54.1
86.1
30.5
54.5
CoCLR"
Ham al. 20206
RGB+OF
S3D
324128
K400(284)
77.8
90.6
52.4
62.9
MVCGC (ours)
RGB+MV
32#128
K400(28d)
78.0
4.5 Comparison with State-of-the-AAt
In this section, compare our MVCGC with the state-of-the-art self-supervised SSVRL approaches on the action classification and video retrieval tasks. For the action classification task, we provide the comparative results in Table by pre-training on UCF101 and Kinetics-400 and then evaluating with either linear probing or fully fine-tuning on UCF101 and HMDB51. We list recent approaches evaluated on the same benchmark, trying to compare with them as fairly as we can, although variations are unavoidable in terms of architecture, training data, and resolution. The "Test View(s)* column refers the view(s) used for test/evaluation, e.g., the result of MVCGC in RRMV views obtained by averaging the predictions from RGB and motion vector networks. We can observe that: (1)

---

Table 5: Comparison with state-of-the-ee art methods in nearest-neight neighbur video retrieval on UCF101 and HMDB51. Videos in the test set are used to retrieve the videos in the training set, and R@k is reported with & E {1, 5, 10, 20, 50). All compared methods pre-trained on UCF101 except SpeedNet pre-trained on larger Kinetics-400. Methods marked with use audio for pre-training.
UCF
HMDB
Method
R@l
R@5
e@ 10
R@
R@ 50
R@
R@s
R@
R@20
R 50
Retrieval with RGB only:
VCP [Luo el al. 2020
18.6
53.5
68.1
24.4
36.3
53.6
76.4
SpeedNet Benam ef dt. 2020
13.0
28.1
37.5
49.5
65.0
DSM
Warg 20211
16.8
33.4
43.4
54.6
70.7
8.2
25.9
38.1
52.0
75.0
PRP
Yao a). 2020
23.2
38.1
46.0
55.7
68.4
10.5
27.2
40.4
56.2
75.9
Mem
[aan
2020a
20.2
40.4
52.4
64.7
7.7
25.7
40.6
57.7
Pace
Wang et at. 2020
31.9
49.7
59.2
68.9
80.2
12.5
32.2
45.4
61.0
80.7
CCL
Kong cf al.
2020
32.7
42.5
50.8
61.2
68.9
CoCLK Han er al.
2020b
53.3
69.4
76.6
82.0
23.2
43.2
53.5
GDT
Patrick
2020
73.4
88.1
92.9
25.4
51.4
63.9
75.0
87.8
MVCGC (ours)
60.8
74.1
79.8
85.8
92.6
24.1
49.7
61.3
73.3
87.5
Retrieval with two views:
IIC Tao er al. 2020
42.4
60.9
69.2
77.1
86.5
19.7
42.9
57.1
70.6
85.9
CoCLR [an et al. 2020b
76.9
82.5
26.1
45.8
57.9
69.7
MVCGC (ours)
66.1
79.1
84.0
89.0
93.7
25.6
49.2
60.7
73.3
86.5
Our MVCGC outperforms most of latest methods under both linear probing and fully fine-tuning settings on both benchmarks.
(2) Our MVCGC using motion vectors from compressed videos clearly beats CMC (Tian et al.) 2020] and CoCLR [Han et al. (2220b] which exploit optical flows for pre-training. (3) Compared to IMRNet (Yu et al.) [021] that also utilizes compressed videos for self-supervised learning, our method achieves significantly better results, demonstrating both efficiency and effectiveness of MVCGC. (4) Our MVCGC outperforms its competitors under the pre-training Kinetics-400 and fine-tuning on UCF101/HMDB: setting, indicating the robustness of MVCGC to process motion vectors that are obtained with different codecs. (5) It can be observed that we outperform some approaches that exploit the correspondence of visual information with text or audio. This is impressive since we only focus on visual representations and only use visual-related views (i.e., RGB, motion vectors, optical flows).
In Table we evaluate our MVCGC in video retrieval on both UCF101 and HMDB51 We can see that our MVCGC significantly outperforms all compared methods in terms of all five metrics UCF101 and four the five metrics on HMDB51, achieving new state-of -tate-of-theeart. Particularly, our obtained retrieval results by retrieval with only motion vectors latest competitor CoCLR [Han et al.) 2020b) which uses optical flows for retrieval, indicating the effectiveness of our MVCGC. Furthermore, obtain comparable results with methods using extra audio modalities Patrick el al. 2020). indicating the effectiveness of MVCGC
higher than those of the
Conclusion
We have proposed a novel self-supervised video representation learning method named Motion Vector based Cross Guidance Contrastive Learning (MVCGC). By introducing on-the-fly decoded motion vectors and cross guidance contrastive learning, our proposed MVCGC has for the first time leveraged contrastive loss in compressed video self supersupeeeieed learning. Extensive experiments carried out to validate the efficiency and effectiveness of MVCGC. Importantly, our approach achieves new state-of-the aftthe-rrt on two downstream tasks across different benchmarks.
Acknowledgments and Disclosure of Funding
This work was supported in part by National Natural Science Foundation of China (61976220 and 61832017), Beijing Outstanding Young Scientist Program (BIJWZYJHOI (0AHHHHH111100022000 2019100020099) and Open Project Program Foundation of Key Laboratory of Opto-Electrooics Information Processing, Chinese Academy of Sciences (OEIP-O-202006) Ping Luo was supported by the General Research Fund of Hong Kong No. No.27208720.
10

---

References
Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. In NeurlPS, 2020.
Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T. Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. Speednet: Learning the speediness in videos. In CVPR, pages 9922-9931, 2020.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In ICML, pages 10709-10719, 2020.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In ICCVW, pages 1483-1492, 2019.
Tengda Han, Weidi Xie, and Andrew Zisserman. Memory-augmented dense predictive coding for video representation learning. In ECCV, pages 312 329, 2020.
Tengda Han, Weidi Xie, and Andrew Zisserman. Self-supervised co-training for video representation learning. In NewrIPS, 2020.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9729-9738, 2020.
Yuqi Huo, Xiaoli Xu, Yao Lu, Yulei Niu, Mingyu Ding, Zhiwu Lu, Tao Xiang, and Ji-Rong Wen. Lightweight action recognition in compressed videos. In ECCVW, 2020.
Yuqi Huo, Mingyu Ding, Haoyu Lu, Ziyuan Huang, Minggian Tang, Zhiwu Lu, and Tao Xiang. Self-supervised video representation learning with constrained spatiotemporal jigsaw. In ICCAI, 2021.
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. arXiv preprint arXiv: 06950, 2017.
Dahun Kim, Donghyeon Cho, and In So Kweon. Self-supervised video representation learning with space-tima cubic puzzles. In AAAI, pages 8545 8552, 2019.
Quan Kong, Wenpeng Wei, Ziwei Deng, Tomoaki Yoshinaga, and Tomokazu Murakami. Cycle-contrast for self-supervised vidco representation learning. In NeurlPS, 2020.
Bruno Korbar, Du Tran, and Lorenzo Torresani, Cooperative learning of audio and video models from self- supervised synchronization. In NeurlPS, pages 7763-7774, 2018.
Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: A large video database for human motion recognition. In ICCV, pages 2556-2563, 2011.
Hsin-Ying Lee, Jia-Bin Huang. Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representation learning by sorting sequences. In ICCV, pages 667-676, 2017.
Dezhao Luo, Chang Liu, Yu Zhou, Dongbao Yang, Can Ma. Qixiang Ye, and Weiping Wang. Video cloze procedure for self-supervised spatio-temporal learning. In AAAI, 2020.
Antoine Micch, Jean-Bape Sean-Baptiste Alayrac, Lueas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End- to-end learning of visual representations from uncurated instructional vidcos. In CVPR, pages 9879-9889, 2020.
Ishan Misra, C. Lawrence Zitnick, and Martial Hebert. Shuffic and learn: Unsupervised learning using temporal order verification. In ECCV, pages 527-544, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: /877 2018
Mandela Patrick, Yuki M. Asano, Ruth Fong, João F. Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi- modal self-supervision from generalized data transformations. arXiv preprint MXXi:2003300229,, 2020.
AJ Piergiovanni, Anelia Angelova, and Michael S. Ryoo. Evolving losses for unsupervised video representation learning. In CVPR, pages 133-142, 2020.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV 115(3):211 2015.
11

---

Zheng Shou, Xudong Lin, Yannis Kalantidis, Laura Sevilla-Lara, Marcus Rohrbach, Shih-Fu Chang, and Zhicheng Yan. Dme-net: Generating discriminative motion cues for fast compressed video action recognition. In CVPR. pages 1268-1277, 2019.
Thomas Sikora. The mpeg-4 video standard verification model. JEEE TCSVT, 7(1):19-1, 1997.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:/212.04022 2012.
Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (heve) standard. JEEE TCSVT, 22(12):1649-1668 2012.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations using contrastive bidirectional transformer. arXiv preprint arXiv: 1906.05743, 2019.
Li Tao, Xucting Wang, and Toshihiko Yamasaki. Self-supervised video representation learning using inter-intra contrastive framework. In ACMMM, pages 2193-2201, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, pages 776-794, 2020.
Jiangliu Wang, Jianbo Jiao, and Yun-Ho Liu. Self-supervised video representation learning by pace prediction. In ECCV, pages 504-521, 2020.
Jinpeng Wang, Yuting Gao, Ke Li, Xinyang Jiang, Xiaowei Guo, Rongrong Ji, and Xing Sun. Enhancing unsupervised video representation learning by decoupling the scene and the motion. In AAAI, 2021.
Donglai Wei, Joseph Lim, Andrew Zisserman, and William T. Freeman. Learning and using the arrow of time. In CVPR, pages 8052-8060, 2018.
Thomas Wiegand, Gary J. Sullivan, Gislc Bjontegaard, and Ajay Luthra. Overview of the h. 264/avc vidco coding standard. JEÉE TCSVT, 13(7):560- 576, 2003.
Chao-Yune Yuan Wu, Manzil Zaheer, Hexiang Hu, R. Manmatha, Alexander J. Smola, and Philipp Krähenbühl. Compressed video action recognition. In CVPR, pages 6026-6035, 2018.
Saining Xie, Chen Sun, Jonathan Huang. Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: accuracy trade-offs in video classification. In ECCV, pages 318 335, 2018.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In CVPR, pages 10334-10343, 2019.
Yuan Yao, Chang Liu, Dezhao Luo. Yu Zhou, and Qixiang Ye. Video playback rate perception for self- self-supervised teatio-temporal representation learning. In CVPR, pages
2020.
Youngjac Yu, Sangho Lec, Gunhee Kim, and Yale Song. Self-supervised learning of compressed vidco representations. In ICLR, 2021.
Bowen Zhang, Limin Wang, Zhe Wang. Yu Qiao, and Hanli Wang. Real-time action recognition with deeply transferred motion vector cans. TIP, 27(5):2326-22 2018.
Chengxu Zhuang, Tianwei She, Alex Andonian, Max Sobol Mark, and Daniel Yamins. Unsupervised learning from video with deep neural embeddings. In CVPR, pages 9563 9972,,97, 2020.
12


Reviews:
[{"review_id": "uX2G34zyECU", "reviewer": "Reviewer_M3h9", "summary": "The paper tackles the problem of self-supervised video representation learning. The authors propose a compressed video based self-supervised learning approach, where the model is forced to learn matching representations between raw RGB clip and its corresponding motion vector clip. This is achieved using a relatively standard contrastive learning formulation. The authors then demonstrate that their model achieves state-of-the-art performance on a few downstream tasks (i.e., action recognition, and video retrieval) while also being significantly more efficient (particularly in terms of the video processing time).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is written clearly, and it addresses a well motivated and an important research problem.\n- A nice idea of using compressed video for self-supervised feature learning.\n- The proposed approach delivers performance that is comparable/better to previous methods (e.g. CoCLR [Han et al., 2020b]) while being much more efficient.\n- Thorough ablation studies.\n\nWeaknesses:\n- The major weakness of this paper is its limited technical novelty. While the idea of using compressed videos in the context of self-supervised learning is new and interesting, overall the proposed approach is largely built on the ideas from CoCLR [Han et al., 2020b]. Having said this, the proposed approach seems to be both more effective and more efficient than CoCLR. Therefore, I think this would be a good \"poster\" paper.\n- The proposed approach performs quite a bit worse than the recent CVPR 2021 methods such as CVRL, and especially the method of Feichtenhofer et al. \"A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning\" (CVPR 2021). Having said this, given a very large computational cost of both of these prior methods (i.e. 800 epoch training), I think that it would be unfair to compare the proposed MVCGC approach directly with these methods. The authors should still add the second reference to their related work section, and also to their quantitative comparisons. A discussion on why the direct comparison between these methods is unfair would also be useful.\n\nQuestions / Minor Details:\n- In L85-88, the authors mention that unlike CoCLR, they construct sample pairs between different views of different clip (i.e., RGB frames of one clip and motion vectors of another clip from the same video). However, the caption of Fig.3 says that the motion vector clip corresponds to the raw RGB clip. Therefore, I was confused, which one is it? Also, if it's the former, could the authors also ablate the latter?\n- Typo in Table 3 (it should be Retrieval instead of Retrival). \n- In the third part of Table 4, the authors highlight/bold the numbers corresponding to their approach even though CVRL achieves higher accuracy.\n- If the authors could include the linear evaluation on Kinetics-400 (similar to \"A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning\") that would be great. I think the evaluation on UCF and HMDB is a bit saturated and outdated. Therefore, it would be useful to have additional numbers on Kinetics, which is a bit more challenging dataset. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations were not discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "Zs2EP5lYtED", "reviewer": "Reviewer_ZVsq", "summary": "This paper proposes a self-supervised method for compressed video representation learning, which helps save both storage and computation. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n- The proposed method directly uses RGB and motion vectors in the compressed video for efficient representation learning.\n- Experimental results show that the proposed method reduces the total (inference) time and storage, and achieves marginal improvement compared to state-of-the-art methods.\n\nWeakness:\n\n- Adding temporal learning objectives such as changing the temporal order of RGB frames or motion vectors within a clip ( as in IMRNet ), which is missing from the proposed method, would greatly help learning video representation.\n- Modeling of the proposed method mostly applies the InfoNCE loss, only selecting of positive and negative samples are updated for RGB and motion vectors.\n- As shown in table 1, the computation overhead largely comes from the preprocessing time. Though not an apple-to-apple comparison, parallel re-encoding for IMRNet may largely mitigate the overhead?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The reviewer would encourage the authors to discuss limitations and impacts of the proposed method, as mentioned in the 'Main Review'. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5 hrs"}, "extra_text": {}}, {"review_id": "YMU5CjQFAUj", "reviewer": "Reviewer_rcKJ", "summary": "This paper presents a self-supervised algorithm for representation learning of videos. The authors propose using RGB clips and Motion Vectors (MV) available in compressed videos to reduce the computational burden of processing. The two streams are seen as views in the contrastive setting: positive pairs are composed of RGB and MV clips of the same video, negative ones have clips belonging to different samples. Moreover, hard positive mining extends the set of positive pairs by seeking other videos that are similar in both views. The approach compares favorably on action datasets UC101 and HMDB51 for both action recognition and video retrieval tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strenghts\n* The contribution of this paper is timely: the computational requirements of contrastive learning hinders its applicability on video representation learning. Compressed videos enables the shrinkage of required time and storage.\n* The work proves how effective use of the less informative MV is on par with the computationally demanding optical flow\n* Extensive experiments on action recognition and retrieval prove that the proposed gets competitive results with state of the art methods: linear evaluation and full fine-tuning accuracy for action recognition and R@k metrics for retrieval.\n\n## Weaknesses\n* The method builds on [1] by extending the single-view contrastive approach to learn the correspondence between two views and changing the hard-positive condition. However, it is well known that cross-view contrastive is beneficial [2, 3, 4]. Yet, the showed technique for harvesting hard positives is an incremental contribution to the literature.\n* How cross guidance and hard positive mining contribute to the final performance is not well disentangled and it's not easy to understand what their impact is. What's the performance loss without hard-positives? What is the effect of using hard positives in the initialization phase? What is the performance of the ensemble of RGB and MV model right after initialization?\n\n### Minor\n* Table 2: I find somewhat surprising that the same technique applied to OF gets lower performances, could the authors provide some explanations/insights?\n* Line 258: concerning the claim \"MVCGC is robust to motion vectors under different encoding configurations\": is not really clear how this is proved. In table 2 MV_CoViAr is used for initialization only.\n* Line 303: I think that it is \"by retrieval with only\" RGB \"are even higher\"\n* Appendix, fig 1: the underlying assumptions is that semantically similar videos are close in both RGB and MV space. How do the authors think the model would handle failures of such assumption, e.g., mopping a gymnastic floor, biking in a volleyball arena, etc.?\n--------------------\n* **Correctness:** The proposed method and the experimental evaluation are sound.\n* **Clarity:** The paper is well written and easy to follow. The content is well organized: contribution is expressed concisely and the experimental part is well explained.\n* **Relation to prior work:** Authors relates their work with previous literature while concisely highlighting their contribution.\n* **Reproducibility:** Architecture, method and optimization hyperparameters are available for implementation, missing augmentation parameters (random cropping, blurring, jittering) and missing code limits the reproducibility of the work though.\n* **Originality**: The contribution of this paper is incremental with respect to cited papers, the real advantages of the technique are not thoroughly highlighted in the ablation part.\n------------------------\n[1] Han, Tengda, Weidi Xie, and Andrew Zisserman. \"Self-supervised Co-Training for Video Representation Learning.\" Advances in Neural Information Processing Systems 33 (2020): 5679-5690.\n\n[2] Tian, Yonglong, Dilip Krishnan, and Phillip Isola. \"Contrastive multiview coding.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16. Springer International Publishing, 2020.\n\n[3] Bachman, Philip, R. Devon Hjelm, and William Buchwalter. \"Learning Representations by Maximizing Mutual Information Across Views.\" Advances in Neural Information Processing Systems 32 (2019): 15535-15545.\n\n[4] Chen, Ting, et al. \"A simple framework for contrastive learning of visual representations.\" International conference on machine learning. PMLR, 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A. However, I would argue that no specific discussion is needed on potential negative societal impact for a contrastive learning method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "SZX6XWrwasR", "reviewer": "Reviewer_Rmsi", "summary": "The authors propose a self-supervised learning (SSL) method for learning strong visual representations based on compressed videos. Compared with prior work, which sees videos in RGB only, the proposed method is more efficient and leverages the motion information that's already accessible in compressed videos. The authors propose to use a contrastive learning method to supervise on kind of modality with correspondence of another modality. On multiple datasets (mainly UCF and HMDB), good performance is achieved, with both linear probing and transfer learning settings.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[originality: medium-high]\n - The idea of using compressed videos for self-supervised learning is novel/original. Most prior work that operates on compressed videos focues on supervised learning setting only. The implementation of this idea is standard/expected (this is not a drawback).\n\n[quality: medium-high]\n - The paper covers useful ablation study that I'd expect. In particular, the authors demonstrate the usefulness of the compressed-video modality (MV) compared to RGB only (Table 3). Compared to prior work, the performance is also strong while being significantly more efficient. The ablation study that compares with optical flow and low-res optical flow is also interesting (Table 2).\n\n[clarity: high]\n - I think this paper is overall clearly writen. One question about implementation details: How is the similarity calculated in the hard-positive mining procedure?\n\n[significance: medium-high]\n - The area of SSL draws a lot of attention in our community and is of great potential. I think this work extends our knowledge from a new dimension with the use of compressed videos. I think this work can lead to promising new directions in SSL research. \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. In the paper, the authors perform experiments/evaluation solely on videos. I was wondering if the method learns good representations for \"image\" tasks as well. Namely, leveraging MV can potentially give us better RGB representations for images, too. I wonder what the authors think about this.\n2. If the hard-positive mining is based on the learned features, would there be any \"chichen-and-egg\" problem introduced by the proposed procedure?\n3. I like the comparison to MV_{CoViAR}, but I wonder how each specific compression setting affects the results. For example, how does GOP size, the use of B-frame vs P-frame, etc. affect results.\n\nRegarding societal impact, I don't see anything particularly concerning (or more concerning) than other SSL work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received the following final ratings: 6, 6, 7, 4.

Reviewer rcKJ raised concerns about the close technical reliance on CoCLR [Tengda et al., NeurIPS 2020] and the lack of understanding about the impact of cross guidance on performance. The authors rebutted these points in their response. Unfortunately, Reviewer rcKJ failed to provide post-rebuttal comments, but the ACs found the points made by the authors convincing.

The other reviewers appreciated the responses provided by the authors, especially the clarification about the similarity calculation, the differences wrt CoCLR, and the inclusion of the linear evaluation on Kinetics-400. As noted by Reviewer Rmsi, an ablation on how the different compression settings affect performance would further strengthen this work. The ACs agree with the majority recommendation of acceptance expressed by the reviewers.
Decision: Accept (Poster)