Self-Supervised Learning of Compressed Video Representations
Authors: Youngjae Yu
Sangho Lee
Gunhee Kim
Yale Song
Venue: ICLR 2021 Poster (2021)
Keywords: Compressed videos
self-supervised learning

Abstract: Self-supervised learning of video representations has received great attention. Existing methods typically require frames to be decoded before being processed, which increases compute and storage requirements and ultimately hinders large-scale training. In this work, we propose an efficient self-supervised approach to learn video representations by eliminating the expensive decoding step. We use a three-stream video architecture that encodes I-frames and P-frames of a compressed video. Unlike existing approaches that encode I-frames and P-frames individually, we propose to jointly encode them by establishing bidirectional dynamic connections across streams. To enable self-supervised learning, we propose two pretext tasks that leverage the multimodal nature (RGB, motion vector, residuals) and the internal GOP structure of compressed videos. The first task asks our network to predict zeroth-order motion statistics in a spatio-temporal pyramid; the second task asks correspondence types between I-frames and P-frames after applying temporal transformations. We show that our approach achieves competitive performance on compressed video recognition both in supervised and self-supervised regimes. 


Published as a conference paper at ICLR 2021
SELF-SUPERVISED LEARNING OF COMPRESSED VIDEO REPRESENTATIONS
Youngjae Yu", Sangho Lee", Gunhee Kim
Scoul National University (yj-yu, sangho. lee) @vision acu.ac gunhee@smu.ac.ke
Yale Song
Microsoft Research
yalesongemiccroooft.com
ABSTRACT
Self-supervised learning of video representations has received great attention. Ex- isting methods typically require frames to be decoded before being processed, which increases compute and storage requirements and ultimately hinders large- scale training. In this work, we propose an efficient self-supervised approach to learn video representations by eliminating the expensive decoding step. We use a three-stream video architecture that encodes I-frames and P-frames of compressed video. Unlike existing approaches that encode I-frames and P-frames individually, we propose to jointly encode them by establishing bidirectional dynamic connec- tions across streams. To enable self-supervised learning, we propose two pretext tasks that leverage the multimodal nature (RGB, motion vector, residuals) and the internal GOP structure of compressed videos. The first task asks our network to predict zeroth-order motion statistics in spatio-temporal pyramid; the second task asks correspondence types between I-frames and P-frames after applying temporal transformations. We show that our approach achieves competitive performance compressed video recognition both in supervised and self-supervised regimes.
INTRODUCTION
There has been significant progress on self-supervised learning of video representations. It learns from unlabeled videos by exploiting their underlying structures and statistics as free supervision signals, which allows us to leverage large amounts of videos available online. Unfortunately, training video models is notoriously difficult to scale. Typically, practitioners have to make trade-offs between compute (decode frames and store them as JPEG images for faster data loading, but at the cost of large storage) and storage (decode frames on-the-fly at the cost of high computational requirements). Therefore, large-batch training of video models is difficult without high-end compute clusters. Although these issues are generally applicable to any video-basee scenarios, they are particularly problematic for self-supervised learning because large-scale training is one key ingredient (Brock et al. 2019 Clark et al. 2019 Devlin et al. 2019) but that is exactly where these issues are aggravated.
Recently, several approaches demonstrated benefits of compressed video recognition (Zhang et al.) 2016 Wu et al. 22018 (Shou et al.] 2019) Wang et al.] (019bb). Without ever needing to decode frames, these approaches can alleviate compute and storage requirements, e.g., resulting in to 10 times faster solutions than traditional video CNNs at a minimal loss on accuracy (Wu et al. 2018 Wang et al. 2019b). Also, motion vectors embedded in compressed videos provide free alternative optical flow which is compute-intensive faster than optical flow-based approaches (Shou et al., 2019). However, all the previous work on compressed video has focused on supervised learning and there has been no study that shows the potential of compressed videos self-supervised learning; this the focus our work.
leveraging this has been shown to be two orders of magnitude
"Equal Contribution

---

Published as a conference paper at ICLR 2021
Compressed video
IMR Network
Self-supervised pretraining
Pyramidal Motion Statisties Prediction
Network
Where is must dynamic reginn?
I-Frames
M - Network
Correspondence Type Prediction
New are I frames and P frames related?
Motion Vectors
R Network
StufTee
Shifted
Residuals
Ridirectional Dynamic Connection
Figure IMR network consists of three sub-networks encoding different information streams provided in compressed videos. We incorporate bidirectional dynamic connections to facilitate information sharing across streams. We train the model using two novel pretext tasks designed by exploiting the underlying structure of compressed videos.
In this work, propose self-supervised approach to learning video representations directly in the compressed video format. We exploit two inherent characteristics of compressed videos: First, video compression packs sequence of images into several Group of Pictures (GOP). Intuitively, the GOP structure provides atomic representation of motion; each GOP contains images with just enough scene changes so video codec can compress them with minimal information loss. Because of this atomic property, we enjoy less spurious, more consistent motion information at the GOP-level than at the frame-level. Second, compressed videos naturally provide multimodal representation (i.e. RGB frames, motion vectors, and residuals) that we can leverage for multimodal correspondence learning. Based on these, we propose two novel pretext task (see Fig. []] The first task asks our model predict zeroth-order motion statistics (e.g. where is the most dynamic region) in pyramidal spatio-temporal grid structure. The second involves predicting correspondence types between I-frames and P-frames after temporal transformation. Solving our tasks require implicitly locating the most salient moving objects and matching their appearance-motion correspondences between I-frames and P-frames; this encourages our model to learn discriminative representation of compressed videos.
compressed video contains three streams of multimodal information - i.e. RGB images, motion vectors, and residuals with dependency structure between I-frame stream and the two P-frame streams punctuated by GOP boundaries. We design our architecture to encode this dependency structure; it contains residuals in P-frames, respectively. Unlike existing approaches that encode I-frames and P-frames individually, we propose to jointly encode them fully exploit the underlying structure compressed videos. To this end, we use a three-stream CNN architecture and establish bidirectional dynamic connections going from each of the P-frame streams into the I-frame stream, and vice versa, and put these connections layer-wise to learn the correlations between them at multiple spatial/temporal scales (see Fig. These connections allow our model fully leverage the internal GOP structure of compressed videos and effectively capture atomic representation of motion.
CNN encoding I-frames and two other CNNs encoding motion vectors and
In summary, our main contributions are two-fold: (1) We propose a three-stream architecture for compressed videos with bidirectional dynamic connections to fully exploit the internal structure of compressed videos. (2) We propose novel pretext tasks to learn from compressed videos self-supervised manner. We demonstrate approach by pretraining the model on Kinetics-400 (Kay et al. 2017) and finetuning it on UCF-101 (Soomro et al. 2012). HMDB-51 (Kuehne et al. 2011). state-of-the-at performance in compressed video classification tasks in
Our model achieves both supervised and self-super regimes, while maintaining a similar computational efficiency as existing compressed video recognition approaches (Wu al. 2018 Shou et al. 2019)
2

---

Published as a conference paper at ICLR 2021
Tyx
HxW
I-Pathway
I-Pathway
0
Pooling
Cacs
3D
Vi-M
Decone
Muntinatal
Mictton
Bidrrcttonall
LxG
1x
B
Dynamic Connectioos
M- or
1xCM
1x
mise
R- R-Pathway
3D Conv
Vye-I
M-Pathway
R- R-Pathway
TyxWWHXCG
TixWxIIX
Figure Bidirectional dynamic connections are established between I M pathways and between I & R pathways to facilitate information sharing between I-frames and P-frames. We incorporate multimodal-guee gated attention to dynamically modulate the connections based on input. Feature tensors (orange and blue cubes) are placed in the T-HxW-C plane.
APPROACH
We use videos compressed according to the MPEG-4 Part specifications (Le Gall, 1991 as our input, following the previous work (Wu et al. 2018 Shou et al. 2019 Wang et al. 2019b). This compression format encodes an RGB image sequence as a series of GOPs (Group of Pictures) where each GOP starts with one I-frame followed by variable number of P-frames. An I-frame stores RGB values of a complete image and can be decoded on its own. A P-frame holds only the changes from the previous reference frame using motion vectors and residuals. The motion vectors store 2D displacements of the most similar patches between the reference and the target frames, and the residuals store pixel-wise differences to correct motion compensation errors. We use all the three modalities contained in compressed videos as our input.
Formally, our input is T GOPs, Go. Gr-1, where each G2 contains one I-frame I, Hxxx3 followed by K 1 pairs of motion vectors Mr,x E HR W x2 and residuals Re, E R#x E [1, K). For efficiency and simplicity, we assume an identical GOP size K for all & (0,
2.1
IMR NETWORK FOR COMPRESSED VIDEOS
Our model consists of three CNNs, each with 3D convolutional kernels modeling spatio-temporal dynamics within each input stream { {Mex). J. {Re,) j0, k E [0, K); we denote these sub-networks by I-network fi. M-network fM, and R-network fn. respectively, and call our model IMR Network (IMRNer). We account for the difference in the amount of information between I-frames and P-frames by adjusting the capacity of networks accordingly. Specifically, following (Wu et al.) 2018), make the capacity of fi larger than JM and In by setting the number of channels in each layer of fi to be y times higher than those of fM and fR (we set 7 64).
Existing models for compressed videos typically perform late fusion (Wu et al., 2018 Shou et al. 2019), i.c. they combine embeddings of I-frames and P-frames only after encoding each stream. However, we find that it is critical to allow our sub-networks share information as they encode their respective input streams. To this end, we establish layer-wise wise lateral connections between fi & fas and between fi fR-
Bidirectional dynamic connections. Lateral connections have been used to combine information from different streams, e.g., RGB images and optical flow images (Feichtenhofer et al. 2016), and RGB images sampled at different frame rates (Feichtenhofer et al.) 2019). In this work, we use to combine information from I-frames and P-frames. Our approach different from previous work in two key aspects: (1) We establish bidirectional connections between streams, instead of unidirectional connections as was typically done in the past (Feichtenhofer et al. 2016 2019). so that information sharing is symmetrical between streams. (2) We incorporate mulzimodal gated attention to dynamically adjust the connections based on multimodal (I-frame and P-frames) information. We call our approach bidirectional dynamic connections to highlight these two aspects and differentiate
3

---

Published as a conference paper at ICLR 2021
(a) r= ([2 x 5), Ans 1,,
(b) r= ([4 4], 1), Ans
Pooling
4
GOPs
way Classification x [t-5]
16 way Classification [t=1]
Figure 3: Pyramidal motion statisties prediction asks our network to find a region with the highest energy of motion. Here we visualize two levels in a spatio-temporal pyramid for illustration.
ours from previous work, e.g., SlowFast networks (Feichtenhofer et al. 2019) establish unidirectional lateral connections and the connections are static regardless the content from the other stream.
We combine embeddings from different sub-networks via channel-wise concatenation, which requires embeddings to match their spatio-temporal dimensions. However, fi processes x times less frames than far and fR. producing embeddings that times smaller in the temporal dimension. Therefore, we transform the embeddings with time-strided 3D ade-jconvolution with (Λ x x 1) kernels, C/8 channels, and (vs, 1, 1) temporal stride: We convolution for fi - fM/ fn decrease the time dimension and deconvolution for fas fn - fi to increase it. Note that simply using the (de- -cony layers will perform static transformation regardless of what is provided from the other sub-network, similar to (Feichtenhofer et al. 2019). However, we find it critical to make the transformations aware of information from both and selectively share only the most relevant information from each sub-network
so that the networks can dynamically adjust the connections
To achieve this, we dynamically modulate (de-)cony layer outputs using multimodal-gan gated attention weights. Let X, E RT, xW x xCI and XM E T XW xH be the embeddings from fi and far. respectively. We max-pool x; and Xas and concatenate them to obtain multimodal embedding 2 RC with Cz = C₁ + CM- We define multimodal gate functions that take as input z and generate attention weights E Rt/8 and aM E R CM/ as
(1) where is sigmoid function, Ç is Leaky ReLU function, and W1, W2 E CRCxxC2 by, b2 E RC2 W3 E CC/88x2z by E gg/// W4 E CM//8xxx 6y E RCM// are weight parameters. Next, we use these attention weights modulate the (de-)conv output embeddings,
a (Wah ba), aM (Wah + ba), h (W1z z b1) 62)
VI+MM = aM @ od_conv(xx), VM-+1 = a, 6 3d_ sd_deconv(xx)
where @ is channel-wise multiplication. We repeat the same process for fr & J'R to obtain V and VR+1. and combine them with the feature embeddings via channel-wise concatenation,
x {x> VM-+1; VR-I]; XM [XM:VI-M] Xn [xx: V1++R]
(3)
Each of these is fed into the next layer in the corresponding sub-network. We establish these lateral connections across multiple layers of our network. To obtain the final embedding, we apply average pooling on the output from the final layer of each sub-network and concatenate them channel-wise.
Note that the design of IMRNet is orthogonal to the design of video CNNs; while we adapt 3D- ResNet (He al. 2016) as the backbone in our experiments, architectures as the backbone, e.g., C3D (Tran et al., 2015). I3D (Carreira & Zisserman 2017). R(2+1)D (Tran al., 2018). What is essential, however, that (i) there are three sub-networks each modeling one of the three input streams, and (ii) information from different networks are combined via bidirectional dynamic connections as they are encoded.
use any of existing CNN
2.2
SUAF-SUPERVISEE LEARNING OBJECTIVES
Compressed videos have unique properties, i.c., the multimodal nature of information (RGB, motion vector, residuals) and the internal GOP structure that provides atomic representation of motion. We turn these properties into free self-supervisory signals and design two novel pretext tasks.
Pyramidal Motion Statistics Prediction (PMSP). One important desideratum of video CNNs is learning visual representation that captures salient objects and motion. We hypothesize that there an implicit videographer bias captured in videos in-the-will that naturally reflect visual saliency: Videos
4

---

Published as a conference paper at ICLR 2021
(a) Aligned
Randon om
(d) Shift
Shufile
Figure 4: Correspondence type prediction asks network to categorize different types of data transformations applied on P-frames. We illustrate four transformation types used in our experiments.
purposely recorded to highlight important objects and their movements Therefore, regions with the highest energy of motion can provide clues to learning the desired video representation. We easily find those regions in compressed videos: the motion vectors in P-frames readily provide magnitude and angular information of motion, which we can harness to find the most vibrant regions.
Based on this intuition, design task that asks model predict the zeroth-order motion statistics (i.e., the most vibrant region) in given video. For this, we must be able to deal with variety of object sizes because salient moving object can appear at any location in any size. A classical solution to this is to perform pyramidal prediction (Grauman & Darrell, 2005 Lazebnik al., 2006):: We divide a video into spatio-temporal 3D grids at multiple scales and ask our network to predict the most vibrant region at each scale.
Specifically, we define a pyramidal classification task with the following loss function,
CPMSP = - -EE vot
log ar
(4)
This is a cross-entropy loss computed at every q-th grid in every r-th level of a spatio-temporal pyramid; i is the sample index. We define 9-level spatio-temporal pyramid with 3 spatial and temporal scales, i.e., r E {(s, t)|s E x[2x2],
2),(3,, [3x3], [4 4]]] I all possible temporal coordinates in the r-th level of the pyramid, e.g., Figure 3 (a), q [0,...
The index iterates over
with r = ([2 x 5). (4) is a one-hot label marking the location with the highest energy of motion in the q-th grid in r-th level in the pyramid, e.g., in Figure 3 (a), Vg.r 4-dimensional one-hot vector. We provide a pseudo-code to obtaín the ground-trutt labels from motion vectors in Appendix. x is the (g, r)th feature in 3D grid; concatenate output embeddings from all three sub-networks,
(+) [x XM XR
(6) Finally, (-) is a 2-layer MLP with a softmax classifier predicting the most vibrant region the given grid; we define such classifier for each r.
(0)
Correspondence Type Prediction (CTP). One idea often used in suf-suuervision is applying certain transformations data and asking network to predict the correspondence type given pair of instances (e.g., true pair or randomly selected pair) (Owens & Efros) 2018 Chen et al. 2020 He et al. 2020; Misra van der Maaten 2020). The multimodal nature of compressed videos makes them an ideal data format to apply such self- supervieeiii technique: The three frame types in compressed videos exhibit different characteristics, yet they are strongly correlated with each other. This allows us consider I-frames as a heavily transformed version of the corresponding P-frames, and vice versa. Learning the correspondence type between I-frames and P-frmmes can therefore encourage our network to learn discriminative representation of videos.
This is, of course, weak hypothesis. But show some convincing empirical evidence in Appendix.
5

---

Published as a conference paper at ICLR 2021
We define correspondence type prediction task with the following loss function,
LCrr = -> Mj log
(x (6)
X x) T M
(5)
where i is the sample index and iterates over set of transformations. Mjj is a one-hot label indicating different correspondence types determined by the type of transformation done, and T(::)) data transformation function that changes the input using the j-th transformation. We define four transformation types (see Figure 4): (1) Aligned keeps the original input (no transformation), (2) Random replaces the data with P-frames from randomly selected video, (3) Shuffle randomly shuffles the GOP order, (4) Shift randomly divides GOPs into two groups and switch the order, e.g.. [1, 2, 4, 5] to [2, 3, 4,5, 5, 1]. Finally, B()) is a 2-layer MLP with a softmax classifier.
Note that there is nuanced difference between random P-frames and shuffled/hiiteed P-frames. The former contains P-frames that come from a different clip, while the latter contains P-frames of the same clip as the I-frames, yet in different frame order. Intuitively, the former encourages network to learn from global (clip-level) correspondence, while the latter formulates a local (frame- CTP task encourages network to learn discriminative
level) correspondence task. Therefore, representations at both global and local levels. We provide empirical evidence showing the importance of this global-local mixed objective in Section 3.2
Final Objective. We optimize our model using a learning objective LPMSP + XLCTP with À = 1. The classifiers o, and are used only during self-supervised training; we detach them thereafter.
3 EXPERIMENTS
Implementation Detail. We adopt 3D ResNet (He et al. 2016) as the backbone; see Appendix for architectural details. We establish bidirectional dynamic connections after (convl, res2, res3, res4] layers. We pretrain our model end-to-end from scratch for 20 epochs, including the initial warm-up period of 5 epochs. For downstream scenarios, we finetune our model for 500 epochs for UCF-101 and for 300 epochs for HMDB-51, including the warm-up period of 30 epochs. For both the pretraining and finetuning stages, we use SGD with momentum 0.9, weight decay 10-4, and half-period cosine learning rate schedule. We use 4 NVIDIA Tesla V100 GPUs and use a batch size of 100.
Data. We pretrain our model on Kinetics-400 Kay et al. 2017). For evaluation, we finetune the pretrained model for action recognition using UCF-101 (Soomro et al. 2012) and HMDB-51 (Kuchne et al. 2011). We use 2-second video clips encoded in 30 FPS with GOP size T 12. We use all = 5 GOPs but subsample every other P-frames within each GOP; this results in 5 I-frames and 25 P-frames. We randomly crop 224 x 224 pixels from videos resized 256 pixels in the shorter side while keeping the aspect ratio. For data augmentation, we resize the video with various scales [,975, .9, .85] and apply random horizontal flip. For test videos, we take three equidistant 224 224 pixel crops from videos resized to 256 pixels to fully cover the spatial region. We approximate the fully-convolutional testing (Wang et al. 2018 by averaging the softmax scores for final prediction.
3.1 SUPERVISED LEARNING EXPERIMENTS
We first demonstrate our proposed IMR network in the fully-supervised setup, training it without using our self-supervised pretext tasks. We use the standard training and evaluation protocols for both UCF-101 (Soomro et al.] 2012) and HMDB-51 (Kuehne et al. 2011). For fair comparisons with existing approaches (Wu et al. 2018 Shou et al., 2019), we report results both when we train the model from scratch and when we pretrain it on Kinetics-400 (Kay et al. 2017) and finetune it on downstream datasets (indicated in column Pretrain).
Table I summarizes the results. When trained from scratch, our model outperforms CoViAR (Wu) ct al. 2018) by a large margin regardless of the chosen backbone. The performance gap is alleviated when the models are pretrained on Kinetics400, but our approach continues to outperform them even in this scenario. This suggest that CoViAR struggles to learn discriminative representations without help from large-scale pretraining data. We believe the performance gap comes from the difference in how the two models encode compressed videos: CoViAR combines information from I-frames and P-frames only after encoding them separately, while combine them in the early layers of CNN.

---

Published as conference paper at ICLR 2021
Models
OF
Pretrain
Backbone
UCFI01
HMDB51
CoViAR
X
Scratch
ResNet152
43.8
27.3
IMR (No connection)
X
Scratch
3D-ResNet188
52.7
34.6
IMR (Unidirectional)
X
Scratch
3D-ResNNt118
69.7
40.8
IMR (No conv)
X
Scratch
3D- ID-ReeNet188
71.7
42.6
IMR (No attention)
X
Scratch
3D-ResNet18
73.2
43.5
IMRNet
X
Scratch
3D-ResNct18
74.1
43.7
IMRNet
X
Scratch
3D-ResNeL50
80.2
55.9
CoViAR
X
ImageNet
ResNet152 (I), ResNet18
90.4
59.1
CoViAR
X
Kinetics400
ResNet152
90.8
59.2
IMRNet (Ours)
X
Kinetics400
3D-ResNet118
91.4
62.8
IMRNet (Ours)
X
Kinetics400
3D-ResNe1500
92.6
67.8
CoViAR
V
ImageNet
ResNet152 (I), ResNet18 (P, OF)
94.9
70.2
DMC-Net!
V
ImageNet
ResNet152 (I), ResNet18 (P)
90.9
62.8
DMC-Net!
"
/
ImageNet
ResNet152 (I), I3D (P)
92.3
71.8
IMRNet (Ours)
Kinetics400
3D-ResNet50 (I. P), I3D (OF)
95.1
72.2
Table I: Results from the supervised setting. Column OF indicates results using optical flow during training. Column Pretrain indicates datasets used for supervised pretraining. 1 published results. : our results based on official implementations by the authors.
Models
R5sNet152'
R(2+1)D
CoViAR
DMC
IMR (RI8)
IMR (R50)
Preprocess (ms)
75.00
75.00
2.87
2.87
2.87
2.87
Inference (ms)
7.50
1.74
1.30
1.91
1.36
1.44
Total (ms)
82.50
76.74
4.17
4.78
4.23
4.31
GFLOPs
11.3
0.96
4.2
4.4
0.66
1.04
Table 2: Runtime analysis of per-frame speed (ms) and FLOOPs The number of input frames are different across models: frame (since is a 2D CNN), 16 frames, $ 25 frames.
CoViAR and DMC-Net reported improved results when they trained using optical flow. Therefore, we also conduct experiments by adding an I3D network (Carreira & Zisserman, 2017) to encode optical flow images; simply concatenate our IMRNet features with the 13D features our final representation (no lateral connections between IMRNet and I3D). This model outperforms both CoViAR and DMC-Net trained with optical flow (bottom group, Table [ DMC-Net improves upon CoViAR by adapting GANs (Goodfellow et al. 2014) to reconstruct optical flow from P-frames. Note that our approach (with 3D-ResNet500 backbone) outperforms DMC-Net. (with ResNet 52/18 backbones) on both datasets even without using optical flow during training and thus significantly simplifies the training setup (no GANs required).
Next, we conduct an ablation study the bidirectional dynamic connection: (a) No connection removes lateral connections and thus is similar to CoViAR, (b) Unidirectional establishes connections from M/R-Networks
I-Network, but not vice versa, i.e., Equation equation] becomes λM = XM. XR = XR. (c) No conv replaces (de-)conv layers with simple up/down- saadloig..p.ingg. (d) No attention removes the multimodal-gated attention module. The results shown in Table We can see that lateral connections are critical component our model (Ours vs. No connection) and doing so bidirectional fashion significantly improves performance (Ours vs. Unidirect ion). We can also see that using (de-)conv layers and dynamically modulating the connection with gate functions improve performance (Ours VS. conv and No attention
Table 2 shows per-frame runtime speed (ms) and GFLOPs measured on an NVIDIA Tesla P100 GPU with Intel E5-2698 v4 CPUs process individual frames. and process 16- and 25-frame sequences, respectively). Our approach has the same preprocessing time of CoViAR and DMC because all three approaches use the same video loader implementation (Wu et al. 2018).
for the
inference speed, IMRNet is comparable to CoViAR and even slightly faster than DMC (we divide the total inference time by #frames following the convention Wu et al. (2018)) This is partly because use lighter backbones (RI 8/R50 vs. R152 used in CoViAR and DMC) to compensate for the expensive 3D convolutional operations, while DMC requires an OF generator network of all-convolutional layers, which adds extra cost. In terms of frrmfr FLOPs, ours is more efficient than ViAR and DMC because the computation is done at the sequence-level rather than per-frames observe similar trend for R(2+1)) (which uses ResNet18) ResNet1 This shows that
7

---

Published as conference paper at ICLR 2021
Models
Compressed Modality
Pretext
Pretrain
Backbone
UCF101
HMDB51
C3D
X
MotPred
Kinetics400
C3D
61.2
33.4
3D-ResNet18
X
V
RotNet3D
Kinetics600
3D-ResNet18
62.9
33.7
3D-ResNet18
XXXXX
V
ST-Puzzle
Kinetics400
3D-ResNet18
65.8
33.7
R(2+I)D-18
V
ClipOrder
UCF101
R(2+1)D-18
72.4
30.9
3D-ResNet34
DPC
Kinetics400
3D-ResNet34
75.7
35.7
Multisensory
A+V
Multisensory
Kinetics400
Audio- Nes
82.1
AVTS
A+V
AVTS
Audioset
MC3
89.0
61.6
ELo
X
A+V
ELo
Kinetics400
(2+ I+D ResNet-50
93.8
67.4
CoVIAR
Scratch
None
55
43.8
IMRNet
V
Scratch
None
3D-ResNet18
74.1
43.7
CoViAR
V
V
AOT
Kinetics400
52
53.6
29.3
CoViAR
Rotation
Kinetics400
55sNet152
56.7
31.4
IMRNct
V
InfoNCE
Kinetics400
3D-ResNet18
73.9
43.7
IMRNet
V
AOT
Kinetics400
3D-ResNet18
74.6
44.0
IMRNet
V
Rotation
Kinetics400
3D-ResNet18
75.1
44.3
CoVIAR
PMSP
Kinetics400
ResNet1.22
63.5
35.9
CoViAR
V
CTP
Kinetics400
ResNet1.52
64.4
37.4
CoViAR
CTP (Binary)
Kinetics400
ResNet152
63.7
37.1
IMRNet
V
PMSP
Kinetics400
3D-ResNet18
76.0
44.9
IMRNet
V
CTP
Kinetics400
3D-
1D-ResNet18
76.7
44.8
IMRNet
CTP (Binary)
Kinetics400
ResNet18
74.6
44.2
IMRNet
PMSP+CTP
Kinetics400
3D-ResNet18
76.8
45.0
Table 3: Results from the self-supervised setting. Column Compressed indicates the methods
that learn directly from compressed videos without decoding them. Modality indicates whether a method used only visual (V) modality or audio-visual modalities (A+V). Pretrain indicates datasets used for self-supervised pretraining. & based on official implementation by the authors.
3D CNN backbones do not bring any significant extra cost compared CoViAR and DMC, and thus our model enjoys all the computational benefits of compressed video processing.
3.2 SELFFSUPERVVSRAI LEARNING EXPERIMENTS
We move to the self-supervise regime and demonstrate our pretext tasks by pretraining our IMRNet on Kinetics400 (Kay al. 2017) and transferring it to action recognition. Because ours the first self-supervised approach to learn compressed video representation, there exist published bascline that we can directly compare with. Therefore, we provide results from existing self-supervised approaches that require the decoding step. We include approaches that learn from RGB images AOT (Wei et al. 2018), Rotation (Jing et al. 2018), MotPred (Wang et al. 2019a), RotNet3D (Jing al. 2018), ST-Puzzle (Kim et al. 2019), ClipOrder (Xu al. (019)) DPC (Han al.
2019) as
well as those that learn from audio and visual channels in videos Multisensory (Owens & Eiros
2018 AVTS KKrbbar al. 2018). Elo (Piergiovanni et 2020)
Table 3 summarizes the results. We first notice that pretraining the models with any pretext tasks improves downstream performance (the first group of results), suggesting self-supervised pretraining is effective in general. We also see that IMRNet pretrained using our pretext tasks (PMSP+CTP) outperforms the baseline pretext tasks (second group) and self-supervised methods for uncompressed videos (third group). This shows the effectiveness of our IMRNet pretrained with our pretext tasks.
Next, we conduct an ablation study by pretraining the base models using either PMSP and CTP alone. We also test CTP (Binary) which is a simplied version of our CTP task with only two modes: Aligned and Random (see Figure 4). Note that this is a typical pair correspondence setup used literature (Arandjelovic & Zisserman 2017). Table (fourth group) shows the results. We can see that using either of our pretext tasks leads to a significant improvements compared to the Scrat ch result. The CTP (Binary) results suggests that the two additional transformation types (Shuffle and Shift in Figure improves the task by making it more difficult to solve; we noticed that the loss curve of CTP (Binary) decreases significantly faster than CTP and quickly saturates thereafter.
4
8

---

Published as a conference paper at ICLR 2021
4 RELATED WORK
Sell-supervised learning of video representation. Self-supervised learning has received significant
attention (Kumar BG et al. 2016, Santa Cruz et al., 2017 Doersch et al. 2013 Wang & Gupta 2015).
Based on strong progress in the image domain, several works proposed to learn video representations
in self-supervised manner. One popular idea is leveraging temporal information (Wang & Gupta
2015; Isola et al. 2015: Jayaraman & Grauman| 2016 Misra et al. 2016: Fernando et al. [2017]
Wei et al. 2018). Temporal coherence of video pixels has been leveraged as self-supervisory
signal (Vondrick et al., 2018 Wang ct al. 2019c). Another popular idea is learning transformation-
invariant representations (Kim et al., 2019| Gidaris et al. 2018: Jing et al., 2018). Also, contrastive
learning (Oord et al. 2018 Hjelm et al.) 2018 He al. 2020: Chen et al. (020) has been successfully
applied to videos (Han et al. 2019). Despite active research in this field, to the best of our knowledge,
there has not been prior work on self-supervised learning from compressed videos.
Compressed video recognition. Compressed video understanding has been tackled in a supervised setting (Zhang al. 2016, Wu et al. 2018 Shou et al. 2019). Existing approaches encode each stream separately and perform late fusion, e.g., feature concatenation (Zhang et al., 2016 Wu et al. 2018). However, as we show in our experiments, this can miss out useful information that can only be learned by modeling the interaction across streams. Unlike previous approaches, our approach shares relevant information across streams during the encoding process. In addition, because compressed videos do not provide continuous RGB frames, it is not easy to directly apply 3D CNNs to encode I-frames. Therefore, existing approaches use 2D CNNs to process compressed video frames, e.g., CoViAR (Wu et al., 2018) uses 2D CNNS to process each stream and perform average pooling over P-frames, which is insufficient to model complex motion dynamics. DMC-Net (Shou et al. 2019) reconstructs the optical flow from P-frames and later use the reconstructed signal as input to I3D CCrreira & Zisserman 2017), but this requires ground-truth optical flow which is compute- intensive. Instead, our IMR network adopts the gated attention Hu al. (2018); Ryoo et al. (2020a) and bidirectional connection Ryoo et al. (2020b): Feichtenhofer et al. (2019) for lateral connection to model complex motion dynamics with I.P frames freely available in compressed videos.
5 CONCLUSION
We introduced an IMR network for compressed video recognition and two pretext tasks for self- supervised learning of compressed video representation. Our work complements and extends existing work on compressed video recognition by (1) proposing the first self-supervised training approach on the compressed videos, and (2) proposing three-stream 3D CNN architecture to encode compressed videos while dynamically modeling interaction between I-frames and P-frames. We demonstrated that IMRNet outperforms state-of-the-an approaches for compressed videos in both fully-supervised and self-supervised settings, and that our pretext tasks yield better performance in downstream tasks.
A Acknowledgement We thank SNUVL lab members, especially Hyeokjun Kwon and Hyungyu Park, for their helpful discussions. This research was supported by Scoul National University. Brain Research Program by National Research Foundation of Korea (NRF) (2017M3CCA1047860) and AIR Lab (AI Research Lab) in Hyundai Motor Company through HMC-SNU AI Consortium Fund, and Institute of Information communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) NNo.2019-0-01088 SW Starl.ab).
REFERENCES
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint 2xXiv:2002.05709 2020.

---

Published as a conference paper at ICLR 2021
Aidan Clark, Jeff Donahue, and Karen Simonyan. Efficient video generation complex datasets.
arXiv preprint arXiv: (rXXiv1907.06571, 2019.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HILT, 2019.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Christoph Feichtenhofer. Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. CVPR, 2016.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019.
Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learning with odd-one-out networks. In CVPR, 2017.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680, 2014.
Kristen Grauman and Trevor Darrell. The pyramid match kernel: Discriminative classification with sets of image features. In ICCV, 2005.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchille Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2018
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitatie networks. In CVPR, 2018.
Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H Adelson. Learning visual groups from co-occurrences in space and time. In ICLR Workshop, 2015.
Dinesh Jayaraman and Kristen Grauman. Slow and steady feature analysis: Higher order temporal coherence in video. In CVPR, 2016.
Longlong Jing, Xiaodong Yang, Jingen Liu, and Yingli Tian. Self-supervised spatiotemporal feature learning via video rotation prediction. arXiv, 2018.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The Kinetics Human Action Video Dataset. arXiv preprint 1705 069500 2017.
Dahun Kim, Donghyeon Cho, and So Kweon. Self-supervised video representation learning with space-time cubic puzzles. In AAAI, 2019.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. In Neurips, 2018
Hildegard Kuchne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. HMDB: A Large Video Database for Human Motion Recognition. In ICCV, 2011.
10

---

Published as a conference paper at ICLR 2021
Vijay Kumar BG, Gustavo Carneiro, and Ian Reid. Learning local image descriptors with deep siamese and triplet convolutional networks by minimising global loss functions. In CVPR, 2016.
Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In CVPR, 2006.
Didier Le Gall. Mpeg: video compression standard for multimedia applications. Communications of the ACM, 34(4), 1991.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariane represttion... In CVPR, 2020.
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv, 2018.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV, 2018.
AJ Piergiovanni, Anelia Angelova, and Michael Ryoo. Evolving losses for unsupervised video representation learning. In CVPR, 2020.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 2015.
Michael S Ryoo, AJ Piergiovanni, and Juhana Kangaspunta. Assemblenett+: Assembling modality
representations via attention connections. 2020a.
Michael Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia Angelova. Assemblenet: Searching for multi-stream neural connectivity in video architectures. In ICLR, 2020b.
Rodrigo Santa Cruz, Basura Fernando, Anoop Cherian, and Stephen Gould. Deeppermnet: Visual permutation learning. In CVPR, 2017.
Zheng Shou, Xudong Lin, Yannis Kalantidis, Laura Sevilla-Lara, Marcus Rohrbach, Shih-Fu Chang, and Zhicheng Yan. Dmc-net: Generating discriminative motion cues for fast compressed video action recognition. In CVPR, 2019.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A Dataset of 101 Human Action Classes From Videos in The Wild. CRCV-TR-I2-01, 2012.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning Spa- tiotemporal Features with 3D Convolutional Networks. In ICCV, 2015.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A Closer Look at Spatiotemporal Convolutions for Action Recognition. In CVPR, 2018.
Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy, Tracking emerges by colorizing videos. In ECCV, 2018.
Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, and Wei Liu. Self-supervised apatio-temporal representation learning for videos by predicting motion and appearance dtatistics. In CVPR, 2019a.
Shiyao Wang. Hongchao Lu, and Zhidong Deng. Fast object detection in compressed video. In ICCV, 2019b.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
11

---

Published as a conference paper at ICLR 2021
Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle-consistency of time. In CVPR, 2019c.
Donglai Wei, Joseph J Lim, Andrew Zisserman, and William T Freeman. Learning and using the arrow of time. In CVPR, 2018
Chao- Yuan Wu, Manzil Zaheer, Hexiang Hu, R Manmatha, Alexander Smola, and Philipp Krähen- bühl. Compressed video action recognition. In CVPR, 2018.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotem- poral learning via video clip order prediction. In CVPR, 2019.
Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, and Hanli Wang. Real-time action recognition with enhanced motion vector cnns. In CVPR, 2016.
12

---

Published as a conference paper at ICLR 2021
A PMSP GROUND-TRUTH LABELS
We obtain the ground-truth labels for the pyramidal motion statistics prediction (PMSP) task directly from motion vectors provided in compressed videos. Algorithm I shows pseudo-code compute the labels at multiple spatio-temporal scales, r E {(s, t)ss E {2 x 2/. 3 x 3/, [4 x 4], E {1,3, 5)
Algorithm I: Self-supervision label for Pyramidal Motion Statistics Prediction
[1] Motion vectors (Mo.1.... Mr,k] } with T GOPs each having K - 1 motion vectors Generate
dx, dy by convolving motion vectors with Prewitt operator. Gz. Gy t f- 1; > Set temporal scale to Y < 0 Empty list for labels i = 02 n = 0f
wum_dr sum(de|n K [12
- K]
sum_dy +- sum(dy n * K
- - K))
magnitude + cartToPolar( (soooarrrumuaaa.. __mnn_dr__r sum_dy)
magnituder2xx1 + makeGrid (maggnituddee spatial = 2)
magnituder3xs] + makeGrid makeGriddmagniiude; spatial = 3)
magnitude(4x4] - makeGrid (magnitude, spatial = (gggaiiill....///ggiiiiieese))))
U[2x2] < arg max .24 (magnitude(exx) Ø/a33] t- arg gall (magnituude(sxx) M[1x]] f- arg maxxel1..... .11 (magniitudejxxa) Y < Y M2xx) ([xx)) V +
PMSP labels, Y, at multiple scales ((s, () [ x 2). [ x 3), x 4]]. (1,3,5)}
B PMSP LABEL VISUALIZATION
In Section 2.3 of the main paper, we motivated the design of our PMSP task by arguing that there is an implicit videographer bias captured in videos in-the-wild that naturally reflects visual saliency: Videos are purposely recorded to highlight important objects and their movements; therefore, regions with the highest energy of motion captured by our PMSP labels can provide clues learning video representation that captures salient moving objects. We acknowledged that this is, of course, a weak hypothesis (footnote in the main paper). However, in this section we provide some convincing empirical evidence.
Figures 12 are generated by visualizing regions with the highest energy of motion - i.e. the PMSP labels at multiple spatio-temporal scales. It needs a bit of explanation on how to read the figures as there is lot going on. Each figure is organized into three rows; each row shows results with multiple spatial regions at particular temporal scale, & E {1, 5). We color-code different spatial scales: Red boxes are in a [2 x 2] spatial scale, green boxes are in a [3 x 3] spatial scale, and blue boxes are in [4 x 4] spatial scale. Notice that all five I-frames in the top of results always contain identical regions. This is because, at the temporal scale t = (meaning, a temporal grid of size 1), compute the regions with the highest motion energy over the entire video (5 GOPs), hence the regions are identical across all I-frames in a video. Conversely, the bottom rows (f= 5. temporal grid of size 5) show the regions computed at each GOP, and hence the regions may differ by every I-frame (recall that each GOP contains a single I-frame). The middle rows (t = 3) show regions computed over GOPs, We overlay the regions the overlapping I-frames, e.g., the third I-frame at f = 3 contains regions computed at all three grid locations spanning over the I-frame indices [1,2,3], [2,3,4], and [3,4.5]. We order the figures an increasing level of complexity. and provide detailed analyses of the results in the captions of the figures.
1) in each set
The results in Figures 5 12 suggest that the most vibrant regions, as computed by our PMSP labels, tend to overlap with semantically important regions, e.g., the most salient moving objects. Intuitively, training our model to detect those regions encourages it to learn visual representations that capture salient objects and motion. This allows our model to learn discriminative visual representation in sepersupere manner.
13

---

Published as a conference paper at ICLR 2021
Figure 5: PMSP label visualization. The most vibrant regions, as highlighted by boxes of varying sizes indicating different spatial scales ([2 2). [3 3). [4 4)). all successfully capture the most salient moving object (the hand with a eye brush) and its motion (applying eye makeup).
J.
Figure 6: PMSP label visualization. At 1, the most vibrant region is the punching bag, correctly capturing the semantic category the video (BoxingPunchingBae As we get temporally finer, the regions start capture the boxer's movement, e,g., the elongated green boxes in the third and the fourth I-frames at 3.
14

---

Published as a conference paper at ICLR 2021
Figure 7: PMSP label visualization. This example highlights the benefit of formulating our task in a spatially pyramidal manner. Notice the boxes at different spatial scales capture different moving objects at varying sizes, i.e., green boxes (33 3) capture the referee, blue boxes ([4 4) capture the hands of the two sumo wrestlers, and red boxes ((2 2)) capture the wrestlers' legs.
Figure 8: PMSP label visualization. This example highlights the benefit of formulating our task in temporally pyramidal manner. While the vibrant regions at f 1 fail to capture the tumbling gymnast, the vibrant regions t= and = 5 successfully track her trajectory (especially the three middle I-frames t 3). In general. different videos will contain moving objects at different speeds; our pyramidal formulation allows us to capture wide variety of moving objects at different speeds via multiple temporal scales.
15

---

Published as a conference paper at ICLR 2021
J.
Figure 9: PMSP label visualization. Another example highlighting the benefit our pyramidal formulation. While some regions at = 1 miss the toddler in swing (see the first and the fifth I-frames the first row), at and f the boxes successfully track the toddler's 's trajectory.
Figure PMSP label visualization. This example contains dynamically moving object (a women with bow and an arrow) spanning across a large region in the frames, representing challenging situation. Notice the boxes at different spatial and temporal scales highlight different parts: at = both the green ([3 x 3)) and the red ([2 2)) boxes capture the bow, which exhibits the sharpest edge with motion (hence the highest motion energy in those spatial scales), while the blue boxes ( x 4)) capture the that takes out arrow. At and 5, the regions start capture different parts, the woman, which exhibits dynamic motion only towards the end of the video.
16

---

Published as a conference paper at ICLR 2021
Figure 11: PMSP label visualization. Another challenging example containing a small, dynamically moving object (Surfing). At t = 1, all the boxes focus on the crushing waves on the bottom right corner, which on average the most vibrant region in this video. Things are not much better at = 3; the surfer is still too fast to capture, and thus the boxes fail to capture the surfer and instead highlight crushing waves. the finest temporal scale 5, the boxes begin capture the surfer (see the first, second and the fifth frames on the bottom).
Till 4011
1200 20
1,00
Kur Pur
Ma. Por
T.
Figure 12: PMSP label visualization. This is a partial- example that shows boxes highlighting game statistics during horse race TV broadcast (see the boxes at 1). The game statistics constantly change frame-by stume-byfiie by-ffrme (e.g., time marks, rank, etc.), which caused those regions to exhibit average the highest energy motion for the entire duration of the video. Learning representations that strictly focus on those regions could lead to non-di non-discriminative information (many sports vidcos show similar game statisties on screen). Fortunately, the boxes begin highlight the horse riders finer temporal scale; see the green boxes ([3 3)) at t = 3 and = 5. This, again, suggests that the pyramidal formulation makes our PMSP task robust to variety of challenging real-world scenarios.
17

---

Published as a conference paper at ICLR 2021
C PMSP PREDICTION RESULTS
Figure 13 and Figure 14 show side-by-side comparisons of the ground-truth PMSP labels and our prediction results. We adopt the same visualization scheme used in the figures in Appendix B. Overall, our prediction results are mostly identical to the ground-truth regions. When our prediction deviates from the ground-truth the predicted regions still tend to capture important moving objects, e-g.. Figure (a) captures different parts of the punching bag, Figure [4 (b) captures different parts of the boxer, and Figure (d) captures different arms of the swimmer.
Ground-truth PMSP labels
Prediction results
(d)
Figure 13: PMSP prediction results. Overall, the predicted regions tend to highlight salient moving objects (although sometimes different from the ground-truth). (a): Buxinngunching (b): BaseballPitch, (c): Archery, (d): ThrowDiscus.
18

---

Published as a conference paper at ICLR 2021
Ground-truth PMSP labels
Prediction results
5=
(b)
U
(d) T
J.
Figure 14: PMSP prediction results. Overall, the predicted regions contain the salient moving objects (although sometimes different from the ground-truth). (a): Biking, (b): BoxingPunchingBag (c): CricketShot, (d): FrontCrawl.
19

---

Published as a conference paper at ICLR 2021
D VIDEO-TO-VIIE VIDEO RETRIEVAL
To demonstrate the quality of video representations learned using our self-supervised learning objectives (Section 2.3 the main paper), we evaluate our method the video-to-video retrieval task. To do this, we measure the cosine similarity between a query video and all the other video in a candidate set, and show the top-1 retrieved video. We compare ours two baselines: Rotat ion Ding et al. (2018) is our IMRNet pretrained using the 3D rotation prediction task (we used the IMRNeI + Rotation pretrained model reported in Table of our main paper), and ImageNet is ResNet1.52 fully-supervised with ImageNet IISVRC-2011 2 et al. (2015). We visualize the results Figures 18 and analyze the results in the caption of each figure.
20

---

Published as a conference paper at ICLR 2021
Queryy
Ours
CE [1
IzhanaSsur
Figure 15: Video-to-video retrieval result. Ours finds the most similar video to the query in terms of both the appearance (a gymnast) and the motion (handspring). The 3D Rotation baseline captures perhaps more similar appearance (a gymnast with the audience in the back) but less similar motion (horizontal bar jump vs. handspring). The ImageNet baseline fails to capture both appearance and motion (ImageNet does not contain category relevant to floor gymnastics).
Query
Ours
[1]tl] CF
Figure 16: Video-to-video retrieval result. Ours finds the most similar video to the query in terms of both the appearance (swim stadium) and motion (swimming). The ImageNet baseline does capture similar appearance (water), but fails to capture motion (swimming vs. surfing). The 3D Rotation baseline shows little semantic similarity the query video.
21

---

Published as a conference paper at ICLR 2021
Query
OurS
12 CRRNNNNN [1)101 CIE
Figure 17: Video-to-video retrieval result. Ours finds the most similar video to the query in terms of both the appearance (scene layout) and the motion (pitching). The ImageNet baseline does capture similar high-level semantics appearance-wise (baseball pitcher) but motion is relatively less similar (different camera angle, no catcher and no hitter). The 3D Rotation baseline shows little to semantic similarity to the query video.
Ours
[ ( Rolll
Figure 18: Video-tovvideo retrieval result. All three retrieval results fail to find vidcos that belong to the same semantic category as the query video (pole vault). However, ours finds a video that contains similar appearance (running track) and similar motion (running and jumping). The ImageNet baseline also captures similar appearance (javelin throw) but less similar motion (running at substantially slower pace). The 3D Rotation baseline shows little to semantic similarity to the query video.
22

---

Published as a conference paper at ICLR 2021
E ARCHITECTURE DETAILS
Table provides architecture details of our IMRNet. In our experiments, we used both 3D ResNet-18 and 3D ResNet-50 the backbone; we provide the details of both models in the table. We also include the details of our bidirectional dynamic connections, which include 3D convoltiioal/deconvlltinn layers that downsanplesuussamplss the computed features along the temporal dimension. We establish the connections after [convl, res2, res3, res4) layers, each with different numbers of channels.
Stage
Pathway
M/R Pathway
Output sizes 7 x S
clip
60 221*
I:5 x 22/²
layer
stride 12, 1
stride 2, 1
M/R: 25 224*
1 x 7",64 64 stride 1,2
5 x 7",8
I:5 X 112"
conv1
stride 1,2?
M/R: 25 112²
1 x 3", maz
5 x 3", mar
I:5 x 56*
pool
stride 1, 2
stride 1,2
M/R: 25 562
(3D ResNet-18)
(3D ResNet-18)
x 3 64
3 32,
x 3 64
x2
x
x2
I: X 562
re82
(3D ResNet-50)
(3D ResNet-50)
M/R: x 562
x 64
x
1 3x 32,64 64
x3
1x32,4
x3
1x 13,256
1x12,16
(3D ResNet-18)
ResNet
1 3 128]
3
1 3x 128
x2
1x
x2
I: X 28²
re83
(3D ResNet-50)
(3D ResNet-00
M/R: x 28²
1 12, 128
3
1 x 3x3 x4
1x
x4
1 12,
1x
(3D ResNet-18)
(3D ResNet-
3 3 256
3 16
1 3", 256
x2
x
x2
I: x 14"
TC84
(3D ResNet-50)
(3D ResNet-50)
M/R: 14
3 12, 256
3 x 167
1 x 32, 256
x6
1x
x6
1 12, 1024
x
(3D ResNet-
(3D ResNet-18)
3 32,33255 512
x 32,32
1 32 512
x 32,32
x2
1:3 1:3x7 72
ress
(3D ResNet-50)
(3D ResNet-50)
M/R: 25 72
3 12, 512
3 x 12, 32
3
x3
32, ,332
x3
1 12, 2048
1 x 12, 128
Stage
COTOV1
re82
ress
TE:4
1 5", 8
x 5",8 stride 5, 12
1 x 5 16 stride 5, 12
1 x 5",32 stride 5, 1
I M/R
stride 5, 1
5 7",
x 72,
5 x 7
x 7 16
M/R
stride 1
stride 5, 1"
stride 5, 1
stride 5,
Table IMRNet architecture details. We show two versions of IMRNet with different backbones: 3D ResNet-18 and 3D ResNet-50, We denote the input dimensions by (temporal size, spatial size" kernels by (temporal size, spatial size?, channel size) and strides by /temporal stride, spatial stride"
23


Reviews:
[{"review_id": "iVW-Upf470R", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors study a new problem – self-supervised learning for compressed video understanding. It can dramatically save the compute and storage requirements for action recognition. The experiment results on several datasets demonstrate that the proposed approach can achieve the best results over the baselines.\n\nStrength:\n1. Self-supervised compressed video understanding is an exciting topic. It can save computation and storage for online video understanding.\n2. The two pretext tasks are efficient and straightforward for compressed video understanding.\n \nQuestions:\n\n1. Figure 2 is misleading, T_I is k times smaller than T_M,3D Deconv should be on the top and 3D Conv should be on the bottom; the image cubes are also incorrect -- they are supposed to be placed in H-W-C plane. Please re-organize the text and the figures to make the main text clear.\n2. The authors design two pretext tasks according to the heuristic: pyramidal motion statistics prediction and correspondence type prediction. I want to say the authors have done a fair evaluation, but the approach's novelty is limited. I would like to see more discussion about the intuition of the proposed method.\n3. Why not compare with the contrastive learning-based approaches for self-supervised learning. We can still get the positive pairs from the same video, though it is difficult to augment the  videos give the compressed videos.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WjWSlMJEuDP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach to self-supervised learning from videos. The approach takes advantage of compressed videos, using the encoded residuals and motion vectors within the video codec. Using encoded videos has been shown to reduce computation time required by decoding videos. Previous works have explored compressed videos for supervised recognition, showing the potential, while this paper introduces a way to leverage compressed videos for self-supervised learning.\n\nThe paper is well written and easy to follow. The proposed IMRNet shows benefit over previous supervised compressed video approaches. The paper proposes two self-supervised tasks related to compressed videos: one to predict the spatial locations with movement and one to predict the temporal matching of the inputs (Correspondence Type Prediction).\n\nThe Correspondence Type Prediction tasks are fairly standard, and have been explored by many previous works (which are cited). The main difference being the use of the compressed video features instead of rgb frame or audio.\n\nAn interesting observation is that the two tasks when combined don't seem to be any better, roughly 0.1%-0.3%, which is likely within noise on smaller datasets like HMDB and UCF101.\n\nThe comparisons in Table 3 are missing comparisons to state-of-the-art approaches. For example,\n- \"Cooperative learning of audio and video models from self-supervised synchronization\", NeurIPS'18 \n- \"Audio-visual scene analysis with self-supervised multisensory features\", ECCV'18\n- \"Evolving Losses for Unsupervised Video Representation Learning\", CVPR'20\n\nall outperform this approach. While those approaches use audio features and are not specifically focused on compressed videos, there's nothing about them making them incompatible with compressed videos using say only I-frames. I think Table 3 could be improved by comparing to existing approaches for non-compressed videos as well as using those approaches on compressed videos. This would help show the benefit of the proposed tasks.\n\nThe paper has some missing details. One of the claims is that compressed videos is faster, which is shown in Table 2. But there are some missing details, for example, the number of training epochs is not reported (only that 30 are used for warmup). How many does this use? How much faster is this method than non-compressed method?\n\nThere are also some other related works, e.g.\n\"AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures\", ICLR'20\nproposes a similar idea to the bidirectional connections and\n\"AssembleNet++: Assembling Modality Representations via Attention Connections\", ECCV'20\nproposes ideas similar to the multimodal gated attention.\n\n\nOverall, the paper is interesting. I think some of the experiments could be strengthen to better compare to existing self-supervised approaches.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7R3cLWxnCPp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is not bad in the sense that:\n1. important problem -- how to leverage compressed video such a natural and efficient format is very important while under-explored by the community\n2. extensive experiments -- clear gains on standard benchmarks demonstrate the proposed method's gains. good ablation studies in like table 3 demonstrate the effectiveness of each proposed pretext task. \n3. the paper explored the how to better fuse multiple streams and two pretext ssl tasks i.e. PMSP+CTP, which all turn out to be successful\n\nYet, the way fusing multiple streams using techniques like multi-modal attention and the pretext tasks i.e PMSP and CTP -- they look fair while do not surprise me. It is quite natural to think of these techniques to improve the current models. For example, correspondence has been used by UCB researchers to do SSL in video yet for other task. \n\nBut I do acknowledge it takes significant work to validate these findings and conduct such comprehensive explorations and comparisons. I do believe these findings are valuable to the broad video community and thus will have good impact. Thus, I lean to accept this paper. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers liked the self-supervised learning of compressed videos, noting that it is an "exciting topic" and an "important problem", although they found the proposed methods (PMSP andCTP) less exciting. Reviewers were satisfied with the execution and the extensive experimental studies. AC felt the community may benefit from the paper's intuitive integration of self-supervised learning and the compressed video's signals (I and P frames, residuals, motion vectors, etc). 
Decision: Accept (Poster)