Self-Supervised Learning of Compressed Video Representations

Self-supervised learning of video representations has received great attention . Existing methods typically require frames to be decoded before being processed , which increases compute and storage requirements and ultimately hinders largescale training . In this work , we propose an efficient self-supervised approach to learn video representations by eliminating the expensive decoding step . We use a three-stream video architecture that encodes I-frames and P-frames of a compressed video . Unlike existing approaches that encode I-frames and P-frames individually , we propose to jointly encode them by establishing bidirectional dynamic connections across streams . To enable self-supervised learning , we propose two pretext tasks that leverage the multimodal nature ( RGB , motion vector , residuals ) and the internal GOP structure of compressed videos . The first task asks our network to predict zeroth-order motion statistics in a spatio-temporal pyramid ; the second task asks correspondence types between I-frames and P-frames after applying temporal transformations . We show that our approach achieves competitive performance on compressed video recognition both in supervised and self-supervised regimes . 1 INTRODUCTION . There has been significant progress on self-supervised learning of video representations . It learns from unlabeled videos by exploiting their underlying structures and statistics as free supervision signals , which allows us to leverage large amounts of videos available online . Unfortunately , training video models is notoriously difficult to scale . Typically , practitioners have to make trade-offs between compute ( decode frames and store them as JPEG images for faster data loading , but at the cost of large storage ) and storage ( decode frames on-the-fly at the cost of high computational requirements ) . Therefore , large-batch training of video models is difficult without high-end compute clusters . Although these issues are generally applicable to any video-based scenarios , they are particularly problematic for self-supervised learning because large-scale training is one key ingredient ( Brock et al. , 2019 ; Clark et al. , 2019 ; Devlin et al. , 2019 ) but that is exactly where these issues are aggravated . Recently , several approaches demonstrated benefits of compressed video recognition ( Zhang et al. , 2016 ; Wu et al. , 2018 ; Shou et al. , 2019 ; Wang et al. , 2019b ) . Without ever needing to decode frames , these approaches can alleviate compute and storage requirements , e.g. , resulting in 3 to 10 times faster solutions than traditional video CNNs at a minimal loss on accuracy ( Wu et al. , 2018 ; Wang et al. , 2019b ) . Also , motion vectors embedded in compressed videos provide a free alternative to optical flow which is compute-intensive ; leveraging this has been shown to be two orders of magnitude faster than optical flow-based approaches ( Shou et al. , 2019 ) . However , all the previous work on compressed video has focused on supervised learning and there has been no study that shows the potential of compressed videos in self-supervised learning ; this is the focus of our work . ? Equal Contribution In this work , we propose a self-supervised approach to learning video representations directly in the compressed video format . We exploit two inherent characteristics of compressed videos : First , video compression packs a sequence of images into several Group of Pictures ( GOP ) . Intuitively , the GOP structure provides atomic representation of motion ; each GOP contains images with just enough scene changes so a video codec can compress them with minimal information loss . Because of this atomic property , we enjoy less spurious , more consistent motion information at the GOP-level than at the frame-level . Second , compressed videos naturally provide multimodal representation ( i.e . RGB frames , motion vectors , and residuals ) that we can leverage for multimodal correspondence learning . Based on these , we propose two novel pretext task ( see Fig . 1 ) : The first task asks our model to predict zeroth-order motion statistics ( e.g.where is the most dynamic region ) in a pyramidal spatio-temporal grid structure . The second involves predicting correspondence types between I-frames and P-frames after temporal transformation . Solving our tasks require implicitly locating the most salient moving objects and matching their appearance-motion correspondences between I-frames and P-frames ; this encourages our model to learn discriminative representation of compressed videos . A compressed video contains three streams of multimodal information – i.e . RGB images , motion vectors , and residuals – with a dependency structure between an I-frame stream and the two P-frame streams punctuated by GOP boundaries . We design our architecture to encode this dependency structure ; it contains one CNN encoding I-frames and two other CNNs encoding motion vectors and residuals in P-frames , respectively . Unlike existing approaches that encode I-frames and P-frames individually , we propose to jointly encode them to fully exploit the underlying structure of compressed videos . To this end , we use a three-stream CNN architecture and establish bidirectional dynamic connections going from each of the two P-frame streams into the I-frame stream , and vice versa , and put these connections layer-wise to learn the correlations between them at multiple spatial/temporal scales ( see Fig . 1 ) . These connections allow our model to fully leverage the internal GOP structure of compressed videos and effectively capture atomic representation of motion . In summary , our main contributions are two-fold : ( 1 ) We propose a three-stream architecture for compressed videos with bidirectional dynamic connections to fully exploit the internal structure of compressed videos . ( 2 ) We propose novel pretext tasks to learn from compressed videos in a self-supervised manner . We demonstrate our approach by pretraining the model on Kinetics-400 ( Kay et al. , 2017 ) and finetuning it on UCF-101 ( Soomro et al. , 2012 ) , HMDB-51 ( Kuehne et al. , 2011 ) . Our model achieves new state-of-the-art performance in compressed video classification tasks in both supervised and self-supervised regimes , while maintaining a similar computational efficiency as existing compressed video recognition approaches ( Wu et al. , 2018 ; Shou et al. , 2019 ) . 2 APPROACH . We use videos compressed according to the MPEG-4 Part 2 specifications ( Le Gall , 1991 ) as our input , following the previous work ( Wu et al. , 2018 ; Shou et al. , 2019 ; Wang et al. , 2019b ) . This compression format encodes an RGB image sequence as a series of GOPs ( Group of Pictures ) where each GOP starts with one I-frame followed by a variable number of P-frames . An I-frame stores RGB values of a complete image and can be decoded on its own . A P-frame holds only the changes from the previous reference frame using motion vectors and residuals . The motion vectors store 2D displacements of the most similar patches between the reference and the target frames , and the residuals store pixel-wise differences to correct motion compensation errors . We use all the three modalities contained in compressed videos as our input . Formally , our input is T GOPs , G0 , · · · , GT−1 , where each Gt contains one I-frame It ∈ RH×W×3 followed by K − 1 pairs of motion vectors Mt , k ∈ RH×W×2 and residuals Rt , k ∈ RH×W×3 , k ∈ [ 1 , K ) . For efficiency and simplicity , we assume an identical GOP size K for all t ∈ [ 0 , T ) . 2.1 IMR NETWORK FOR COMPRESSED VIDEOS . Our model consists of three CNNs , each with 3D convolutional kernels modeling spatio-temporal dynamics within each input stream { It } , { Mt , k } , { Rt , k } , t ∈ [ 0 , T ) , k ∈ [ 0 , K ) ; we denote these sub-networks by I-network fI , M-network fM , and R-network fR , respectively , and call our model IMR Network ( IMRNet ) . We account for the difference in the amount of information between I-frames and P-frames by adjusting the capacity of networks accordingly . Specifically , following ( Wu et al. , 2018 ) , we make the capacity of fI larger than fM and fR by setting the number of channels in each layer of fI to be γ times higher than those of fM and fR ( we set γ = 64 ) . Existing models for compressed videos typically perform late fusion ( Wu et al. , 2018 ; Shou et al. , 2019 ) , i.e. , they combine embeddings of I-frames and P-frames only after encoding each stream . However , we find that it is critical to allow our sub-networks to share information as they encode their respective input streams . To this end , we establish layer-wise lateral connections between fI & fM and between fI & fR . Bidirectional dynamic connections . Lateral connections have been used to combine information from different streams , e.g. , RGB images and optical flow images ( Feichtenhofer et al. , 2016 ) , and RGB images sampled at different frame rates ( Feichtenhofer et al. , 2019 ) . In this work , we use it to combine information from I-frames and P-frames . Our approach is different from previous work in two key aspects : ( 1 ) We establish bidirectional connections between streams , instead of unidirectional connections as was typically done in the past ( Feichtenhofer et al. , 2016 ; 2019 ) , so that information sharing is symmetrical between streams . ( 2 ) We incorporate multimodal gated attention to dynamically adjust the connections based on multimodal ( I-frame and P-frames ) information . We call our approach bidirectional dynamic connections to highlight these two aspects and differentiate ours from previous work , e.g. , SlowFast networks ( Feichtenhofer et al. , 2019 ) establish unidirectional lateral connections and the connections are static regardless of the content from the other stream . We combine embeddings from different sub-networks via channel-wise concatenation , which requires embeddings to match their spatio-temporal dimensions . However , fI processes κ times less frames than fM and fR , producing embeddings that are κ times smaller in the temporal dimension . Therefore , we transform the embeddings with time-strided 3D ( de- ) convolution with ( κ× 1× 1 ) kernels , C/8 channels , and ( κ , 1 , 1 ) temporal stride : We use convolution for fI → fM/fR to decrease the time dimension and deconvolution for fM/fR → fI to increase it . Note that simply using the ( de- ) conv layers will perform static transformation regardless of what is provided from the other sub-network , similar to ( Feichtenhofer et al. , 2019 ) . However , we find it critical to make the transformations aware of information from both sub-networks so that the networks can dynamically adjust the connections and selectively share only the most relevant information from each sub-network . To achieve this , we dynamically modulate ( de- ) conv layer outputs using multimodal-gated attention weights . Let xI ∈ RTI×W×H×CI and xM ∈ RTM×W×H×CM be the embeddings from fI and fM , respectively . We max-pool xI and xM and concatenate them to obtain multimodal embedding z ∈ RCZ with CZ = CI + CM . We define multimodal gate functions that take as input z and generate attention weights aI ∈ RCI/8 and aM ∈ RCM/8 as aI = σ ( W3h+ b3 ) , aM = σ ( W4h+ b4 ) , h = ζ ( W2ζ ( W1z+ b1 ) + b2 ) ( 1 ) where σ is a sigmoid function , ζ is a Leaky ReLU function , and W1 , W2 ∈ RCZ×CZ , b1 , b2 ∈ RCZ , W3 ∈ RCI/8×CZ , b3 ∈ RcI/8 , W4 ∈ RCM/8×CZ , b4 ∈ RCM/8 are weight parameters . Next , we use these attention weights to modulate the ( de- ) conv output embeddings , vI→M = aM ⊗ 3d_conv ( xI ) , vM→I = aI ⊗ 3d_deconv ( xM ) ( 2 ) where ⊗ is channel-wise multiplication . We repeat the same process for fI & fR to obtain vI→R and vR→I , and combine them with the feature embeddings via channel-wise concatenation , x̂I = [ xI ; vM→I ; vR→I ] , x̂M = [ xM ; vI→M ] , x̂R = [ xR ; vI→R ] ( 3 ) Each of these is fed into the next layer in the corresponding sub-network . We establish these lateral connections across multiple layers of our network . To obtain the final embedding , we apply average pooling on the output from the final layer of each sub-network and concatenate them channel-wise . Note that the design of IMRNet is orthogonal to the design of video CNNs ; while we adapt 3DResNet ( He et al. , 2016 ) as the backbone in our experiments , we can use any of existing CNN architectures as the backbone , e.g. , C3D ( Tran et al. , 2015 ) , I3D ( Carreira & Zisserman , 2017 ) , R ( 2+1 ) D ( Tran et al. , 2018 ) . What is essential , however , is that ( i ) there are three sub-networks , each modeling one of the three input streams , and ( ii ) information from different networks are combined via bidirectional dynamic connections as they are encoded .

Summary: This paper proposes an approach to self-supervised learning from videos. The approach takes advantage of compressed videos, using the encoded residuals and motion vectors within the video codec. Using encoded videos has been shown to reduce computation time required by decoding videos. Previous works have explored compressed videos for supervised recognition, showing the potential, while this paper introduces a way to leverage compressed videos for self-supervised learning.