VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning

Video understanding relies on perceiving the overall global content and modeling its internal connections ( e.g. , causality , movement , and spatio-temporal correspondence ) . To learn these interactions , we apply a mask-then-predict pre-training task on the discretized video tokens generated via VQ-VAE . Unlike language , where the text tokens are more independent , neighboring video tokens typically have strong correlations ( e.g. , consecutive video frames usually look similar ) , and hence uniformly masking individual tokens will make the task too trivial to learn useful representations . To deal with this issue , we propose a block masking strategy where we mask neighboring video tokens in both spatial and temporal domains . We also add a contrastive learning objective to further capture the global content by predicting whether the video clips are sampled from the same video . We pre-train our model on uncurated videos and show that our pre-trained model can reach state-of-the-art results on several video understanding datasets ( e.g. , SSV2 , Diving48 ) . Lastly , we provide detailed analyses of the model scalability and pre-training method design . 1 INTRODUCTION . In recent years , state-of-the-art self-supervised methods have been exploring different directions for pre-training images and text representations , with Contrastive Learning ( CL ) providing strong results for vision representation learning ( Oord et al. , 2018 ; Chen et al. , 2020b ; He et al. , 2020 ; Chen et al. , 2020c ; Tian et al. , 2020 ) , and Language Modeling ( LM ) becoming the de-facto standard in language pre-training ( Devlin et al. , 2019 ; Liu et al. , 2019 ; Yang et al. , 2019 ; Lan et al. , 2019 ) . Both approaches are quite different from each other . A contrastive objective compares positive/negative examples at a coarse/sample level , focusing on global-content ( e.g. , for image classification ) while a token modeling objective predict missing tokens from context at a much finer/sub-sample level to model sequential and short range interactions between tokens ( e.g . in text generation tasks ) . Interestingly , video understanding naturally combines both types of requirements . 2D processing along the spatial dimensions of the video bears similarity to image processing , while 1D processing along the temporal dimension often involves modeling sequential events and short range coherence . Hence , in this work , we propose to combine both text and image representation learning approaches for improved video pre-training , taking advantage of recent advances in self-supervised methods of both fields . We name our method as VIMPAC : VIdeo pre-training via Masked token Prediction And Contrastive learning . From language research , we adopt a ‘ masked language model ’ pretraining objective ( Devlin et al. , 2019 ) where a model is trained to reconstruct local masked regions in videos . From the computer vision world , we borrow a contrastive learning objective , specifically the InfoNCE ( Oord et al. , 2018 ) objective is applied on positive/negative video samples . While the masked language model objective encourages models to learn low-level semantics and sequential interaction , the contrastive loss provide a supervision for the model to learn more global and separable representations that are useful for many downstream tasks ( e.g. , action classification ( Soomro et al. , 2012b ; Carreira & Zisserman , 2017 ) ) . The two objectives provide complementary signals for training : while short range correlations can be predominantly modeled from the training signal of the mask-and-predict task , the contrastive learning objective can provide signals on a more coarse-grained global-context and semantic level . However , unlike language which is composed of discrete tokens from a compact vocabulary , videos are typically represented as RGB pixels in an almost continuous , high dimensional vector space . Naively masking pixels in videos induces a prohibitive computation cost while also tending to overemphasize local details . To overcome these issues , we first tokenize input videos using the latent codes of a pretrained Vector Quantized-Variational Auto-Encoder ( VQ-VAE ) ( van den Oord et al. , 2017 ; Ramesh et al. , 2021 ) to encode them in smaller quantized representations on which a reconstruction model can then be trained with a masked token modeling objective . In practice , we also discovered that models trained with a uniform random token masking strategy can fail to learn meaningful and useful visual representations as neighboring pixels may contain similar and correlated content ( in particular along the temporal dimension ) , making the task of predicting a randomly masked token from its visible neighbors trivial . We therefore also introduce a block-masking scheme that simultaneously masking video tokens in a 3D spatio-temporal block . Reconstructing such an extended spatio-temporal cube requires performing long-range predictions , forcing the models to learn a more complex set of relations between the video tokens , resulting in better visual representations . Our contrastive learning approaches also departs from previous work in several aspects . First , since we apply the contrastive objective on token-discretized video samples and in combination with the token modeling loss , we observe strong performance without requiring the usual extensive set of data augmentations ( Chen et al. , 2020b ; c ; Qian et al. , 2021 ; Feichtenhofer et al. , 2021 ) . Second , we are able to leverage positive clip pairs that are temporally distant from each other ( can be as far as 400 seconds away ) , while previous work favors using positives within a shorter range ( maximum 36 seconds for uncurated videos in Feichtenhofer et al . ( 2021 ) or 10 seconds in Qian et al . ( 2021 ) ) . We evaluate the performances of our method VIMPAC on several video understanding datasets , including two temporally-heavy tasks , SSV2 and Diving48 on which it achieves state-of-the-art results with regard to both self-supervised and supervised pre-training works , and a set of more spatially-heavy datasets , UCF101 , HMDB51 , and Kinetics-400 , on which it also achieves competitive results . Overall , taking advantage of VQ-VAE discretized video tokens , we present a method for self-supervised learning of video representations that combines two general streams of research in self-supervision : masked language modeling and contrastive learning . Our contribution is 3-folds : ( i ) We apply the mask-then-predict task to video understanding and introduce the use of block masking . ( ii ) We propose a contrastive learning method which is able to achieve strong performance without spatial data augmentation . ( iii ) We empirically show that this method achieves strong performance on several video classification datasets , especially on temporally-heavy datasets , SSV2 and Diving48 , where it sets new state-of-the-art results . We also present comprehensive ablation studies to analyze the various aspects of our proposed approach . 2 RELATED WORK . Unsupervised representation learning , with the promise of learning from large-scale unlabeled data , has drawn increasing attention in recent years , in both computer vision and natural language processing ( NLP ) communities . Most mainstream self-supervised methods can be categorized into three general directions : generative , denoising , and discriminative ( Chen et al. , 2020b ; Grill et al. , 2020 ; Doersch et al. , 2015 ) . Generative and denoising methods seek to generate or reconstruct corrupted text/image/video tokens according to their empirical distributions . In generative and auto-regressive methods , next tokens are predicted given a causal context ( Chen et al. , 2020a ; van den Oord et al. , 2016 ) while denoising methods seek to reconstruct corrupted or masked tokens given an extended context ( Devlin et al. , 2019 ; Raffel et al. , 2019 ) . For text , since the tokens ( words or sub-words ( Sennrich et al. , 2016 ; Wu et al. , 2016 ) ) are discrete and has relatively high entropy rate , language modeling has became the de-facto approach for pre-training models for most natural language tasks ( Ruder et al. , 2019 ) . In the case of images , generative approaches often operate on pixel space ( Bertalmio et al. , 2001 ; Yu et al. , 2018 ; Kim et al. , 2019 ; Chen et al. , 2020a ; van den Oord et al. , 2016 ) , which can be extremely expensive for larger input size like videos and has hence limited the widespread adoption of these methods . Recently , discretizing images and videos with discrete variational auto-encoders ( VQ-VAE ) , has been explored in compression and generative setups ( van den Oord et al. , 2017 ; Razavi et al. , 2019 ; Walker et al. , 2021 ; Ramesh et al. , 2021 ; Yan et al. , 2021 ) , and Sun et al . ( 2019 ) tackles the video-language problem by frame-level quantization . Such approaches avoid modeling pixel-level details and have enabled the use of generative models for images and videos ( Walker et al. , 2021 ; Ramesh et al. , 2021 ) . Differing from these works , our framework investigates the use of such quantized representations in a denoising/reconstruction setup rather than generative , which has been shown in the NLP community to learn better representations ( Raffel et al. , 2019 ; Devlin et al. , 2019 ) . Moreover , beyond simply applying MLM to the video tokens , we propose a block masking strategy to reduce the strong local correlation in neighboring video tokens . This 3D block masking strategy is inspired from recent span-masking schemes ( Raffel et al. , 2019 ; Joshi et al. , 2020 ) for language modeling . The concurrent work ( Bao et al. , 2021 ) explores using the VQ-VAE tokens as labels for masked patches in the image domain . The other direction of research , which our framework combines , is discriminative methods which start from the hypothesis that learning to reconstruct local details is not necessary for learning good visual representations . In some of these approaches , an objective is constructed around hand-crafted heuristics tasks like spatial arrangement , color , playback speed or frame order predictions ( Doersch et al. , 2015 ; Zhang et al. , 2016 ; Gidaris et al. , 2018 ; Fernando et al. , 2017 ; Lee et al. , 2017 ; Wei et al. , 2018 ; Epstein et al. , 2020 ; Benaim et al. , 2020 ; Sun et al. , 2021 ) . Another line of discriminative approaches is contrastive learning which aims at training a model to be able to recognize different views ( e.g. , different augmentation of images or different temporal samples of videos ) of the same image or video , as a way to learn general representations ( Chen et al. , 2020b ; He et al. , 2020 ; Chen et al. , 2020c ; Grill et al. , 2020 ; Caron et al. , 2020 ; Feichtenhofer et al. , 2021 ) . This direction of research is reminiscent of sentence-order prediction tasks introduced in NLP ( Devlin et al. , 2019 ; Lan et al. , 2019 ) with the goal of predicting whether two text sequences should be juxtaposed or not , an approach challenged in more recent literature ( Liu et al. , 2019 ; Lan et al. , 2019 ) . In the present work , inspired by visual representation rather than text representation learning literature , we adapt the contrastive learning approach to video by training a model to differentiate pairs of clips from a single video from pairs of clips from disparate videos . Another thread of research focuses on the way to adapt transformer models to video tasks ( Bertasius et al. , 2021 ; Arnab et al. , 2021 ) . Recent works ( Fan et al. , 2021 ; Liu et al. , 2021 ) extend it with hierarchical modeling and efficient attention patterns . These methods focus on the modeling and achieve good results with supervised pre-training . Besides the difference in focus ( modeling vs. pretraining ) to our paper , the hierarchical design is also not directly applicable to our mask prediction tasks .

Summary: This paper proposes a new video pretraining method by combining masked token prediction and contrastive learning. Off-she-shelf VQ-VAE is used in this paper for discrete video tokens generation. In order to make the masked token prediction more effective, the authors proposed a block mask strategy where spatial-temporal neighboring video tokens are masked together. Experimental results are shown to validate its effectiveness.