VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning

Video understanding relies on perceiving the overall global content and modeling its internal connections ( e.g. , causality , movement , and spatio-temporal correspondence ) . To learn these interactions , we apply a mask-then-predict pre-training task on the discretized video tokens generated via VQ-VAE . Unlike language , where the text tokens are more independent , neighboring video tokens typically have strong correlations ( e.g. , consecutive video frames usually look similar ) , and hence uniformly masking individual tokens will make the task too trivial to learn useful representations . To deal with this issue , we propose a block masking strategy where we mask neighboring video tokens in both spatial and temporal domains . We also add a contrastive learning objective to further capture the global content by predicting whether the video clips are sampled from the same video . We pre-train our model on uncurated videos and show that our pre-trained model can reach state-of-the-art results on several video understanding datasets ( e.g. , SSV2 , Diving48 ) . Lastly , we provide detailed analyses of the model scalability and pre-training method design . 1 INTRODUCTION . In recent years , state-of-the-art self-supervised methods have been exploring different directions for pre-training images and text representations , with Contrastive Learning ( CL ) providing strong results for vision representation learning ( Oord et al. , 2018 ; Chen et al. , 2020b ; He et al. , 2020 ; Chen et al. , 2020c ; Tian et al. , 2020 ) , and Language Modeling ( LM ) becoming the de-facto standard in language pre-training ( Devlin et al. , 2019 ; Liu et al. , 2019 ; Yang et al. , 2019 ; Lan et al. , 2019 ) . Both approaches are quite different from each other . A contrastive objective compares positive/negative examples at a coarse/sample level , focusing on global-content ( e.g. , for image classification ) while a token modeling objective predict missing tokens from context at a much finer/sub-sample level to model sequential and short range interactions between tokens ( e.g . in text generation tasks ) . Interestingly , video understanding naturally combines both types of requirements . 2D processing along the spatial dimensions of the video bears similarity to image processing , while 1D processing along the temporal dimension often involves modeling sequential events and short range coherence . Hence , in this work , we propose to combine both text and image representation learning approaches for improved video pre-training , taking advantage of recent advances in self-supervised methods of both fields . We name our method as VIMPAC : VIdeo pre-training via Masked token Prediction And Contrastive learning . From language research , we adopt a ‘ masked language model ’ pretraining objective ( Devlin et al. , 2019 ) where a model is trained to reconstruct local masked regions in videos . From the computer vision world , we borrow a contrastive learning objective , specifically the InfoNCE ( Oord et al. , 2018 ) objective is applied on positive/negative video samples . While the masked language model objective encourages models to learn low-level semantics and sequential interaction , the contrastive loss provide a supervision for the model to learn more global and separable representations that are useful for many downstream tasks ( e.g. , action classification ( Soomro et al. , 2012b ; Carreira & Zisserman , 2017 ) ) . The two objectives provide complementary signals for training : while short range correlations can be predominantly modeled from the training signal of the mask-and-predict task , the contrastive learning objective can provide signals on a more coarse-grained global-context and semantic level . However , unlike language which is composed of discrete tokens from a compact vocabulary , videos are typically represented as RGB pixels in an almost continuous , high dimensional vector space . Naively masking pixels in videos induces a prohibitive computation cost while also tending to overemphasize local details . To overcome these issues , we first tokenize input videos using the latent codes of a pretrained Vector Quantized-Variational Auto-Encoder ( VQ-VAE ) ( van den Oord et al. , 2017 ; Ramesh et al. , 2021 ) to encode them in smaller quantized representations on which a reconstruction model can then be trained with a masked token modeling objective . In practice , we also discovered that models trained with a uniform random token masking strategy can fail to learn meaningful and useful visual representations as neighboring pixels may contain similar and correlated content ( in particular along the temporal dimension ) , making the task of predicting a randomly masked token from its visible neighbors trivial . We therefore also introduce a block-masking scheme that simultaneously masking video tokens in a 3D spatio-temporal block . Reconstructing such an extended spatio-temporal cube requires performing long-range predictions , forcing the models to learn a more complex set of relations between the video tokens , resulting in better visual representations . Our contrastive learning approaches also departs from previous work in several aspects . First , since we apply the contrastive objective on token-discretized video samples and in combination with the token modeling loss , we observe strong performance without requiring the usual extensive set of data augmentations ( Chen et al. , 2020b ; c ; Qian et al. , 2021 ; Feichtenhofer et al. , 2021 ) . Second , we are able to leverage positive clip pairs that are temporally distant from each other ( can be as far as 400 seconds away ) , while previous work favors using positives within a shorter range ( maximum 36 seconds for uncurated videos in Feichtenhofer et al . ( 2021 ) or 10 seconds in Qian et al . ( 2021 ) ) . We evaluate the performances of our method VIMPAC on several video understanding datasets , including two temporally-heavy tasks , SSV2 and Diving48 on which it achieves state-of-the-art results with regard to both self-supervised and supervised pre-training works , and a set of more spatially-heavy datasets , UCF101 , HMDB51 , and Kinetics-400 , on which it also achieves competitive results . Overall , taking advantage of VQ-VAE discretized video tokens , we present a method for self-supervised learning of video representations that combines two general streams of research in self-supervision : masked language modeling and contrastive learning . Our contribution is 3-folds : ( i ) We apply the mask-then-predict task to video understanding and introduce the use of block masking . ( ii ) We propose a contrastive learning method which is able to achieve strong performance without spatial data augmentation . ( iii ) We empirically show that this method achieves strong performance on several video classification datasets , especially on temporally-heavy datasets , SSV2 and Diving48 , where it sets new state-of-the-art results . We also present comprehensive ablation studies to analyze the various aspects of our proposed approach . 2 RELATED WORK . Unsupervised representation learning , with the promise of learning from large-scale unlabeled data , has drawn increasing attention in recent years , in both computer vision and natural language processing ( NLP ) communities . Most mainstream self-supervised methods can be categorized into three general directions : generative , denoising , and discriminative ( Chen et al. , 2020b ; Grill et al. , 2020 ; Doersch et al. , 2015 ) . Generative and denoising methods seek to generate or reconstruct corrupted text/image/video tokens according to their empirical distributions . In generative and auto-regressive methods , next tokens are predicted given a causal context ( Chen et al. , 2020a ; van den Oord et al. , 2016 ) while denoising methods seek to reconstruct corrupted or masked tokens given an extended context ( Devlin et al. , 2019 ; Raffel et al. , 2019 ) . For text , since the tokens ( words or sub-words ( Sennrich et al. , 2016 ; Wu et al. , 2016 ) ) are discrete and has relatively high entropy rate , language modeling has became the de-facto approach for pre-training models for most natural language tasks ( Ruder et al. , 2019 ) . In the case of images , generative approaches often operate on pixel space ( Bertalmio et al. , 2001 ; Yu et al. , 2018 ; Kim et al. , 2019 ; Chen et al. , 2020a ; van den Oord et al. , 2016 ) , which can be extremely expensive for larger input size like videos and has hence limited the widespread adoption of these methods . Recently , discretizing images and videos with discrete variational auto-encoders ( VQ-VAE ) , has been explored in compression and generative setups ( van den Oord et al. , 2017 ; Razavi et al. , 2019 ; Walker et al. , 2021 ; Ramesh et al. , 2021 ; Yan et al. , 2021 ) , and Sun et al . ( 2019 ) tackles the video-language problem by frame-level quantization . Such approaches avoid modeling pixel-level details and have enabled the use of generative models for images and videos ( Walker et al. , 2021 ; Ramesh et al. , 2021 ) . Differing from these works , our framework investigates the use of such quantized representations in a denoising/reconstruction setup rather than generative , which has been shown in the NLP community to learn better representations ( Raffel et al. , 2019 ; Devlin et al. , 2019 ) . Moreover , beyond simply applying MLM to the video tokens , we propose a block masking strategy to reduce the strong local correlation in neighboring video tokens . This 3D block masking strategy is inspired from recent span-masking schemes ( Raffel et al. , 2019 ; Joshi et al. , 2020 ) for language modeling . The concurrent work ( Bao et al. , 2021 ) explores using the VQ-VAE tokens as labels for masked patches in the image domain . The other direction of research , which our framework combines , is discriminative methods which start from the hypothesis that learning to reconstruct local details is not necessary for learning good visual representations . In some of these approaches , an objective is constructed around hand-crafted heuristics tasks like spatial arrangement , color , playback speed or frame order predictions ( Doersch et al. , 2015 ; Zhang et al. , 2016 ; Gidaris et al. , 2018 ; Fernando et al. , 2017 ; Lee et al. , 2017 ; Wei et al. , 2018 ; Epstein et al. , 2020 ; Benaim et al. , 2020 ; Sun et al. , 2021 ) . Another line of discriminative approaches is contrastive learning which aims at training a model to be able to recognize different views ( e.g. , different augmentation of images or different temporal samples of videos ) of the same image or video , as a way to learn general representations ( Chen et al. , 2020b ; He et al. , 2020 ; Chen et al. , 2020c ; Grill et al. , 2020 ; Caron et al. , 2020 ; Feichtenhofer et al. , 2021 ) . This direction of research is reminiscent of sentence-order prediction tasks introduced in NLP ( Devlin et al. , 2019 ; Lan et al. , 2019 ) with the goal of predicting whether two text sequences should be juxtaposed or not , an approach challenged in more recent literature ( Liu et al. , 2019 ; Lan et al. , 2019 ) . In the present work , inspired by visual representation rather than text representation learning literature , we adapt the contrastive learning approach to video by training a model to differentiate pairs of clips from a single video from pairs of clips from disparate videos . Another thread of research focuses on the way to adapt transformer models to video tasks ( Bertasius et al. , 2021 ; Arnab et al. , 2021 ) . Recent works ( Fan et al. , 2021 ; Liu et al. , 2021 ) extend it with hierarchical modeling and efficient attention patterns . These methods focus on the modeling and achieve good results with supervised pre-training . Besides the difference in focus ( modeling vs. pretraining ) to our paper , the hierarchical design is also not directly applicable to our mask prediction tasks .

Summary: The paper proposes a few new techniques: 1) A new video modeling architecture that uses a pre-trained VQ-VAE to tokenize frames, followed by a transformer encoder that aggregates the features and produces the final action class label. 2) Pre-training such an architecture using self-supervision by a) Masked prediction: Authors mask out blocks of tokens and predict them using the context (akin to BERT) and b) contrastive learning: Authors use the representation for two clips from same video as a positive match and otherwise negative match. The final model is trained with a linear combination of the masked prediction and contrastive losses, and finally finetuned for downstream tasks. The model is pretrained on HowTo100M dataset, and finetuned on multiple downstream datasets, where it obtains gains on more temporal datasets like SS-v2.