VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
Authors: Hao Tan
Jie Lei
Thomas Wolf
Mohit Bansal
Venue: ICLR 2022 Submitted (2022)
Keywords: video
self-supervised learning
representation learning
pre-training
action recognition

Abstract: Video understanding relies on perceiving the overall global content and modeling its internal connections (e.g., causality, movement, and spatio-temporal correspondence). To learn these interactions, we apply a mask-then-predict pre-training task on the discretized video tokens generated via VQ-VAE. Unlike language, where the text tokens are more independent, neighboring video tokens typically have strong correlations (e.g., consecutive video frames usually look very similar), and hence uniformly masking individual tokens will make the task too trivial to learn useful representations. To deal with this issue, we propose a block-wise masking strategy where we mask neighboring video tokens in both spatial and temporal domains. We also add an augmentation-free contrastive learning method to further capture the global content by predicting whether the video clips are sampled from the same video. We pre-train our model on uncurated videos and show that our pre-trained model can reach state-of-the-art results on several video understanding datasets (e.g., SSV2, Diving48). Lastly, we provide detailed analyses of the model scalability and pre-training method design. 

Under review as a conference paper at ICLR 2022
VIMPAC: VIDEO PRE-TRAINING VIA MASKED TOKEN PREDICTION AND CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Video understanding relies on perceiving the overall global content and modeling its internal connections (e.g., causality, movement, and teek-tthen-rrdicct dence). To learn these interactions, we apply mask-then predict pre-training task on the discretized video tokens generated via VQ-VAE. Unlike language, where the text tokens are more independent, neighboring video tokens typically have strong correlations (e.g., consecutive video frames usually look similar), and hence uniformly masking individual tokens will make the task too trivial to learn useful representations. To deal with this issue, we propose a block masking strat- egy where we mask neighboring video tokens in both spatial and temporal do- mains. We also add contrastive learning objective to further capture the global content by predicting whether the video clips are sampled from the same video. We pre-train our model on uncurated videos and show that our pre-trained model can reach state-of-the-ar results on several video understanding datasets (c.g.. SSV2, 48). Lastly, provide detailed analyses of the model scalability and pre-tranning method design.
correspon-
INTRODUCTION
In recent years, state-of-the-art self-supervised methods have been exploring different directions for pre-training images and text representations, with Contrastive Learning (CL) providing strong results for vision representation learning (Oord et al., 2018; Chen et al., 2020b; He et al., 2020; Chen al., 2020c; Tian al., 2020), and Language Modeling (L.) becoming the de-facto standard in language pre-training (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Lan et al., 2019). Both approaches are quite different from each other. A contrastive objective compares positive/negative examples at coarse/sample level, focusing global-content (e.g.. for image classification) while a token modeling objective predict missing tokens from context at a much finer/sub-sample level to model sequential and short range interactions between tokens (e.g. in text generation tasks). Interestingly, video understanding naturally combines both types of requirements. 2D processing along the spatial dimensions of the video bears similarity to image processing, while 1D processing along the temporal dimension often involves modeling sequential events and short range coherence.
Hence, in this work, we propose to combine both text and image representation learning approaches for improved video pre-training, taking advantage of recent advances in self-supervised methods of both fields. We name our method VIMPAC: VIdeo pre-training via Masked token Predic- tion And Contrastive learning. From language research, we adopt a 'masked language model' pre- training objective (Devlin al., 2019) where a model is trained to reconstruct local masked regions in videos. From the computer vision world, we borrow a contrastive learning objective, specifically the InfoNCE (Oord al., 2018) objective applied on positive/negative video samples. While the masked language model objective encourages models to learn low-level semantics and sequential interaction, the contrastive loss provide a supervision for the model to learn more global and sepa- rable representations that are useful for many downstream tasks (e.g., action classification (Soomro et al., 2012b; Carreira & Zisserman, 2017)). The two objectives provide complementary signals for training: while short range correlations nal of the mask-and and-predict task, the contrastive learning objective can provide signals on a more coarse-grained global-context and semantic level.
be predominantly modeled from the training sig-

---

Under review as a conference paper at ICLR 2022
However, unlike language which is composed of discrete tokens from compact vocabulary, videos are typically represented as RGB pixels in an almost continuous, high dimensional vector space. Naively masking pixels in videos induces a prohibitive computation cost while also tending to over- emphasize local details. To overcome these issues, we first tokenize input videos using the latent codes of a pretrained Vector Muantized-Variation 2017; Ramesh et al., 2021) to encode them in smaller quantized representations on which a re- construction model can then be trained with masked token modeling objective. In practice, we also discovered that models trained with a uniform random token masking strategy can fail to learn meaningful and useful visual representations as neighboring pixels may contain similar and corre- lated content (in particular along the temporal dimension), making the task of predicting randomly masked token from its visible neighbors trivial. We therefore also introduce block-masking scheme that simultaneously masking video tokens in a 3D spatio-temporal block. Reconstructing such an ex- tended spatio-temport cube requires performing long- range predictions, forcing the models to learn more complex set of relations between the video tokens, resulting in better visual representations.
Auto-Encoder (VQ-VAE) (van den Oord al.,
Our contrastive learning approaches also departs from previous work in several aspects. First, since apply the contrastive objective on token- discretized video samples and in combination with the token modeling loss, we observe strong performance without requiring the usual extensive set of data augmentations (Chen et al., 2020b;e; Qian et al., 2021; Feichtenhofer al., 2021). Second, we are able to leverage positive clip pairs that are temporally distant from each other (can be as far as 400 seconds away), while previous work favors using positives within shorter range (maximum 36 seconds for uncurated videos in Feichtenhofer et al. (2021) or 10 seconds in Qian et al. (2021)).
We evaluate the performances of our method VIMPAC on several video understanding datasets, including two temporally-heavy tasks, SSV2 and Diving48 on which it achieves state-of-the-art results with regard both self-supervised and supervised pre-training works, and a set of more spatially-heary datasets, UCF101, HMDB51, and Kinetics-400, on which it also achieves competi- tive results. Overall, taking advantage of VQ-VAE discretized video tokens, we present a method for self-supervised learning of video representations that combines two general streams of research in self-supervision: masked language modeling and contrastive learning. Our contribution is 3-folds: (i) We apply the mask-then-predee ing. (ti) We propose a contrastive learning method which is able to achieve strong performance without spatial data augmentation. (iii) We empirically show that this method achieves strong per- formance on several video classification datasets, especially on temporally- and Diving48, where it sets new state-of-the-ar results. We also present comprehensive ablation studies to analyze the various aspects of our proposed approach.
task to video understanding and introduce the use of block mask-
datasets, SSV2
2 RELATED WORK
Unsupervised representation learning, with the promise of learning from large-scale unlabeled data, has drawn increasing attention in recent years, in both computer vision and natural language pro- cessing (NLP) communities. Most mainstream self-supervised methods can be categorized into three general directions: generative, denoising, and discriminative (Chen al., 2020b; Grill al., 2020; Doersch et al., 2015).
Generative and denoising methods seek to generate reconstruct corrupted text/image/videeo tokens according to their empirical distributions. In generative and auto-regressive methods, next tokens are predieted given causal context (Chen et al., 2020a; van den Oord et al., 2016) while denoising methods seek to reconstruct corrupted or masked tokens given an extended context (Devlin et al., 2019; Raffel et al., 2019). For text, since the tokens (words or sub-words (Sennrich et al., 2016; Wu et al., 2016)) are discrete and has relatively high entropy rate, language modeling has became the de-facto approach for pre-training models for most natural language tasks (Ruder et al., 2019). In the case of images, generative approaches often operate on pixel space (Bertalmio et al., 2001; Yu et al., 2018; Kim et al., 2019; Chen et al., 2020a; van den Oord et al., 2016), which can be extremely expensive for larger input size like videos and has hence limited the widespread adoption of these methods. Recently, discretizing images and videos with discrete variational auto-encoders (VQ-VAE), has been explored in compression and generative setups (van den Oord et al., 2017; Razavi et al., 2019; Walker et al., 2021: Ramesh et al., 2021; Yan et al., 1) and Sun et al. (2019) tackles the video-language problem by frame-level quantization. Such approaches avoid modeling
2

---

Under review as a conference paper at ICLR 2022
Video Clip
Block- Masked Tokens
Frame
Frame 2
VO-VAE Erooder
VQ-VAE Encedder
[CLS] Token
Spatio-Temporal Transformer
Task 2): Contrastive Learning
Task 1): Mask-then-
Overview of VIMPAC. Sampled video frames are discretized by VQ-VAE encoder into discrete tokens, which are then block clock-masked (in light yellow blocks). The model is self self-suuer by two tasks: 1) mask-ther teee--hhe-pppr thee-ct task predicts the masked tokens from their visible context; 2) contrastive learning task differentiates between positive and negative clips (details in Fig. 2) with [CLS] token feature. For brevity, we only show 2 frames with small token maps.
Figure 1:
pixel-level details and have enabled the use of generative models for images and videos (Walker et al., 2021; Ramesh et al., 2021). Differing from these works, our framework investigates the use of such quantized representations in denoissingrrcoonstuutiin setup rather than generative, which has been shown in the NLP community to learn better representations (Raffel et al., 2019; Devlin et al., 2019). Moreover, beyond simply applying MI. to the video tokens, we propose a block masking strategy to reduce the strong local correlation in neighboring video tokens. This 3D block masking strategy is inspired from recent span-min mannmaskinn schemes (Raffel et al., 2019; Joshi et al., 2020) for language modeling. The concurrent work (Bao et al., 2021) explores using the VQ-VAE tokens as labels for masked patches in the image domain.
The other direction of research, which our framework combines, is discriminative methods which start from the hypothesis that learning to reconstruct local details is not necessary for learning good visual representations. In some of these approaches, heuristics tasks like spatial arrangement, color, playback speed or frame order predictions (Doersch al., 2015; Zhang et al., 2016; Gidaris et al., 2018 8; Fernando et al., 2017; Lee et al., 2017; Wei et al., 2018; Epstein et al., 2020; Benaim et al., 2020; Sun et al., 2021). Another line of discriminative approaches is contrastive learning which aims training model to be able to recognize different views (e.g., different augmentation of images or different temporal samples of videos) of the same image or video, as way learn general representations (Chen al., 2020b; He al., 2020; Chen et al., 2020c; Grill et al., 2020; Caron et al., 2020; Feichtenhofer et al., 2021). This direction of research reminiscent of sentence-order prediction tasks introduced NL.P (Devlin et al., 2019; Lan et al., 2019) with the goal of predicting whether two text sequences should be juxtaposed not, an approach challenged in more recent literature (Liu et al., 2019; Lan et al., 2019). In the present work, inspired by visual representation rather than text representation learning literature, we adapt the contrastive learning approach to video by training a model to differentiate pairs of clips from a single video from pairs of clips from disparate videos.
objective is constructed around hand-crafted
Another thread of research focuses on the way to adapt transformer models to video tasks (Bertasius al., 2021; Arnab al., 2021). Recent works (Fan et al., 2021; Liu al., 2021) extend it with hierarchical modeling and efficient attention patterns. These methods focus on the modeling and achieve good results with supervised pre-training, Besides the difference in focus (modeling vs. pre- training) to our paper, the hierarchical design is also not directly applicable to our mask prediction tasks.
METHODS
In this section, we present our proposed video pre-training method VIMPAC (Fig. 1) as well its de- tailed components. We first introduce the mask-ther then--redict task in Sec. 3.1, and then the contrastive learning task in Sec. 3.2. Lastly, discuss how these tasks are combined in Sec. 3.3.
3

---

Under review as a conference paper at ICLR 2022
Video Frames
Video Frames
Reference
Video
VQ-VAE Encoder
Temporal Trans- former
Positive
Video 2
Negatives
Block Masking
L.I.D. Masking
Video
(a) Illustration of Block and I.I.D. Masking.
(b) Illustration of Contrastive Learning.
Illustration of pre-training tasks. (a): block masking constructs the 3D-contiguous mask- ing cube while i.i.d masking independently samples masked tokens. (b): given the reference clip, the positive clip is uniformly sampled from the same video (video 1) while negative clips are sampled from other videos (video 2, 3). No spatial augmentations applied to the raw video clips.
Figure 2:
3.1 MASK-THEN- TEEN-PEEE TASK
Suppose that video clip input comprises T frames J2......... Sr), the mask-then- task learns video representations by predicting the masked contents from their spatio-temporal context. Denote the of mask-token locations as M, we learn to predict the original tokens { (see details below) by optimizing the negative log-likelihood:
Lmask
Σ
log (#r,a (ee
(1)
where M is the complement of M and thus indicates the unmasked context.
Video Quantization with VQ-VAE. Since directly applying mask-then-predde over raw pixels and masking/prediccing pixels leads to prohibitive computational costs and also tends make the model overfit detailed low-level visual information, we quantize the input videos with Vector Quantized- Quantized-Variationaal Auto Encoder VAE) (van den Oord et al., 2017; Ramesh et al., 2021). The VQ-VAE encoder takes an image input and produces a token map, where the tokens belong to a predefined vocabulary V. The VQ- decoder then tries to reconstruct the original image from these latent codes. In method, a compressor that converts an input from an original input space RR. H Wx3 into a discretized space [V] x We independently apply the VQ-VAE encoder to each frame f: inside clip. Specifically, we use the VQ- VQ-VAE trained in DALL-E (Ramesh et al., 2021). We keep the VQ-VAL weights frozen and do not finetune adapt this model on our corpus.
frozen and pretrained generic VQ-VAE encoder as
Block Masking For sampling tokens to mask, the original BERT methods proposes the i.i.d. (in- dependent and identically distributed) random mask Mia that constitutes of masked tokens:
Mia (,, Me,s[0, 1] <)
where Ue [0, 1] is the uniform distribution from 0 to 1. Intuitively, & is the expectation of masked- token ratio and hence controls the difficulty of our tessk-ttee-prediin task. In our early experiments, we found it easy to infer masked token from its direct spatio-temporal neighbours (e.g., neighbor- ing frames in video tend look similar thus contain similar tokens). To overcome this issue, we propose to use block masking (see Fig. 2 (a)). which masks continuous tokens inside spatio-temporal blocks. For each mask block B, we randomly sample lower (Bw.o) and upper boundaries (B,,) for each of the temporal (I), height (H), and width (W) dimensions. The direct product of the intervals delimited by these boundaries constructs the block mask. The final mask Moteck is the union of them:
Motock UplBr Br.)
[BH,,, BH.] [Bw,o, Bw.].
3.2 CONTRASTIVE LEARNING
Contrastive learning aims to distinguishing positive pairs from negative pairs (see Fig. 2 (b)). For each video videos, uniformly and independently sample clips ct, of positive pair, while
4

---

Under review as a conference paper at ICLR 2022
the clips in a batch belonging to other videos used to construct negative pairs. A model (de- scribed in Sec. 3.4) processes clips o₁, c; to build respective vector representations fi, f and an InfoNCE (Oord et al., 2018) loss is used to distinguishes the positive feature pair (ft. f/) from the negative pairs UHL(s, (K) & of i} for each clip c::
exp (f. 81/7)
(1) = log Enst exp
(4)
exp (f. fu/h) LE exp ft/h)
which we combine with the symmetric loss (IatONCE(1) for paired clip sample c4. The final loss for a mini batch La is the average loss for all n clips the mini-batch:
1 n2
1
L
CINONCE (1) +
CINONCE
(5)
3.3 PRE-TRAINING OBJECTIVE
We combine the pre-training methods discussed above to define the overall objective
L Lmssk + Cry
(6)
where a is hyperparameter controlling the weight of the contrastive loss and multiplying the tem- perature y will smooth training (Grill et al., 2020; Chen et al., 2021). The inputs for both tasks with the contrastive learning loss using the same block-masked inputs
shared in necessary for the mask-then-predic task. We highlight that the masked tokens are the only noise in- troduced in the contrastive learning, and that no other data augmentation applied to raw pixels, in contrast to previous vision contrastive learning methods in which data-augmentation was paramount the final performances of the model. This phenomenon empirically studied in Sec. 5.2.3.
3.4 MODELING
The model architecture follows the standard transformer architecture in its post-layer-norm vari- ant (Vaswani et al., 2017; Devlin et al., 2019) with two more recent additions: divided temporal- spatial attention (Bertasius et al., 2021), and sparse spatial attention (Child et al., 2019). We detail both additions in the Appendix. The model embedding layer maps the discrete tokens { } of a quantized input video (see Sec. 3.1) into dense vectors and sum them with positional embeddings. The backbone transformer model then outputs corresponding features j.. We append an addi- tional [CLS] token to each input sequence following Devlin et al. (2019) and its output feature hets as a representation for the whole video. For pre-training, we use two heads: a 2-layer MLP after each token outputs (hr,,,) for the mask-then-preee preeict task following BERT (Devlin et al., 2019), and a 3-layer MLP after the CLS output hes for the contrastive learning task following SimCLR (Chen al., 2020b). For Ane-tuning on classification tasks, we remove the pre-training heads and add a fully-connected layer to the [CLS] output hass More implementation details are in Appendix.
4 EXPERIMENTS AND RESULTS
4.1 DATASETS
For pre-training, we use the HowTo100M dataset (Miech et al., 2019) 1. This dataset is constructed by searching YouTuBe videos with list of text queries, it is significantly larger and more diverse than human-annotated datasets such as Kinetics 400 (Carreira et al., 2019). HowTo100M has 1.2M uncurated videos, with average duration of 6.5 minutes. We only videos and do not use other signals such as ASR captions in this dataset. For downstream evaluation, we experiment with several action classification datasets: UCF101 (Soomro al., 2012a), HMDB51 (Kuehne et al., 201 Kinetics-400 (Carreira & Zisserman, 2017), SSV2 (Goyal et al., 2017), and Diving48 (Li et al., 2018) 8). It is important note that in many cases, actions in UCF101, HMDB51, and Kinetics- 400 can be recognized from single frame of the video, thus these datasets are 'spatiall-heaary.. As
'The VQ-VAE compression method is trained on 250M images as in (Ramesh et al., 2021). However, the VQ-VAE does not help with hetter representation for the video pre-training as shown in Sec. 5.2.1, where directly training on the VQ-VAF tokens provides poor results.
5

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison with state-of-the-art. Our model outperforms previous works on SSV2
and Diving48 dataset while showing competitive results on other datasets. Results on UCF101
and HMDB51 are average over three train-val splits. V.A,T refer to Visual, Audio, and Text modalities, respectively. "(Grill et al., 2020; Feichtenhofer et al., 2021), "(iiech et al., 2020),
"(Alayrac al., 2020), "(He al., 2020; Feichtenhofer et al., 2021), "(Bertasius (Arnab et al., 2021), "(Feichtenhofer et al., 2019), "(Kalfaoglu et al., 2020), J 'TTann et al., 2018),
al., 2021),
& "Wang et al., 2019), (Kondratyuk et al., 2021). K400=Kinetics-400 (Carreira & Zisserman, 2017), HT=How www.wTT00000 To100M (Miech et al., 2019), AudioSet (Gemmeke et al., 2017). IG-Uncurated (Ghadi- yaram et al., 2019), IN2IK=ImageNet-211 (Russakovsky et al., 2015). Note that some SotA models pre-trained with extremely large weekkl--Juppervise datasets, e.g. IG65M (Ghadiyaram al., 2019) in A Kalfaoglu et al. (2020) and JFT-300M (Sun et al., 2017) in Arna et al. (2021).
Tempurally
Spatially- Heeav
Method
Modality
Pre-Train Dataset
SSV2
Diving48
UCFI01
HMDB51
K400
(Weakly) Supervised Pre-
Pre-Training
K400 Sup.
V
K400
63.19
96.8
82.5
81.5
TimeSSormer' space-time
V
IN21K
62.3
81.0
80.7
TimeSformer' space-only
V
IN21K
36.6
77.6
VIVIT!
V
IN2IK/FT30OM
65.4
84.8
R(2+1)D BERT
V
IG6SM
98.7
85.1
Self-supervised Pre-Training on Uncurated Videos
MIL-NCE
V+T
HT
91.3
61.0
MMV
V+A+T
AudioSet HT
95.2
75.0
BYOL
V
K400
55.8
96.3
75.0
MoCodd
V
IG-Uncurated
53.2
92.9
VIMPAC
V
HT
68.1
85.5
92.7
65.9
75.3
consequence, image-level methods (Bertasius et al., 2021; Radford et al., 2021) show competitive results without modeling the temporal interactions inside the videos. To test the video model's abil- ity beyond recognizing static images, we lay our focus on 'temporally-heayy D8ving48), which action recognition from a single frame more difficult. For example, it is al- most impossible to distinguish two SSV2 classes moving something up and moving something down without reasoning across frames, and the same for different diving classes in Diving48. Additional dataset details (e.g., statistics) presented in Appendix.
datasets (SSV2 and
4.2 EXPERIMENTAL SETUP
Our model shapes follow BERTLARGE with 24 layers and hidden size 1024, but with halved attention head size and MI.P intermediate size as in Child et al. (2019). For pre-training, we train the model for 100 epochs on HowTo100M with frames sampled at 2 FPS. We sample two clips from each video as model inputs described in Sec. 3.2. To reduce computation cost, train the first 90 epochs with a smaller input resolution (#frames 7=5 and frame size S=128) and increase the spatial resolution (T=5, S=256) for the last 10 epochs following Devlin al. (2019). Positional embed- dings are interpolated as Dosovitskiy et al. (2021) when input resolution changes. Importantly, our pre-training scheme does not involve spatial augmentations: all frames are resized and centered cropped without random flipping, color distortion, etc. We use a batch size of 1024 in pre-training. The number of negative clips used for contrastive learning is 255 for the first 90 epochs and 127 for the last 10 epochs. The number of negative pairs used in our ablation analyses is kept constant at 127. More details are in Appendix. For fine-tuning, we use more input frames (I'=10 and S=256), and batch size 128. We sample frames at 2 FPS for datasets with longer videos (i.e., UCF101 and Kinetics-400), and sample 4 FPS for datasets with shorter videos (i.e., HMDB51, SSV2, Diving48). During inference, we follow Feichtenhofer et al. (2019; 2021) to use 3 spatial crops and 10 tempo- ral crops (in total 30 crops), and average their prediction scores as the final score.2 ? All models are trained with AdamW (Loshchilov & Hutter, 2018) optimizer with linear warm-up and linear learning rate decay. We observe similar pre-training instability as reported Chen et al. (2020a; 2021) 1) and follow their practice sequentially choose learning rate at 1e-3, 5c-4, 3e-4, .... until convergence.
2 As in Bertasius et al. (2021); Arnab al. (2021), observe that the performance is saturated at 4-5 temporal crops for model.

---

Under review as a conference paper at ICLR 2022
Impact of model size. 'Speed" is the normalized pre-training speed measured by #videos/second on one V100 GPU. "Mask-Accu.' and 'CL-Loss' are mask-then-preddee accuracy and contrastive learning loss to indicate the pre-training performance. 'UCF101' the fine-tuning accuracy on UCF101 dataset. By default, we use the configuration in the first line in our analysis. The configuration that produced the final results are underlined.
Table 2:
Layers
Dim
Params
Speed
Mask-Accu.
CL-Loss L
UCFIOI1
6
512
29. 19.4M
32.0
17.2
1.06
69.4
6
768
63.0M
21.0
17.7
1.03
75.0
12
512
54.7M
18.1
17.9
1.02
76.6
12
768
119.7M
11.2
18.4
1.00
78.1
24
1024
210.1M
5.0
18.7
0.98
78.5
4.3 RESULTS
Table shows our primary results. We mainly compare with self-supervised pre-training methods on uncurated videos. In addition, we list methods with (weakly) supervised pre-training as refer- ences, though they can not be fairly compared with our method they use large-scale (weakly) labeled data. As discussed in Sec. 4. recognizing actions in SSV2 and Diving48 require a strong temporal reasoning ability, while in the other datasets, spatial understanding is dominant. To bet- ter illustrate the differences between temporally-heay. and spatially-heav. datasets, we compare two variants of TimeSformer (Bertasius et al., 2021), one with attention on space-time, and one on space only. Note the gaps between these two variants are significantly larger for temporally- heavy datasets (SSV2) than apatially-heay datasets (Kinetics-400),, demonstrating the importance of temporal modeling for temporally-heavy datasets. On the two temporally-heavy datasets SSV2 and Diving48, when comparing to previous best models among all self-supervised and supervised pre-training methods, our model VIMPAC sets new state of the art, where we achieve 2.7% and 4.5% absolute improvement, respectively. This especially surprising considering the two previous SotA models ViVIT (Amaa et al., 2021) and TimeSformer (Bertasius et al., 2021) both use large- scale supervised pre-training, and ViVIT also uses various regularization techniques (e.g., stochastic depth (Huang et al., 2016), random augment (Cubuk et al., 2020), and mixup (Zhang et al., 2018)). spatially-heavy datasets,UCF10n HMDB51 and Kinetics-400, VIMPAC achieves competitive results to self-supervised pre-training methods, while being lower when compared to supervised methods. These relatively low results of our VIMPAC (c.g.. UCF101) are possibly due to the spa- tial information loss during the VQ-VQA quantization process. Concurrent work BEiT (Bao et al., 2021) addresses this issue by using image patches instead of VQ-VAE tokens as inputs, where they show strong performance on image classification tasks. We encourage future work to study using image patches as inputs for better spatial modeling under our framework. Previous self-supervised pre-training methods such as BOL (Grill et al., 2020; Feichtenhofer et al., 2021) and MoCo (He et al., 2020; Feichtenhofer et al., 2021) are good at global understanding, but the pre-training schema does not consider the internal interactions inside videos (especially for the temporal dimension). As a result. it could reach or even outperform the supervised alternatives on UCF101. However, it shows lower results on SSV2 compared to the transformers (Bertasius et al., 2021; Arnab et al., 2021) (although with different backbones) that warm up from -mega-pee- models and learn the temporal interactions directly from the downstream tasks. We also show the cross-modal self- supervised learning methods, MIL-NCE (Miech et al., 2020) and MMV (Alayrac et al., 2020) that are trained on uncurated videos but leverage other modalities (e.g., text) to help video learning.
ANALYSIS
We also analyze the model's scalability and the effectiveness of our pre-training methods. To save computation, for all analyses, we use smaller model (6-layer transformer with hidden dimension 512) and smaller input resolution (5 input frames with spatial size 128, i.e., 7=5,, S=128) throughout this section, unless otherwise stated. We also perform pre-training with fewer epochs (i.e., 10). For downstream tasks, we use the same input resolution as pre- gre-trannig (i.e., T=5, S=128), and we use 2 temporal crops for inference. All results are reported on the train-val split 1 if applicable.
'The image model CL.IP (Radford et al., 2021) achieves 92.0% the spatially-heavy UCF-101 dataset.
7

---

Under review as a conference paper at ICLR 2022
Table Impact of masking strategy. Models are pre-trained with only mask-then then-predict
Table Impact of masking ratio. Models are pre-trained with only mask-then-predict. De- fault setup underlined.
Strategy Frame Size S Mask-Accu.1
UCFI01
Strategy #Blocks Ratio Mcsk-Accu.1 UCF101
block i.dd.
128
17.6
68.3
128
24.3
63.5 (-4.8)
block
11.9%
17.9
66.8
block
256
11.2
69.5
block
5
14.5%
17.6
68.3
i.idd
256
19.5
61.4 (-8.1)
block 6
17.0%
17.3
67.3
5.1
SCALABILITY
In Table 2. we illustrate the scalability of our method with different model sizes (i.e., number of layers and hidden dimensions). Larger models have more parameters ("Params') and higher compu- tational cost (measured by the normalized pre-training 'Speed'). To evaluate the pre-training tasks performance, we provide both pre-training metrics (mask-then-predits accuracy denoted by 'Mask- Accu.', and contrastive learning loss denoted by 'CL-Loss") and UCF101 downstream fine-tuning results. As the size of the model grows, the fine-tuning results show consistent improvement with the pre-trainn pre-training metries. Note that for the last row in Table 2, we halve the attention head and MLP intermediate dimensions. We also illustrate the scalability over input resolution in Appendix F.2.
5.2
PRE-TRAINING METHOSS
5.2.1 THE IMPACT OF PRE-TRAINING
We first compare different pre-training tasks and the non-pre-training results. As shown in Table 3, mask- datasets
Table 3: Impact of pre-training tasks.
"MP'=Mask-thee' teask-tten-Predict CoottConraastive Learning task.
then-predict is good at comporally-heay (SSV2, Diving48) while contrastive learning im- proves the spatially-heavy datasets." 4 We also com- pare with the non-pre-training results (the first row of Table 3) and observe that both tasks significantly improve the results. We notice that these non-pre- training results are lower than previous from-scratch models, which might be caused by the difficulty in training video transformers (Bertasius et al., 2021; Arnab et al., 2021) and the information loss in our input quantization process (Ramesh et al., 2021).
Temporally-
Spatially-Heavy
MP CL
SSV2
Diving48
UCFI01 HMDB51 K400
x
x
1.2
10.0
41.3
19.0
41.0
x
/
32.5
26.3
57.1
30.7
47.0
41.4
68.3
35.3
53.7
41.1
37.8
5.2.2 MASK-THEN THEN-HREDICTT
In this analysis, we exclude the contrastive learning loss (i.e., loss weight a=0) avoid side effects.
Block Masking versus L.I.. Masking, We first compare our proposed block masking strategy and the uniform i.i.d. masking strategy (discussed in Sec. 3.1 and illustrated in Fig. 2). As shown in Table 4, although the i.i.d. masking achieves higher pre-training mask-token -rreettptniittoon accuracy ('Mask-Accu.)) shows lower downstream results ("UCF101') than block masking. The higher mask accuracy is possibly due to the easier i.i.d. mask-then tteen-predic task. We show in Appendix that simply copy-paste already yield reasonable reconstruction results. The existence of such a trivial solution potentially prevents the model from learning useful video representations for downstream tasks. Meanwhile, we also find that the model with larger input frame size 256 benefits more from the block masking strategy, because the adjacent tokens are closer in the original 2D image for these larger frames. Hence, the spatial locality amplified.
Masking Ratio. In Table 5, study the impact of masking ratio, by varying the number of masked blocks for block masking. Empirically, the result differences among different masking ratios are marginal and the original BERT's 15% masking ratio (with roughly 5 masking blocks) works slightly
'Empirically, we observe that improvement contrastive learning (CL) becomes higher when training with more epochs and larger architectures (as also shown in Chen et al. (2020b); Qian et al. (2021)). With the Base model (but also using decreased training epochs), CL gives 3% improvement on UCF101 over the pure musk-thhe-preede pre-training. In this section, we provide the comprehensive ablation studies based on the small model because of the limited budget.
8

---

Under review as a conference paper at ICLR 2022
Table 6: Impact of maximum sampling dis- tance dmax (sec.) between two positive clips.
Table 7: Impact of #negative samples.
#samples Mask-Accu.
CL--Loss UCF1011
dmx Mask-Accu, CL--Lsss UCFI011
128-1
17.2
1.06
69.4
30
17.2
1.06
69.4
256-1
17.1
1.30
30
17.3
0.77
69.0 (-0.4)
512-1
17.2
1.56
70.4
10
17.4
0.61
68.3 (-1.1)
1024
17.0
1.86
69.8
17.5
0.41
66.7 (-2.7)
better.
Thus we always select the number of mask blocks whose induced masking ratio closest to The detailed choices of number of masking blocks are listed in Appendix.
15%.
5.2.3 CONTRASTIVE LEARNING
Positive Sampling Distance. As illustrated in Sec. 3.2 and Fig. 2.(b), we uniformly sample positive clip pairs across the whole video without any distance restriction. To analyze the effect of such a sampling strategy, We perform set of experiments by varying the maximum sampling distance dmaz (in seconds) between two positive clips. The results are shown in Table dma== denotes our default setup without any distance restriction. dmaz== samples two same clips, and dmaa=10 samples two positive clips with maximum distance of 10 seconds. Although previous contrastive learning methods (Qian et al., 2021; Feichtenhofer et al., 2021) favor the sampling of temporal positives within a shorter range (e.g., maximum 36 seconds for uncurated videos in Feichtenhofer et al. (2021)), we observe a performance gain when using larger distance. We also want to emphasize that the results with 10m and dmaz=0 are not better than the model pre-trained with only mask- then- then-predict (UCF101 accuracy 68.3), which suggests that short-range contrastive learning does not improve upon our mask-then-predice task. This is potentially because our mask-then-pree -eeee-tredict already gives the model the ability to model local interactions, thus contrastive learning objective only be useful when it focuses on longer-range interactions.
Number of Negative Samples. Previous constrastive learning methods (Chen al., 2020b; 2021; Feichtenhofer et al., 2021) benefit from more negative samples. In this section, we show that the number of negative samples has less impact on our method when mask-then-predice task is added. As shown in Table 7, we use different contrastive learning sample sizes (i.e., in Sec. 3.2) and always accumulate the gradients to 1024 samples before updating the parameters. Although increasing sample size makes the contrastive learning task harder (reflected by 'CL-Loss'), it does not show clear evidence of improving UCF101 downstream performance.
Augmentation. Most self- supervised visual representation learning meth- ods (Chen et al., 2020c;b; Grill et al., 2020; Feicht- enhofer et al., 2021; Qian et al., 2021) based on con- trastive learning suffer from large drop when moving strong spatial augmentations. In contrast, our pre-training does not use any spatial augmenta- tions on raw frames. However, as we tie the input be- tween mask-then-pe ieteentreedct and contrastive learning reduce computation cost, the random masking noise naturally introduced. We here investigate im- pact in Table 8. When pre-trained jointly with mask-then-prediie adding mask noise improves UCF101 accuracy by +2.0; however, when pre-trained without it, adding mask noise hurts the per- formance (-1.6). We hypothesize that this is due to the large input mismatches between pre-training and fine-tuning when mask-then- then-predict objective is not applied. Noisy masking creates 'holes' to the input token maps during pre- pre-trainin,, while for fine-tuning the input token maps intact.
Input Masking
Table 8: Impact of mask augmentation in contrastive learning. "MP'=Mask-thee
Predict. "CL-Mask'=UR input mask in CL. Default setup is underlined.
MP CL-Mask CL-Loss] UCFI01
1.07
57.1
x
/
1.08
55.5
x
17.2
1.04
67.4
<
17.2
1.06
69.4
6 CONCLUSION
We present the video pre-training framework VIMPAC that introduces mask-then tockk-then-puue then-predict task to video self-supervised learning. mask-then -reen--reedicct task helps model spatio-temporal interactions that important for video understanding. We the VQ-VAE quantizer and propose the block mask-

---

Under review as a conference paper at ICLR 2022
ing method that is essential to overcome the strong locality in video. The contrastive learning task also added to learn separable global features. Different from previous methods, our contrastive learning does not use data augmentation over raw frames and is less sensitive to the temporal sam- pling distribution for positive pairs. We show that our frameworks could achieve state-of-the-an performance two temporally-heavy dataset (SSV2 and Diving48) and reach competitive results on other datasets. Detailed analysis is provided regarding the model scalability and task design.
Reproducibility statement. The code to reproduce the results in this paper is submitted in the supplementary material and will be made public. The code is runnable and contains the scripts with faithful hyperparameters for our experiments. We include comprehensive instructions about the feature extraction, pre-training, and fine-tuning. A detailed "readme" file is attached in the codebase, and we will publicly release the pre-trained weights as well.
Ethical considerations and limitations. The main purpose of this work is design video pre- training method to capture internal interactions. In real life, a lot tasks are temporally-hese such that the task completion relies understanding the past. Video understanding is proxy to these embodied studies eventually and thus could one day benefit our daily life. Our work is also potentially useful for recovering corrupted videos (but this could also be possibly misused, hence we recommend careful and safe use of this technology, including previous works). In this paper, we mainly consider the action classification tasks with different characteristics (i.e., matially-heavy and temporally-henn and we show the recovering ability of our model. It's also possible to employ VIMPAC backbone to video detection tasks where tokens' outputs can be viewed as anchors. The DALL-E VQ-VAE code and model are published under MIT license. We are among the first line of work that use masked token reconstruction, etc. as self-supervised learning approach for vision, and we do agree the the use of VQ-VAE tokens might limit the models' ability in certain aspects, such as spatial info loss. Future work could explore combining the techniques from BEiT (Bao et al., 2021) for more capable approach. Meanwhile, token-level approach has its unique advantage of generative modeling (as shown in VQ-GAN (Esser et al., 2021). DALL-E (Ramesh et al., 2021)), )), more robust to the input noise, and has the possibility to transfer to other modality (e.g., text; since they share the same types of input). This cross-modality transferability is another future direction that we consider.
REFERENCES
lean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised mul- timodal versatile networks. In NewrlPS, 2020. 6.
Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. arXiv preprint arXiv:2103.15691, 2021. 3, 6, 7,8, 15
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. In NeurlPS, 2016. 15, 16
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint wrXiv:2106.08254 2021. 3, 7, 10
Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. Speednet: Learning the speediness in videos. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Marcelo Bertalmio, Andrea L Bertozzi, and Guillermo Sapiro. Navier-stokes, fluid dynamics, and image and video inpainting. In Proceedings of the 2001 IEEE Computer Society Conference Computer Vision and Pattern Recognition. CVPR 2001, volume 1, pp. I-I. IEEE, 2001.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In JCML., 2021. 3,5, 6, 7, 8, 15, 16, 20
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.
Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 09882, 2020.
10

---

Under review as a conference paper at ICLR 2022
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017. 1,5, 6,
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics- human action dataset. arXiv preprint arXiv:1907.069900 2019. 5, 19
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691- 1703. PMLR, 2020a. 2, 6
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. arXiv preprint arXiv:2002.05709 2020b. 1, 2. 3, 5, 5.9.9
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.00 N0Xx:2003304297, 2020c. 1, 3,
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervvsed visual transformers. arXiv e-prints, pp. arXiv-2104, 2021. 5, 6, 9, 15, 17, 19
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 0004//044///// 10509, 2019. 5, 6, 15, 16, 19, 20
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Jacob Devlin, Ming- Min Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, 2019. 2, 3, 5, 6, 15, 16, 17,
19
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference computer vision, pp- 1430, 2015. 2,
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 6, 15, 20
Dave Epstein, Boyuan Chen, and Carl Vondrick. Oops! predicting unintentional action in video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 919-929, 2020. 3
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nilion, pp. 12873-12883, 2021. 10
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. Proceedings of the IEEE/CVF Inter- national Conference Computer Vision (ICCV) pp. 6824-6833 6835, October 2021.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp- 6202-6211, 2019. 6, 19, 20
Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. arXiv preprint arXiv:2104.1455 2021.
2, 3, 6, 19
11

---

Under review as a conference paper at ICLR 2022
Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video rep- resentation learning with odd-one-out networks. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 3636-3645, 2017. 2017.
Jort Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human labeled dataset for audio events. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776-780. IEEE, 2017. 6
Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan. Large-scale supervs- pre-training for video action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12046-12055, 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.0778, 2018. 3
Raghay Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzyúska, Susanne West- phal, Heuma Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense, 2017. 5, 23
Grill, Florian Strub, Florent Altché, Corentin Tallec Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent new approach to self-supervised learning. In NeurIPS, 2020. 2, 3, 6, 7,9
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020. 6,7
Dan Hendrycks and Kevin Gimpel.
Gaussian error linear units (gelus). arXiv preprint
160.v:1600.08845. 2016. 16
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646-661. Springer, 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-456. PMLR, 2015. 17
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Span- bert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2020.
M Esat Kalfaoglu, Sinan Kalkan, and A Aydin Alatan. Late temporal modeling in 3d cnn architec- tures with bert for action recognition. In European Conference on Computer Vision, pp. 731-747. Springer, 2020.
Dahun Kim, Sanghyun Woo, Joon- Joon-Young Lee, and In Kweon. Deep video impainting. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5792-5801, 2019. 2
Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Bo- qing Gong. Movinets: Mobile video networks for efficient video recognition. arXiv preprint arXiv:2103. 11511, 2021. 6
Hilde Kuehne, Hueihan Jhuang, E. Garrote, T. Poggio, and Thomas Serre. Hmdb: A large video database for human motion recognition. 2011 International Conference on Computer Vision, pp. 2556-2563, 2a. 5
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In 2017 International conference on computer vision, pp. 2556-2563, IEEE, 201 lb. 23
12

---

Under review as a conference paper at ICLR 2022
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self supervsee learning of language representations. In International Conference on Learning Representations, 2019. 1,3
Hsin-Yine Ying Lee, Jia Huang, Maneesh Singh, and Ming- King-Hsuan Yang. Unsupervised represen- tation learning by sorting sequences. In Proceedings of the IEEE International Conference on Computer Vision, pp. 667-676, 2017.
Yingwei Li, Li, and Nuno Vasconcelos. Resound: Towards action recognition without representa- tion bias. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 513-528, 2018. 5, 23
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907. 2019. 1,
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. transformer. arXiv preprint arXiv:2106. 13230, 2021. 3
Video swin
lyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Reprexentations, 2018.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTol 100:: Learning Text- Video Embedding by Watching Hundred Million Narrated Video Clips. ICCV, 2019. 5, 6, 19
Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisser- man. End-to-end learning of visual representations from uncurated instructional videos. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9879- 9889, 2020. 6,7
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv: 1007:::807,,0374,, 2018. 1,5
Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 157-163, 2017. 16
Rui Qian, Tianjian Meng, Boqing Gong, Ming- Hsuan Yang. Huisheng Wang. Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In CVPR, 2021. 2, 8, 9, 19
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv: 20.. 00020, 2021. 6,7
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: /00.//0800. 2019. 2, 3
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102, 2021. 2, 4, 5, 8, 10, 15, 17, 18, 19
Ali Razavi, Aäiron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq- In NeurlPS, 2019.
Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pp. 15-18. Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 18.18653/v1/N19 5004. URL https::///............tt..tt.ttt /www a el web. weeeb.org/aaee hot ogy /N1 5-000, 2
13

---

Under review as a conference paper at ICLR 2022
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252 2015. 6. 20
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1715-1725, 2016.
K. Soomro, A. Zamir, and M. Shah. 1cf101: A dataset of 101 human actions classes from videos in the wild. ArXiv, abs/1212.0400 2012a.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset 101 human actions classes from videos in the wild. arXiv preprini arXiv: 222204422 2012b. 1, 23
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In Proceedings of the IEEE international conference
computer vision, pp. 843 852, 2017. 6, 20
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF Interna- lional Conference on Computer Vision, pp. 7464-7473, 2014-997,,099 2
Chen Sun, Arsha Nagrani, Yonglong Tian, and Cordelia Schmid. Composable augmentation encod- ing for video representation learning, arXiv preprint arXiv:2104.006 2021. 3
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint /025. 2020. 1
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 6450-6459, 2018.
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, K. Kavukcuoglu, Oriol Vinyals, and A. Graves. Conditional image generation with pixelenn decoders. In NIPS, 2016. 2
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In NewrIPS, 2017. 2, 4, 15, 17
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is you need. In Advances in neural information processing systems, pp. 5998-6008, 2017. 5, 15
Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with vqvae. arXiv preprint arXiv:2103.01950, 2021. 2, 3, 15
Lei Wang, Piotr Koniusz, and Du Q Huynh. Hallucinating idt descriptors and i3d optical flow fea- tures for action recognition with cnns. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 8698-8708. 2019. 6
Donglai Wei, Joseph Lim, Andrew Zisserman, and William T Freeman. Learning and using the ar- row of time. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8052-8060, 2018.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv, 2016. 2
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vac and transformers. arXiv preprint arXiv:2104. 10157, 2021. 2
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5764, 2019.
14

---

Under review as a conference paper at ICLR 2022
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505-5514, 2018. 2
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. 7
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer-
ence on computer vision, pp. 649-666. Springer, 2016. 3
In this supplementary materials, we start with describing details of the model (Sec. A), pre-training (Sec. B), experiments (Sec. C), and dataset (Sec. D). We then provide additional analysis results in Sec. and visualization in Sec. G.
A MODEL ARCHITECTURE
As described in Sec. 3.4, a transformer model on top of the discrete video tokens generated by VQ-VAE. Since transformers have different variants, we here show details our architecture for clarity. The design of method largely follows the practice in BERT (Devlin et al., 2019), TimeSformer (Bertasius et al., 2021), Sparser Transformer (Child et al., 2019), ViVIT (Arnab et al., 2021), and MoCoV3 (Chen et al., 2021).
A.1 BACKBONE MODEL
Embedding, Given the video frames (fe E "'x t [T]}, we first use VQ-VAE (van den Oord et al., 2017; Ramesh et al., 2021) to discrete them into video tokens { e [V] te [7] i
[M] [W]3. We the specific VQ-VAE in DALL-E (Ramesh al., 2021) which trained trained on 250 million images from the Internet. Since the VQ-VAE encoder largely compresses 8x8x3 vector (ranging from 0-255) to an integer of 0-8191, 1, is considered with image prior. We do not use the VQVAE (Walker et al., 2021) method since the image- trained VQVAE has been pretrained on very large image corpus and as consequence cover much more diverse set of visual scenes and elements. We next use an embedding layer (embedding) that to map these discrete tokens to continuous vectors. Since transformer layers are permutation- invariant, we follow (Dosovitskiy et al., 2021; Devlin et al., 2019) to add positional information into the input. The positional embedding (pos) is factorized as a sum of the temporal embedding pos", the height embedding pos", and the width embedding pos". This factorization reduces the number of trainable parameters to encode positional information, which empirically shows a slightly better result. Finally, Layer-Normalizzan (Ba al., 2016) layer is added to get the initial hidden outputs hba..
compression method
hi.... = Layer (en + pos(t, i,))),
pos(t, i, = pos (t) + pis + pos"(j):
(8)
where we use the superscript to denote that the hidden outputs before the first transformer layer.
Attention Blocks. Before introducing the detailed model architecture, we first describe the basic building components: the attention block. An attention block built based on the attention operator (i.e., 'Attn") with residual connection. The attention operator takes a single query vector æ and its context {y:} as input. It first computes the attention score between x and each context vector yis then the attention scores are normalized by the softmax. Lastly, the output is a weighted- over all the context vectors (transferred by a 'value" matrix W Wratee):
Attn(æ, (vA)) Σ softmax, (Wquery x:) Waey Ve Waauue
value V4-
To compose the attention block from the previous attention operator, the residual connection and layer normalization (i.c., "Lrye'Norm") are added. We follow the original transformer model (Vaswani et al., 2017) that uses a post-layer-norm layout:
AttnBlock(z, (vi)) LayerNorm(z Won Attn(x, (v:1))
(10)
15

---

Under review as a conference paper at ICLR 2022
In order reduce computational cost and memory, we also adapt the attention block suggested in Sparse Transformer (Child et al., 2019) that takes two sets of context vectors {3r} and {2g} as input. This special attention block computes attention for the two context-vector sets separately and concatenates their output together. In our case, suppose {yi} and (2z) ) are the rows and columns of a square matrix, then reduces the computation cost of calculating attention scores from ((x*) to
Onn"), where n is the number of rows/columns:
AutuBlock(r, (r,
= LayerNorm(x + Won [Attn(x; (us)), Attn(z; (=)1)))
(II)
Spatio-Tempon Transformer Layer. The spatio-temporal transformer layer is composed with the previously-introdueed attention blocks and an additional MLP block. The /-th layer takes the output of the previous layer (177g) as input and outputs the hidden states (M,,,) We separate the attention into two attention blocks: the temporal attention block AttnBlock and the spatial Without loss of generality, we will use T and gsMce to denote
attention block the intermediate results from temporal and spatial attention blocks, respectively. First, the temporal attention block attends to the tokens at the same spatial location but in different frames (i.c., at different timesteps): (Mers e [T]}- Next, the spatial attention block attends to the tokens in the same frame: fol.s. (i,
[M] x [W]]. To reduce the computational cost, we incorporate the sparse attention block (Child et al., 2019) (detailed in the previous paragraph) that factorizes the attention over height and width: (gl i [f]] (ol.s. | [W]}. The MLP block has two fully-connected layers with GeLU (Hendrycks & Gimpel, 2016) activation in the middle. Overall, the formula of one spatio-temporal transformer layer is:
Pr,i.j
TIME
AttnBlockrua (h (h (,, TIME
Ite (TTH)
(12)
9t...j
SPACE
AatnBBlocssncc(@@.... (gi,i
i [
TIME jee [W]]
(13)
LayerNorm(
SPACE MLP(O
(14)
[CLS] Token. Following the practice in BERT (Devlin al., 2019) design, we add special [CLS] (abbreviation of 'classification") token and take its output as the representation of the whole sequence. We follow TimeSformer (Bertasius et al., 2021) to compute the its output: the [CLS] token attends over the context separately and then the outputs are averaged. We take the temporal attention layer as an example. Suppose h/1-1 is the [CLS] feature vector output by layer ( - 1, then the temporal attention layer do the following computation:
TIME Jets
ΣΣ HW
ALtnBlockraa (M (177) | Le (TJ)).
(15)
The other attention blocks process the [CLS] token similarly.
A.2 PRE-TRAINING AND FINE-TUNING HEADS
Pre- Pre-training or fine-tuming usually requires few additional modules (i.e.. heads) on top of the transformer layers that convert the output features to the desired probabilities or vectors. We next describe the heads used in our pre-training and fine-tuning process.
Token Head for Mask-then-Predict We first define the prediction head over the tokens follow- ing BERT(Devlin et al., 2019). It first processes the last-layer hidden outputs hf,,, using fully- connected layer (with GELU activation (Hendrycks & Gimpel, 2016) and layer normalization (Ba et al., 2016)):
Me,..j LayerNorm (GELU (inken (nkke(( (N()),)) + byoken))
(16)
In our maak-tthen-un then-ired method (Sec. 3.1), we will predict the masked tokens (i.c., the token before masking) from their context. We thus further convert this hidden vector into a distribution over the token vocabulary:
Peas(oras k) softmax, ( (Woord word Weii.j + bword].
(17)
The weight Wonnd is shared with input word embedding layer embedding (Press & Wolf, 2017; Devlin et al., 2019) while the bias bward is trained independently.
16

---

Under review as a conference paper at ICLR 2022
Table 9: Induced Masking ratio w.r.. different input resolutions and #masking blocks. The numbers of blocks/masking ratio for each resolution setting used in our experiments are shown in bold.
Input Resolution
#Masking Blocks
Lenghh
Frame Size
Token Map Size
4
5
7
5
128
11.9
14.5
17.0
19.4
21.7
5
256
32
10.6
13.1
15.2
17.5
19.5
10
128
16
10.4
12.8
15.0
17.1
19.2
10
256
32
9.3
11.4
13.4
15.4
17.2
Contrastive Learning Head Next we discuss the pre-training heads for contrastive learning. on top of the [CLS] hidden output hers- We encode the hidden state with MLP. We use batch normalization (loffe & Szegedy, 2015) inside the MLP head following the practice in (Chen et al., 2021).
fas MLP
(18)
This feus feature is used in computing the contrastive loss as in Sec. 3.2.
FC Layer for Fine-Tuning, When fine-tuning for action classification task, we add a fully- connected (FC) layer the [CLS] output has. We initialize its weight and bias zero
Special Tokens. Besides the V token types introduced in the vocabulary of the VQ- (see Sec. 3.1), add several special tokens into the 'vocabulary', namely [CLS] token is introduced as a stub for the whole-video representation, a [PAD] token is used when the actual clip length is less than the model's expected input length. For the mask-then- teask-then-predicee peennppeeicttte task, follow BERT (Devlin et al., 2019) to replace the masked tokens with a specific [MASK] token.
B PRE-TRAINING DETAILS
B. MASKING BLOCKS
As described in Sec. 3.1, we mask the tokens by blocks (a cube-shape set of tokens). To avoid masking all the tokens in the clip, we control the maximum block length for the time domain, height, and width. For spatial dimensions (i.e., height and width), the maximum length is half of the full length (e.g., the maximum block length will be 16 for a token map of length 32). For temporal dimension (i.c., the clip length), the maximum length will be 2/3 (round up) of the full length so that it allows long-range modeling. Under these constraints, we uniformly sample fixed number of mask blocks and take their union to construct the final mask. The number of blocks is decided by the induced masking ratio, which depends on the input resolutions. In Table 9, we show the induced masking ratio w.r.t. different input resolutions and #masking blocks. We take the VA-VAE (van den Oord et al., 2017) provided in DALL-E (Ramesh et al., 2021) that has a compression factor of 8, thus the length of the token map is always 1/8 of the frame size. For each input resolution, select the number of blocks (shown in bold in Table 9) whose induced masking ratio is closet to 15% following BERT (Devlin et al., 2019).
B.2 CONTRASTIVE LEARNING Loss
For completeness, we list the two losses used in contrastive learning here. The first loss for clip C; from video; is:
exp (f. f1/7)
LinFaNCA(() log Σx exp (f. fu/7) + Ee exp f6/7)
(19)
The symmetric loss CintsNNIs (i) for feature of the other clip sample de from video; (and its feature
f!) is:
exp f(TT)
CitoNCE(i) = log
(20)
Σx exp ( fw/t) + Exss exp (S. (1/7)
17

---

Under review as a conference paper at ICLR 2022
Table 10: Model Configuration. The 'Small' model is mainly used in the analysis (Sec. 5) while "Large-Half" model mainly used in the results (Sec. 4.3) for the final large-scale experiments. 'Vocab Size" is the number of token types in our model, defined by the pre-trained VQ-VAE model (Ramesh et al., 2021).
Small (in Sec. 5)
Base
Large-Half (in Sec. 4.3)
Layers
6
12
24
Dimensions
512
768
1024
Attention Heads
8
12
16
Attention Head Dim
64
64
32
MLP Intermediate Size
2048
3072
2048
Vocab Size
8192
8192
8192
Params
29.4M
119.7M
210.1M
Table 11: Training Hyperparameters. 'Pre-Train (Results)' our final model Sec. 4.3 that takes large-half model. 'Pre-Train (Analysis) is the pre-training in analysis (Sec. 5). *The batch size for pre-training the number of samples in updating the weights, Since we use gradient accumulation, is not correlated to the number of negative examples in contrastive learning.
Pre-Train (Results)
Pre- Pre-Train Train (Analysis)
SSV2
Diving48
UCPI0L HMDB51
Kinetics-400
Optinization
Number of Epochs
100
22
50
50
50
30
Number Updates
120K
29K
5.8K
3.7K
1.4K
48K
Learning Rate
3c-4
Ic-3
1c-4 for small/basc model, So-5 for large-half model
Warm-UUp Ratio
0.05
0.1
LR Decay
Lincar 0.1
Lincar
Backbone Dropout
0.1
Lost PC Dropout
0.0
Optimizer
Adam
Adami
Batch Size
1024*
128
Weight-Decay
0.05
0.01
Adam Betal
0.9
0.9
Adam Beta2
0.999
Adam Epsilon
Ic-8
Ic-8
Grad- Crad-Clipping Norm
Augmentation
Color Distutiion/Gray Scale
No (Prame Sixe) (Center) (Random Uniform)
No
Training Sparial Resize
(Frame Size, Frame Size 1.25)
Training Spatial Crops
3 (Top Left, Center. Bottom Right) (Random Uniform)
Training Temporal Crops
Inference Sputial Resize
(Frame Size)
(Frame Size) (Uniform)
Inforence Temporal Crops
(Random Uniform)
Training Spatial Flip
No (Cente))
Yes
Inforence Spatial Crops
(Center)
3 (Top-Tefft Center, Rottom-Right)
18

---

Under review as a conference paper at ICLR 2022
C EXPERIMENT DETAILS
In this section, we show our model configuration and training hyperparameters in details to support the reproducibility of our experiments.
Model Configuration. Our model configuration details is shown in Table 10. Most analysis results (Sec. 5) take 'Smalll models and our final results (Sec. 4.3) take 'Large-Half" model. Other models are used in Sec. 5.1. The final 'Large-Hall model halves the attention head dimension and MLP intermediate size in (Child al., 2019). For the pre-training heads, we follow BERT to take the intermediate dimension of the token-head to be the same as the hackbone's hidden dimension. For the CLS head, we take 3 layers in MLP and 4096 intermediate dimensions. The output dimension is 256. We test with different number of layers and hidden dimensions of CLS head and generally find that larger head gives better results (as in (Qian al., 2021; Chen et al., 2021)). This CLS head contributes to about 1% pre-traie vre-training computational cost overhead.
Training Hy pyppaaaraetees. We list the training hyperparameters in Table 11. Most of the hyper- parameters are inherited from previous works to allow fair comparison and reduce tuning effort. For optimizer hyperparameters, we mostly follow the implementation of DALL-E (Ramesh et al., 2021) and BERT (Devlin et al., 2019). SSV2 follows the epoch number in (Feichtenhofer et al., 2019) and (Feichtenhofer et al., 2021). To reduce the computational cost, we pre-extract the VQ-VAE tokens thus we employ a fixed set of spatial data augmentations. As listed in the bottom of Table 11, we exclude any color distortion and gray scale augmentation. We resize the video clip to the desired frame size and center-crop during pre-training. For downstream tasks, resize the video clip to frame size or 1.5 times of the frame size, then crop the clip (with frame-size by frame-size spatial size) from the top-left, center, and bottom-right. We apply (horizontal) flip to the raw frames thus total of 12 spatial augmentations are extracted (12 2 resize x 3 crops x 3 flip/no-fipp. The only exception is SSV2. This dataset needs distinguish left/right motions thus we exclude the flip aug- mentation and only use the center crop during inference following previous works (Feichtenhofer al., 2019; 2021). During pre-training, we always accumulate the gradient to a batch size of 1024 before updating the weights but use different numbers of negative examples. We analyze this effect in Sec. 5.2.3.
Following previous works (Feichtenhofer et al., 2019), we increase the training epochs for the non- pre-training models by 4x for small datasets (Diving48, UCF101, HMDB51) and 1.5x for larger datasets (SSV2, Kinetics-400).
When analyzing the mask-then-predich task in Sec. 5.2.2 (and all other analysis focusing on mask- then-predict), we exclude the contrastive learning loss (by setting loss weight a=0) to preclude potential side effects. However, we still use masked prediction loss when assessing the contrastive learning task in Sec. 5.2.3 as we observe very low performance with only contrastive learning ob- jective.
Pre-Training with Kinetics datasets. Besides pre- pre-training on the HowTo100M (Miech et al., 2019) dataset, we have an experiment with smaller models (i.c., base model in Table 10) ppe-ttaiieed on K600 dataset. We found that the pure mask-prediction task on K600 (Carreira al., 2019) can reach 88% on UCF101, but adding the contrastive learning task does not show significant further improvement (+0.5% according to the hyperparameters and seeds). This is possibly due to K600 dataset is designed specifically for action recognition on shorter video-clip where the average length of the video is 10 second. This short-range video would limit the success of the masked prediction and contrastive learning combination (as illustrated in Table 6 on HowTo100M dataset, dmux= = 10s). Given these observations, we did not run the full model (large model in Table 10) on K600 because of the budget constraint.
D DATASET DETAILS
In Table 12, we list the key statistics of the datasets used in our paper. HowTo100M is our pre- training datasets that has long- long-duration uncurated videos. The videos are collected from YouTube by searching key phrases thus the scale could be easily increased. SSV2 and Kinetics-400 are two large downstream datasets, where SSV2 focuses more on the actions and Kinetics-400 focuses more
19

---

Under review as a conference paper at ICLR 2022
Table 12: Key statisties of video datasets used in this paper. HowTol00M, is used for pre-training while others are downstream datasets. The number of training/validation examples in HMDB51 and UCF101 are reported for the train-val split I.
HowTo100M
SSV2
Diving48
UCF101
HMDB51
Kinetics-400
Training Validation
1238791
168913
15027
9537
3570
205418
24777
1970
3783
1530
17686
Number Classes
174
48
101
400
Average Video Duration
6.5min
48
68
7s
4s
10s
Table 13: Results of different attention-module layouts and layer-normalization positions.
'Speedd is the normalized pre-training speed (i.e., number of samples GPU / second). Models are pre-trained on HowTo 1000 for epochs. The result numbers represent UCF101 accuracy.
Params
Speed
Pre-LayerNorm Post-LayerNorm
TxHxW
23.
12.6
65.9
T.HxW (Bertasius et al., 2021)
27.1M
20.0
69.0
T.H.W
35.8M
26.4
69.0
69.6
T.H- (ours)
29.4M
32.0
67.6
69.4
on the scenes. Diving48, UCFI0I, HMDB51 are three small datasets. Different from previous datasets on classifying different action types (thus might be potentially inferred from single frames), Diving48 studies the three stages (takeoff, flight, and entry) of competitive diving. Thus achieving good results on Diving48 requires an understanding of the whole video clip. The license of each dataset allows academic use. SSV2 uses the 'Free Academic License'. HMDB51 is licensed under 'Creative Commons Attribution 4.0 International License'. 'The kinetics dataset is licensed by Google Inc. under Creative Commons Attribution 4.0 International License." We use the YouTube videos under the Fair Use.
E COMPUTATIONAL COST
The pre-training takes around 8.9K V100 GPU hours. This computational cost is at the same level as ViT (Dosovitskiy et al., 2021) supervised training (5.5K hours on ImageNet-21KK (Russakovsky et al., 2015) and 12.8K on JFT(Sun et al., 2017)). It is also at the same level of supervised training a model on Kinetics-400 dataset (6.4K for SlowFast (Feichtenhofer et al., 2019), about 5.6K for TimeSformer-L (Bertasius et al., 2021)). For fine-tuning, SSV2, Diving48, UCF101, HMDB51, and Kinetics-400 take 1K, 200, 150, 40, 2K GPU hours, respectively. For analysis, the pre-training takes about 160 GPU hours. Besides the final model training. energy is also spent tuning the model and finding the best configuration. As shown Sec. 5.2.3, our method is more robust to the hyperparameters.
F ADDITIONAL ANALYSIS RESULTS
F.1 MODEL ARCHITECTURE COMPARISONS
Attention Layouts. We here compare different alternative model architectures in Table 13. We first experiment with different attention layouts discussed in Sec. 3.4. We consider the sparse at- tention as proposed (Child et al., 2019) and the sequential attention blocks as in (Bertasius al., 2021). The 'xHx model is the basic attention module that takes the flattened tokens as input (of shape H x W). At each layer, each token attends to all other tokens. The 'T.HxW' model sepa- rates the temporal attention and spatial attention ('Divided Space-Time" in (Bertasius et al., 2021)). The 'T.H.W" model processes three attention sequentially ('Axial Attention' in (Bertasius et al., 2021)). The 'T, H W" model is our default model that sequentially conduct temporal attention and spatial attention, where the spatial attention are parallel into the height attention and width attention. As shown in Table 13, 'T.H- W' reaches balance between speed and accuracy.
20

---

Under review as a conference paper at ICLR 2022
Table 14: Impact of input resolutions T and S. 'Mauk-Accu'' and 'CL-Loss' are the pre- training metrics. 'UCF101' indicates the UCFI0I fine-tuning results with the pre-training reso- lution. UUCF101-FulI-ee Reso- indicates the full-resolution fine-tuning with T=10 and S=256.
lframes Frame Size
Params
Pre-train Speed
Mask-Aceu.
CL-Loss]
UCF1011
UCF101-FullResu..
128
29.4M
32.0
17.2
1.06
69.4
73.8
128
29.4M
16.5
17.2
0,96
74.2
74.6
5
256
29.4M
8.4
10.8
0.93
72.9
75.7
10
256
29.4M
4.4
10.6
0.85
78.1
78.1
Table 15: Impact of masking ratio. All models are pre-trained with only mask then- teas--then-preiect task. #Blocks is the number of masking blocks. Default setup is underlined.
Strategy
#Blocks
Ratio
Mask-Accu.
UCF101
Block
4
11.9%
17.9
66.8
Block
5
14.5%
17.6
68.3
Block
6
17.0%
17.3
67.3
i.i.d.
11.9%
25.6
64.5
i.i.d.
14.5%
24.3
63.5
i.i.d.
17.0%
24.0
64.0
Pre-Layer-Normalizaton vs. Por-maaye-oormalizztionn. Besides the architectures listed above, we also consider the pre-layer-norm (used in GPT and ViT) and post-layer-norm (used in BERT) variation. We empirically find that post-layer-norm architecture is better for our pre-training tasks as shown Table 13 (comparing the last 2 columns).
F.2 INPUT RESOLUTION
In Table 14, we show model scalability over input resolution (i.e., #frames T and frame size S). With the same frame size S, longer clips perform better than shorter clips (c.g.. T==00, $=128 is better than 7=5,, $=128). With the same number of input frames T. larger frame size improves the performance (e.g., T=10, S=256 is better than T=10, $=128). For each pre-training resolution, we also try to fine-tune under full-resolution with T=10, S=256 (denoted UUCF101-Full-Rest in pre-training, fine-tuning with larger resolution generally improves the results. Although longer and smaller clips (7=10, S=128) show better results than shorter and larger clips (T=5,, S=256) when using the same pre-training and fine-tuning resolutions. they show different trends with the full-resolution fine-tuning. Increasing frame size during fine-tuning (the second block in Table 14) only improves the UCF101 result by 0.4, while increasing the clip length (the third block) improves the UCF101 result by 3.8. These results call for a need of pre-training with large spatial size, and follow this practice in our large-scale experiments in Sec. 4.3.
As
F.3 NOISY MASKING FOR MASK-THEN PHEEICP
Our default masking strategy replaces all masked tokens with special [MASK] symbol. We also experiment with BERT's masking strategy that only replaces 80% masked tokens to the MASK symbol. For other tokens, 10% randomly sampled from the 'vocabulary' and 10% are kept the same. For smaller experiments, the two masking strategies show similar results. However, this BERT's noisy masking strategy has lower convergence stability the larger model pre-training. The pre-training diverges after about 10 epochs (out of the 100 epochs).
F.4 MASKING RATIO FOR BLOCK-MASKING AND I.I.. MASKING
We test the effect of different masking ratios. the main text, we control the number of blocks for block masking. In Table 15, here also show the results of matched masking ratio for i..d. masking for completeness. Empirically, the result differences among various masking ratios are marginal and the original BERT's 15% masking ratio (with roughly 5 masking blocks) works slightly
21

---

Under review as a conference paper at ICLR 2022
Table 16: Impact of contrastive learning loss weight a. Default setup underlined.
Mask-Accu.
CL-Loss!
UCF1011
0.0
17.6
68.3
0.5
17.5
1.07
70.2
1.0
17.2
1.06
69.4
2.0
16.9
1.05
68.0
x
1.07
57.1
Temporally-Heavy Datasets
Dataset: SSV2, Action: 'Open the bottle cap"
(b) Dataset: Diving48, Action: 'Forward', '15som', 'NoTwis', 'PIKE"
Spatially-Heavy Datasets
(c) Dataset: UCF101, Action: "ApplyEyeMakeup"
(c) HMDB51 Action: 'Ride_ Horse"
(d) Dataset: Kinetics400,
Action: "PettingCat"
Figure 3: Data samples from temporally-hese heavy and spatially- -eeavy datasets. While temporally- heavy datasets need the temporal information to make decisions, most actions in spatially-hear datasets could be inferred from just single frame.
better. Thus always select the number of mask blocks whose induced masking ratio is closest 15%. For all masking ratios, block masking shows significantly better results than the i.i.d. masking.
F.5 IMPACT OF CONTRASTIVE LEARNING LOss WEIGHT
In Table 16, show the impact of loss weight (see Sec. 3.3). Since the loss have been calibrated by multiplying the temperature, a=1 shows stable results and o=0.5 is slightly better. Setting a=2.0 will let the model to focus mostly contrastive learning task and its result is worse than pure mask- then-predict pre-training (i.c., o=0.0). We also list the pure contrastive learning pre-training results here (denoted a= but excludes the mask-then-predict loss and set o=1.0) for reference.
22

---

Under review as a conference paper at ICLR 2022
(a) Original Image
(b) 15% masked. (c) 15% masked, (d) 45% masked,
(e) 45% masked. spatio-temporal neighbours
spatio-terpooal spalial neighbours spatial neighbours
neighbours
1. Block Masking
2. L.I.D. Masking
Figure 4: Nearest-neighhour reconstruction of block masking and i.i.d masking. We mask tokens at different ratios and reconstruct them by simply copying their spatial spatio-temporal neighbours. Even under heavy masking (e.g., 45% masked), this simple reconstruction strategy still yields a reasonable results for i.i.d masking, e.g., reconstructed images, especially the one reconstructed from spatio-temporal neighbours. However, this becomes significantly more difficult when using block masking.
easily recognize the action 'petting cat' from the
G VISUALIZATIONS
G.1 TEMPORALLY-HEAVY VS. SPATIALLY-HEAVY DATASETS
We illustrate the differences between temporally -havy and spatially-heaw datasets in Fig. We here show do not cherry-pick the data but aim for showing the nature of each dataset. Overall, understanding in temporally-heay datasets needs temporal modeling. whereas the action labels of spatially-heavy datasets could be inferred from single frame. To understand the SSV2(Goyal et al., 2017) exam- ple in Fig. 3.(a), the model needs to understand the causality, i.e., the order of the frames decides the action label. In Fig. 3.(b), the competitive diving dataset Diving48 (Li et al., 2018) also re- quires considering nearly all frames make the decision. However, for the spatially-hean (UCF101 (Soomro et al., 2012b), HMDB51 (Kuehne et al., 201 1b), Kinetics-400 (Carreira & Zisser- man, 2017)), the action label could be inferred from any single sampled frame. These observations result in the pretty high frame-level accuracy (i.e., not modeling temporal interactions) in Sec. 4.3.
frames from the video and the label of the video clip. Note that we
datasets
G.2 MASKING STRATEGY COMPARISONS
We propose block masking (in Sec. 3.1) since i.i.d. masking may lead trivial solutions for the ter---hhe--rrediee task given the strong localities in videos. We illustrate this point in Fig. with a simple copy-paste reconstruction method. Specifically, after masking, we first replace the masked tokens with their nearest visible neighbours (i.e., the unmasked token that has the short- est distance spatial or spatio-temporal domain), and then forward the reconstructed tokens the VQ- (QEV decoder to generate the RGB images. For the default 15% masking ratio, i.i.d. masking recoverable while block masking causes striped noisy patterns." case of masking 45% tokens (in Fig. 4 (d), (e)). The block-masked images are hard to reconstruct, however, some objects reconstructed images from i.idd masking are still recognizable. When comparing images under the same masking strategy, recovered images using ppatio-tempont neigh- bours is better than using only spatial neighbours, especially when comparing the images under 45% i...d. masking (i.e., (d).2 and (e).2 in Fig. 4). Overall, these results indicate that using i.i.d. mask- ing in mask-then- mask-then-predict predict task has potential trivial solution by copying the neighbourhood, while block-masking resolves this issue.
also test with the extreme
'We highlight the masking region in Fig. 4(b) and show the raw RGB images in other cases.
23

---

Under review as a conference paper at ICLR 2022
(a)SSV2 Dataset
1. Original Frames
2. After VQ-VAE Compression
3. Masked Area
4. Model Reconsiruction
(b) Kinetics 400 Dataset
Original Frames
VO-VAE
Compression
3. Masked Area
4. Model Reconstruction
Figure 5: Masked-token model reconstruction for SSV2 and Kinetics-4PP 4400 datasets. Comparing 1. and 4., our model could redraw regions.
and spatially-plausible -lauuibble patches for the masked
G.3 MODEL RECONSTRUCTION
Since model is trained with musk-then-predicc task, it is able to reconstruct masked tokens. In this section, we showcase the reconstructed video frames by our final model (i.e., 24 layers, 1024 dimensions, clip length, and 256 frame size). As shown in Fig. 5, provide two examples from the SSV2 and Kinetics-400 dataset. We uniformly sample consecutive frames from the video at frame per second. We show the original frames in the first rows (Fig. 5.(a).1, (b).1). As illustrated in Sec. G.I, the eemporally-eaay SSV2 dataset has object motions between frames while the spatially- heavy Kinetics-400 dataset has almost static frames. In the second rows, show the images after VQ-VAE compression. To do this, we first use VQ-VAE encoder to encode the images, and then use VQ- DQ-V decoder to reconstruct the images, without any corruptions between. We see that there some information loss caused by the VAE compression (e.g., the text 'comfort' in Fig. 1.) potentially contributes to relative lower results on spatially-heavy datasets. the third and fourth rows, illustrate the masked area and the prediction from our model. As shown in Fig. 5.(a).4, our model could faithfully redraw the shape and texture the object. As shown in Fig. 5.(b).4, the shape of the cat's 's head is pretty similar to the original frames while the shading is different (but still consistent in different frames).
24


Reviews:
[{"review_id": "xzRQGYHcvXe", "reviewer": "Reviewer_PC2U", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## **Strengths**\n* VIMPAC achieves strong empirical results on temporally-heavy datasets.\n* Although the mask token prediction task has been proven to be an effective pretraining objective for image tasks, the block masking method designed by authors is well motivated.\n* Tokenizing all videos without spatial augmentation is a good way to reduce storage and io cost.\n\n## **Weaknesses**\n* This paper is utilizing a transformer as the main architecture, with a pretrained VQ-VAE as the tokenizer. I think it's great to have the baselines with the same arch for fair comparison. Bert-large is a relatively large model comparing to previous CNN models, and large models tend to work better in pretraining with big datasets.\n* The contrastive learning objective part doesn't make a lot of sense to me. It seems adding contrastive learning objective only provides marginal improvement, and might even hurt the performance on temporally-heavy datasets. Ablation studies on contrastive learning also lead to some counter-intuitive conclusions so I would expect more analysis and insights from this part.\n* I'm also interested if VIMPAC can work for tasks beyond action classification (might be hard due to the architecture design).", "summary_of_the_paper": "This paper proposed a new pre-training method named VIMPAC for video understanding, which is a combination of specially-designed masked token prediction objective and contrastive learning objective. VIMPAC is well-motivated and shows strong empirical results on several video understanding tasks.", "summary_of_the_review": "With strengths and weaknesses I listed in previous section, I think this paper makes some reasonable contributions but is marginally below the iclr acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ChwBji79v6n", "reviewer": "Reviewer_hrwa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n1. The proposed model is a fresh approach to video modeling and performs well for temporal reasoning tasks like SSv2. The proposed pre-training methods leverage ideas from BERT and shows it can be effective for computer vision tasks too (similar to BEIT). \n\n2. The ablations are interesting, eg table 1 which shows pre-training is imperative for a model like this and lends significant gains.\n\n## Weaknesses\n\n1. Missing/incorrect details\n- The paper in Table 1 compares their method to prior work and notes the pre-training dataset used. Authors report their method only uses \"HT\" for pretraining. However, the tokenizer is using a pre-trained DALL-E model, the training data for which should also be mentioned here? Given this is a paper about learning from unlabeled data, it is important authors are very explicit about *all* the data being used.\n\n2. Continuing from 1), it is unfortunate that no results in Table 1 are comparable to each other; authors use a different pre-training dataset (HT+DALL-E), model architecture (DALL-E + Transformer) and modalities (video only) which makes it impossible to perform any apples-to-apples comparisons. The standard setting in Feichtenhofer et al (2020) seems to be to perform all comparisons with unlabeled Kinetics dataset as pre-training. It would have been ideal if authors kept at least some of the axes of variation fixed. In absence of that, it is only fair to compare the proposed method to state of the art. In that regard it struggles on spatially heavy datasets, and on temporal datasets it performs better although is comparable to more recent SOTA (eg MVIT -- 68.7 without large scale pretraining or HowTo100M pretraining, though it does use K600 labels). \n\n3. The architecture seems quite constrained as it relies on tokenization which looses spatial information. As authors themselves point out, this is likely the reason it struggles on spatial datasets. However, this also means this limitation would limit the impact of the model as it might be very useful for spatial tasks (video detection/segmentation etc). Hence, even though it might be good at learning representations, it is limited to classification. Using a BEIT style framework (as authors also recommend for future work) would make this paper much stronger in that regard.\n\n4. Missing related work: Authors should cite MVIT and other recent transformer based video models. Also while it is not technically published work, I would encourage the authors to cite and compare with video swin transformer.", "summary_of_the_paper": "The paper proposes a few new techniques: 1) A new video modeling architecture that uses a pre-trained VQ-VAE to tokenize frames, followed by a transformer encoder that aggregates the features and produces the final action class label. 2) Pre-training such an architecture using self-supervision by a) Masked prediction: Authors mask out blocks of tokens and predict them using the context (akin to BERT) and b) contrastive learning: Authors use the representation for two clips from same video as a positive match and otherwise negative match. The final model is trained with a linear combination of the masked prediction and contrastive losses, and finally finetuned for downstream tasks. The model is pretrained on HowTo100M dataset, and finetuned on multiple downstream datasets, where it obtains gains on more temporal datasets like SS-v2.", "summary_of_the_review": "The proposed model is interesting and in-line with recent work like BEIT etc, and obtains some decent results. However I find it hard to draw meaningful conclusions from the paper due to lack of proper apples-to-apples comparisons. Moreover, given the architecture is likely to have limited impact in the field (as I discuss in my weaknesses), I am borderline on this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9EzK2uXub_", "reviewer": "Reviewer_E8qx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ the paper is well written, main ideas are clear enough, it is also easy to follow for readers.\n+ the proposed block masking scheme makes sense in terms of avoiding information leakage from nearby tokens and it is also validated to be quite better than i.i.d.\n+ extensive experiments are carried out\n\nWeakness:\n- novelty: \n（1）Combining masked token prediction (MP) and contrastive learning (CL) is quite straightforward. Both MP and CL are studied previously, simpling combining these techniques is kind of incremental. Yes, the proposed block masking strategy is a plus, but it will have problem by directly combining it to CL. In my opinion, global information loss will be caused by block masking, it is actually harmful to the CL task. If better combination mechanism of the two tasks can be invented, it will be a good work.\n  (2) Discrete video token generation is also off-the-shelf technique. VQ-VAE is not new.\n\n- performance and evaluation: \n  (1) the performances on \"spatially-heavy\" datasets, such as UCF101, K400 and HMDB51, are far from the state-of-the-arts, rather than comparable. Such results show that the proposed pretraining method is only good at \"local\" modeling, its global representation is not strong enough. This may not only be caused by the discretization of VQ-VAE, but also has something to do with the harmful blocking masking which can harm the CL.\n (2)  Now that VQ-VAE can be harmful to the spatial information, why it should still be leveraged. There is also no evaluation on this part. What if VQ-VAE is replaced by other feature encoder and we regress the masked contiguous video feature points? Why not using video frame patches as discrete input tokens? ", "summary_of_the_paper": "This paper proposes a new video pretraining method by combining masked token prediction and contrastive learning. Off-she-shelf VQ-VAE is used in this paper for discrete video tokens generation. In order to make the masked token prediction more effective, the authors proposed a block mask strategy where spatial-temporal neighboring video tokens are masked together. Experimental results are shown to validate its effectiveness.  ", "summary_of_the_review": "Given the aforementioned weakness of this paper, I think the novelty of this paper is currently limited, and its performance is not good.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: None of the reviewers recommended this paper. There were concerns that it is hard to draw meaningful conclusions from the experimental work due to the comparisons provided.  While the design of the block masking + contrastive learning proposed in this paper was rated as potentially being quite important, there remained some concern that subsequent tokenization steps could be problematic for "spatial heavy" datasets.

The AC recommends rejection.
Decision: Reject