Masked Autoencoders As Spatiotemporal Learners
Authors: Christoph Feichtenhofer
Haoqi Fan
Yanghao Li
Kaiming He
Venue: NeurIPS 2022 Accept (2022)

Abstract: This paper studies a conceptually simple extension of Masked Autoencoders (MAE) to spatiotemporal representation learning from videos. We randomly mask out spacetime patches in videos and learn an autoencoder to reconstruct them in pixels. Interestingly, we show that our MAE method can learn strong representations with almost no inductive bias on spacetime (only except for patch and positional embeddings), and spacetime-agnostic random masking performs the best. We observe that the optimal masking ratio is as high as 90% (vs. 75% on images), supporting the hypothesis that this ratio is related to information redundancy of the data. A high masking ratio leads to a large speedup, e.g., > 4x in wall-clock time or even more. We report competitive results on several challenging video datasets using vanilla Vision Transformers. We observe that MAE can outperform supervised pre-training by large margins. We further report encouraging results of training on real-world, uncurated Instagram data. Our study suggests that the general framework of masked autoencoding (BERT, MAE, etc.) can be a unified methodology for representation learning with minimal domain knowledge.

Masked Autoencoders As Spatiotemporal Learners
Christoph Feichtenhofer'
Haoqi Fan"
Yanghao Li Kaiming He
Meta AI, FAIR
Abstract
This paper studies a conceptually simple extension of Masked Autoencoders (MAE) [31] spatiotemporal representation learning from videos. We randomly mask out spacetime patches in videos and learn an autoencoder to reconstruct them in pixels. Interestingly, we show that our MAE method can learn strong representations with almost no inductive bias on spacetime (only except for patch and positional embeddings), and spacetime-agnostic random masking performs the best. We observe that the optimal masking ratio is as high as 90% (vs. 75% on images [31]), supporting the hypothesis that this ratio is related to information redundancy of the data. A high masking ratio leads to large speedup, e.g., > xx in wall-clock time or even more. We report competitive results on several challenging video datasets using vanilla Vision Transformers [18]. We observe that MAE can outperform supervised pre-training by large margins. We further report encouraging results of training real- world, uncurated Instagram data. Our study suggests that the general framework of masked autoencoding (BERT [15], MAE [31], etc.) can be a unified methodology for representation learning with minimal domain knowledge.
Introduction
The deep learning community is experiencing trend of unifying methodologies for solving problems in different areas, such as language, vision, speech, and more. For architectures, Transformers [67] have been successfully introduced into computer vision [18] and established
general building block in both language and vision. For self-supervised representation learning, the denoising/masked autoencoding methodology [68] in BERT [15] has been shown effective on learning visual represen- tations from images [31]. Towards unifying methodologies, less domain knowledge ("fewer inductive biases" [18] introduced for specific problem, which urges the models learn useful knowledge almost purely from data.
Following this philosophy, we study extending Masked Autoencoders (MAH) [31] the problem of spatiotemporal representation learning. Our method is simple: we randomly mask out spacetime patches in videos and learn autoencoder to reconstruct them (Fig. 1). Our method has minimal domain knowledge: the only spacetime-specific inductive bias is on embedding the patches and their positions; all other components are agnostic to the spacetime nature of the problem. In particular, our encoder and decoder are both vanilla Vision Transformers [18] with no factorization or hierarchy, and our random mask sampling is agnostic the spacetime structures. Our method predicts pixel values and uses no extra problem-spectie set of spacetime patches. Despite minimal inductive biases, our method achieves strong empirical results, suggesting that useful knowledge can be learned from data.
tokenizer. In nutshell, our method is simply MAE applied to the
It is hypothesized in [31] that the masking ratio (i.e., percentage of removed tokens) in masked autoencoding methods is related to the information redundancy of the problems. For example, natural images are more information-redudda.. (e.g., than BERT [15]). Our observations on video data support this hypothesis. We find that the optimal masking ratio of MAE is 90% for videos (Fig. 2), higher than the masking ratio of 75% for its image counterpart [31]. This can be understood as a consequence of natural video being correlated.
than languages and thus the optimal masking ratio higher
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

encoder
W
W
input
target
Figure I: Masked Autoencoders as spatiotemporal learners. We mask large subset (e.g., 90%) of random patches in spacetime. An encoder operates the set of visible patches. A small decoder then processes the full set of encoded patches and mask tokens to reconstruct the input. Except for patch and positional embeddings, neither the encoder, the decoder, nor the masking strategy, has any spatiotemporal inductive bias.
To the extreme, if a video has T identical static frames, randomly sampling 1/r of all spacetime patches would reveal most of the static frame. Because slow motion is more likely than fast motion in natural videos, the masking ratio can be very high as we observe empirically.
The higher masking ratio leads to more efficient solution in practice. Following the MAE in [31] that applies the encoder only on visible tokens, masking ratio of 90% reduces the encoder time and memory complexity to <1/10. Put together with small decoder [31], the MAE pre-training can achieve a theoretically 7.7 reduction in computation vs. encoding all tokens. In fact, the computation reduction is large that the data loading time becomes new bottleneck; even so, record x wall-clock speedup. Such a significant speedup is of great importance for video research that is large-scale and time-consuming.
We report strong results on variety of video recognition datasets. Our MAE pre-training greatly improves generalization performance: Kinetics-400 [35], it increases the accuracy of ViT-Large [18] by absolute 13% vs. training from scratch, while it takes less wall-clock training time overall (pre-training plus fine-tuning). Our MAE pre-training can outperform its supervised pre-training counterpart by big margins. Using vanilla VIT [18], our method achieves competitive results with previous tate-of-the-art art methods that incorporate more domain knowledge. We also report encourag- ing results using MAE pre-trained on million random, uncurated Instagram videos. These results suggest that supervuee learning videos can be tackled in way similar to its counterparts language [15] and images [31], under a unified framework.
Related Work
Denoising autoencoders. Denoising autoencoders (DAE) [68, 69] present general methodology for learning representations by reconstructing clean signals from corrupted inputs. Masking type of noise dates back to least a decade ago [69]. One of its most successful developments BERT [15), which is conceptually masked autoencoding on language tokens.
Denoising/masked autoencoding methods for computer vision have been making continuous progress [50, 9, 18, 31]. A series of recent methods are based on Transformer architectures [67] and are towards a unified solution between vision and language. iGPT [9] pioneers this direction by training Transformers on pixels as tokens. The VIT paper [18] makes revolutionary step forward by using patches as tokens. not only establishes strong Transformer architectures for vision tasks, but also explores masked prediction with patches. MAE [31] returns to the basics of the autoencoding concept [68] and draws attention to the decoding aspect. The presence a meaningful decoder provides more flexibility, e.g., enabling the encoder to operate only on visible patches and leading to more efficient solution. empirically shows that a high masking ratio is essential for image tasks [31]. Our study follows this line of research.

---

Visualizations on the Kinetics-400 [35] validation set (masking ratio 90 %).. We show the original video (top), masked video (middle), and MAE output (bottom) for each sample. This model reconstructs the original pixels. The video size is 16x224x224 and the spacetime patch size is 2 16 16 (the temporal patch size of is not visualized here). Each sample has 8x 144 14=1568 tokens with 156 being visible. For better visualizations, the known patches in the output are from the original input. Fig. shows more examples.
Figure 2:
Figure 3: Visualizations of the same pre-trained model in Fig. 2 but with a masking ratio of 95 %.
Instead of predicting pixels [9, 18, 31, 80]. another line research focuses on the tokenization of the prediction targets [3, 17, 77]. BEiT [3] proposes to pre-trained dVAE [47, 55] as the reconstruction target. The dVAE tokenizer can improved by perceptual or adversarial losses [17]. MaskFeat [77] shows that HoG [13] as prediction targets performs strongly.
Self-supervised learning on videos. The presence of the temporal dimension focus of self- supervised learning on video data. Related topics include temporal coherence ('slowness') [79, 25], future prediction [61, 72, 70, 45, 44, 71, 16], object motion [1, 75, 49, 76], temporal ordering [46, 23, 38, 78, 81]. spatiotemporal contrast [58, 62, 30, 22, 51, 56], etc.
Our method also relies on the temporal coherence of videos, but it approaches this goal implicitly. In fact, as our method is largely agnostic to spacetime, the main opportunity for it to make use of the temporal coherence is higher masking ratio (e.g., 90%), which assumes that videos are more informaation-rdduddan than images.
3

---

(a) agnostic. 90% (b) space-only, 90%
(c) time-only, 75%
(d) block, 75%
Figure 4: Mask sampling. (a): Random sampling that is spacetime-agnexiic. (b): Space-only random sampling, broadcasted to all time steps ("tube" masking [77]). (c): Time-only random sampling, broadcasted to all spatial locations ("frame" masking [77D. (d): Block-wise sampling [3] in spacetime, removing large regions ("cube" masking [77]). In this illustration, TxHxW is 8x 14x14;; green tokens are kept and others are masked out.
There has been growing interest in masking- methods for self-supervised learning on videos. Previous works focus on tokenizing the prediction targets for the use of videos [65, 73, 77]. Our autoencoding method operates on pixels, which is simpler and requires no extra data or domain knowledge on the tokenizer. Importantly, our method greatly improves the efficiency of learning. The practical speedup is of central importance for video-related research, which is in general larger-scale and more time-consuming.
Our work is done independently and concurrently with [66] on related method.
3 Method
Our method is a simple extension of MAE [31] to spacetime data (Fig. 1). Our goal is to develop the method under a general and unified framework, with as little domain knowledge as possible.
Patch embedding. Following the original VIT [18], given video clip, we divide it into regular grid of non-overlapping patches in spacetime [4, 2, 19, 77]. The patches are flattened and embedded by linear projection [18]. Positional embeddings [67] are added to the embedded patches. The patch and positional embedding process is the only process that is spacetime-aware
Masking. We sample random patches without replacement from the set of embedded patches. This random sampling is agnostic to the spacetime structure (Fig. 4 (a)). This structure-agnostic sampling strategy is analogous to that of BERT in ID [15] and MAE in 2D [31].
It is hypothesized in [31] that the optimal masking ratio is related to the information redundancy of the data. With unstructured random masking, BERT [15] uses a masking ratio of 15% for language and MAE [31] uses ratio of 75% for images, suggesting that images are more information-redumm redundant than language. Our empirical results on videos support this hypothesis. The optimal masking ratio we observe is 90%. This is in line with the common assumption that natural videos are more information- redundant than images because of temporal coherence. Fig. 2 and 3 present our MAE reconstruction results on unseen validation data with masking ratio of 90% and 95%.
The spacetime-agnun agnostic sampling can be more effective than struture-awaaee aware sampling strategies, e.g., space-only, time-only, or block-wise sampling (Fig. (b-d)). As neighboring patches in space or in time (Fig. 4(b, c)) are coherent, with a very high masking ratio, space-only or time-only sampling may retain less information and yield an overly difficult pre-training task. For example, time-only sampling from 8 frames with a masking ratio of 87.5% means keeping only a single frame, which presents an overly challenging task of predicting the future and past given only one frame. We observe that optimal masking ratios for structure-aware sampling are in general lower. In contrast, the spacetime-agnostic sampling better utilizes the limited number of visible patches and thus allows to use higher masking ratio.
Autoencoding. Our encoder is a vanilla ViT [18] applied only on the visible set of embedded patches, following [31]. This design greatly reduces time and memory complexity and leads to more practical solution. A masking ratio of 90% reduces the encoder complexity to <1 (noting that self-attention is guadratically- complex w.r.t. the token set size).

---

Our decoder is another vanilla VIT on the union of the encoded patch set and set of mask tokens [31]. Decoder-specific positional embeddings are added this set [31]. The decoder is designed to be smaller than the encoder [31]. Although the decoder processes the full set, its complexity is smaller than the encoder (e.g- ~1 122 per token). In our default setting, the overall autoencoder has a complexity reduction of 7.7x vs. full encoding (more discussions are in Sec. 5.1 and Table 1).
The decoder predicts the patches in the pixel space. principle we can simply predict full spacetime patch (e.g.. xx16 16); in practice, we find it sufficient to predict single time slice of the patch (16x16), which keeps the prediction layer's size manageable. We predict the original pixels or their per-patch normalized values [31] (compared in Table 2b). The training loss function is the mean squared error (MSE) between the prediction and its target, averaged over unknown patches [15].
The encoder and decoder are agnostic to the spacetime structure of the problem. There is no hierarchy or spacetime factorization, in contrast to the leading architectures [4, 2, 19). Our method relies on the global self-attention to learn useful knowledge from data, following the spirit of [18].
4 Implementation
Data pre-processing. For MAE pre-training, our default input size is 16 frames each with x24 pixels (i.e., 16x224x224). The 16 frames are sampled from the raw video with a temporal stride of 4 (i.e., 16x4 sampling in the literature [21]), and the starting frame is randomly sampled. In the spatial domain, we perform random resized cropping [63] with scale range of [0.5, 1), and random horizontal flipping. We do not apply other data augmentations unless noted.
Our MAE pre- pre-training is so fast in computation that data loading becomes new bottleneck that dominates running time in our setup. We adopt repeated sampling [33]' to alleviate this problem. Each time a raw video is loaded and decompressed, This reduces the data loading and decompressing time per sample. We note that repeated sampling does not change the number of samples seen; only influences the orders of the samples seen during training. We always count epochs as "effective epochs", i.e., how many times each raw video is sampled throughout training.
take multiple (4 by default) samples from it.
Architecture. Our encoder and decoder are the vanilla ViT architectures [18]. We use a temporal patch size of [2, 19, 77] and spatial patch size of 16x 16 [18), denoted as 16x6 16.16 We the same patch size for ViT-B/L/H [18] for simplicity, For a 16 224x 224 input, this patch size produces 8x14x14 14 tokens.
We adopt separable positional embeddings for the encoder. We have two positional embeddings, one for space and the other for time. The spacetime positional embeddings the sum of them. This separable implementation prevents the size of positional embeddings growing too large in 3D. We use learnable positional embeddings; the sin-cos variant [67] works similarly.
Settings. Our MAE pre- pre-trainigg configuration mostly follows [31]. We use the Adam optimizer [43] with batch size of 512. We evaluate the pre-training quality by end-to-end fine-tuning. The choice of evaluating by fine-tuning (instead of linear probing) follows [3, 31]. Our inference process follows the common practice of multi-view testing [74, 21]: it takes K temporal clips (by default K=7 on Kinetics) to cover the video length, and for each clip it takes 3 spatial views to cover the longer spatial axis (denoted as Kx3), The final prediction is the average of all views. The implementation details and hyper-parameters are in the appendix.
5 Experiments
In Sec. 5.1 and Sec. 5.2 we perform ablation experiments on Kinetics-400 (K400) [35]. We do MAE self-supervised pre-training and then fine-tune the encoder with supervision for evaluation. We report top-1 classification accuracy (%) ) on the K400 validation set. In Sec. 5.3 we study more pre-training datasets and downstream tasks.
In case, repeated sampling involves data augmentation and mask sampling.
5

---

MAE
scratch
MAE
(50
60-
(ilepect
1-view
60.7
73.4
(+12.7)
40-
100 spocis
multi-view
71.4
84.4 (+13.0)
20
MAE
500 opects
tine DouSs
Figure 5: MAE pre-training plus fine-tuning is much more accurate and faster than training from scratch. Here the x-axis is the wall-clock training time (128 A100 GPUs), and the y-axis is the 1-view accuracy on Kinetics-400 validation. The table shows the final accuracy. The model ViT-L.
MAE w/
acc.
FLOPs
compute
load+compute
encoder w/ [M]
84.3
627.5 G
141.1 hr
147.5 hr
encoder w/o [M]
84.4
81.0
24.5 hr
35.8 hr
gain
7.7x
5.8x
4.1x
Table 1: Training time comparison between a dense encoder (w/ [M] and a sparse encoder (w/o [M]) in MAE. The encoder is ViT-L (1024-d, 24- 24-block); the decoder is our default (512-d, 4-block). With a masking ratio of 90%, the sparse variant reduces FLOPs by 7.7x. This reduces computation time by 5.8x. In infra, computation is fast that data loading becomes bottleneck, which leads to an actual speedup of 4. x. Profiling is with synchronized SGD over 16 nodes, each with 8 A100 GPUs and 80 CPU cores. The training length 800 epochs.
5.1 Performance
Fig. 5 compares MAE pre-training vs, no pre-training (i.e., training from scratch), using vanilla ViT-L [18]. The from-scratch recipe follows [77] and has 71.4% accuracy. As comparison, using MAE pre-training for 800 epochs, the same vanilla ViT-L achieves 84.4% accuracy, which has a large increase of 13.0% absolute vs. training from scratch. This gap is much larger than that on image recognition tasks (~3% [31]) suggesting that MAE pre-training is more helpful for video recognition.
In addition to the accuracy gain, MAE pre-training can reduce the overall training cost, as plotted Fig. 5. The 800-epoch MAE pre-training only takes 35.8 hours. A short fine-tuning (100 epochs here), which takes 16.3 hours, achieves good accuracy thanks to pre-training. The overall training time can be shorter than training from scratch (e.g., 400 epochs, 65.2 hours), which converges more slowly without pre-training. This shows that MAE is a practical solution to video recognition.
MAE pre-training is fast because its encoder is only applied on the sparse set of visible patches, without the mask token [M]. We profile the pre-training performance in Table 1. With a masking ratio of 90%, the sparse encoder reduces the FLOPs (floating-pi) -pontt operations) by > 10x. After counting the decoder, the sparse design of MAE reduces FLOPs by 7.7 x. In our implementation. this reduction should produce a 5.8 computational speedup. if the video data were already pre- processes and loaded in memory. Our speedup ratio is high that the video me-processing and loading time becomes a new bottleneck. In our system, the data loading step increases the wall-clock training time from 24.5 hours to 35.8 hours. Nevertheless, this still leads to significant speedup of x.1x.²
5.2 Ablation experiments
Masking ratio. Fig. 6 shows the influence of the masking ratio jointly with the pre-training length. The ratio of 90% works the best. The ratio of 95% performs surprisingly well, which can catch up if trained long enough (Fig. 6 left). A higher masking ratio leads fewer tokens encoded by the encoder: to have a more comprehensive look, we plot the results w.r.t. the total number of encoded tokens (Fig, right). Under this measure, the ratios of 90% and 95% perform closely.
The lower masking ratios of 75% and 50% perform worse, even though the encoder sees more tokens and has higher computation cost. The ratio of 75% is optimal for its image counterpart [31], but not for videos. This observation can be explained by the assumption that video data is more information-redundani
The VIT-B result is 68.5% [77] trained from scratch using this recipe.
The speedup is closer to 5,8x if using slower GPUs (V100 instead of A100) that can hide the loading time.

---

85
84
84
S000ENC (5)
800uraC (%)
83
83
82
mask nitio 50%
82
mask ratio 50%
mask ratio 75%
mask ratio 75%
81
mark nitio 90%
81
mask ratio 90%
mask ratio 95%
mask ratio 95%
80
80
200
400
800
1600
8
16
32
64
128
256
epochs (log-scale)
encoded (billion. scalsc
Figure 6: Masking ratio. Every point represents a single pre-training and fine-tuning experiment. Left: x-axis is the epochs (proportional to the number of decoded tokens). Right: x-axis is the number of encoded tokens.
case
ratio
acc.
case
acc.
case
acc.
agnostic
90
84.4
pixel (w/o norm)
83.8
center crop
83.9
space-only
83.5
pixel (w/ norm)
84.4
rand crop
84.4
time-only
75
79.1
HOG
84.0
rand crop (stronger)
83.4
block
75
83.2
dVAE token
83.8
rand crop color jit
83.8
(a) Mask sampling. also Fig. 4.
(b) Reconstruction target. Pixels as reconstruction targets work well with no domain knowledge.
(c) Data augmentation. Strong aug- mentation is unnecessary.
Random sampling that is spacetime- agnostic works the best.
rep.
acc.
speed
dim
acc.
blocks
acc.
83.7
1.0x
128
80.8
83.2
2
84.3
1.8x
256
83.1
83.6
84.4
3.0x
512
84.4
84.4
1024
83.7
84.3
(d) Repeated sampling. All entries the # samples. Data loading overhead is reduced.
(e) Decoder width. Unlike the image counterpart [31]. overly narrow de- coder degrades accuracy noticeably.
(f) Decoder depth. Unlike the image counterpart [31], overly shallow de- coder degrades accuracy.
Table 2: Ablation experiments on Kinetics-400. The model is VIT-L, with an input size 16x224x222 and a spacetime patch size of 2 16x 16. The pre-training length is 800 epochs. The entries marked in gray are the same, which specify the default settings. This table format follows [31].
Mask sampling strategy. Our method follows the structure-agnosti random sampling methodology in BERT [15] and MAE [31]. Table 2a reports that this simple solution works the best in our method.
We compare with other strategies as illustrated in Fig. 4. Space-only sampling, which samples on the 2D spatial axes and broadcasts along the temporal axis, works reasonably well (83.5%) Time-only sampling, with masking ratio of 75% (i.e., keep 2 time steps out of 8), performs poorly (79.1%) we increase masking ratio to 87.5% (keep out 8), the accuracy drops further to 75.4%. Time-only sampling is related to future/past frame prediction, which can be an overly difficult task in our scenario. Blick-wise sampling [3]. in its spacetime variant [77], has 83.2% accuracy with 75% masking ratio (a higher ratio is worse).
Reconstruction target. Our method performs decently by reconstructing the original, unmodified pixels (83.8%, Table 2b). Using per per-patch normalized pixels [31] improves by 0.6%. This observation is similar to that of its image counterpart [31]. Using HOG [13] as the target [77] works strongly too.
The autoencoding nature of our method (i.e., predicting pixels) provides self-contained solution. In contrast, an extra tokenizer (e.g., dVAE [47, 9]), as is used in [3, 73], may require external data to train and additional domain knowledge to design (e.g., the dVAE used is ConvNet [37]). Applying the extra dVAE tokenizer to each frame is computationally heavy, which slows down training by 1.6x x in our implementation. Our pixel-based method is simpler and performs better (Table 2b).
Data augmentation. Temporal data can provide natural augmentation, egg., on view points, motion, deformation, occlusion. These forms of natural augmentation have been incorporated by random temporal sampling. Table 2c compares additional augmentation on spatial domain. Even using no spatial augmentation (center crop only) works competitively, similar to the observation images [31]. Random cropping with a mild scale range of [0.5, 1] works well, while stronger cropping (range [0.08, 1]. [63]) reduces accuracy; adding color jittering reduces accuracy too, similar to [3]].
7

---

pre Pre-train set
# pre-train data
pre wre-train method
K400
AVA
SSv2
none (from scratch)
71.4
INIK
1.28M
supervised
78.6
17.8
50.2
INIK
1.28M
MAE
82.3
27.2
65.6
K400
240k
supervised
22.2
55.7
K400
240k
MAE
84.8
32.3
72.1
K600
387k
MAE
84.9
33.7
73.0
K700
537k
MAE
n/a"
34.2
73.6
IG-uncurated
IM
MAE
84.4
35.1
73.6
Table 3: Influence of pre-training data, evaluated on K400, AVA, and SSv2 as the downstream tasks.
The MAE pre-training length is 1600 epochs on K400/600/700 and IG-uncurated. No intermediate fine-tuning is used, The model is ViT-I., 1 The K700 training set has 13. duplicated videos with the K400 validation set (19.9) 9.),,, so it is not legitimate to train on K700 to get K400 results.
is practically valuable for self-supervised learning methods to be less dependent on data augmenta- tion. There variety of applications in which augmentation is not valid is hard to induce, e.g.. medical imaging, hyper-spectral imaging, remote sensing, geometric data (point cloud, key points, etc.), and their temporal extensions. Our method could be generalized to these cases.
Repeated sampling. As our method fast computation, we adopt repeated sampling [33] to reduce the data loading overhead. Table 2d reports its influence. Using 2 or 4 repetitions increases wall-clock speed by 1.8x or 3.0x, as a loaded and decompressed file is reused multiple times.
Decoder capacity. Table 2e and 2 report the influence of the decoder width and depth. Using an overly small decoder degrades accuracy by large margins. This is unlike its image counterpart [31], in which 128-d or 1-block decoder has degradation if fine-tune fine-tuning is applied. We hypothesize that the higher-dimensional video data are more complex and thus require higher decoding capacity. On the other hand, our optimal decoder (512-d, 4-block) is still substantially smaller than the encoder (1024-d, 24-block). This is similar to the observation on its image counterpart [31].
5.3 Influence of Data
Transfer learning ablation. Table 3 studies pre-training on different datasets and transferring to vari- downstream tasks. The pre-training datasets include ImageNet- K (INIK) [14] and Kinetics-400, 600, and 700 [35, 6, 7]. The downstream tasks include K400, AVA [29], and SomethingSomething v2 (SSv2) [27]. We do not perform any intermediate fine-tuning (see appendix), so the comparison here is influenced by the data scale/tistribution but not by the number of their labels.
First we compare with pre-training the INIK images. MAE pre-training on INIK' is 3.7% better than INIK supervised pre-training (78.6% to 82.3%); this image-based MAE is even better than K400 supervised pre-training, both AVA (21.6% to 26.3%) and SSv2 (55.7% 65.6%).
MAE pre-training on K400 has massive gains over supervised pre-training on K400: improves by 10.1% 1% on AVA (22.2% to 32.3%) and 16.4% SSv2 (55.7%) to 72..%) MAE pre-training on K400 videos also substantially outperforms MAE pre-training on IN1K images: it increases by 2.5% on K400 (82.3% to 84.8%), 5.1% 1% on AVA (27.2% to 32.3%), and 6.5% on SSv2 (65.6% to 72.1 1)), suggesting that MAE pre-training on videos is highly beneficial for these video tasks.
With more pre-training data (K600/K700) without labels, we observe noticeable improvements on AVA and SSv2: comparing with K400 pre-training, MAE with K700 has an extra gain of 1.9% gain on AVA (32.3% to 34.2%) and 1.5% on SSv2 (72.1% % to 73.6%)
Real-world data. We further study MAE pre-training on real-world Instagram videos. We study two sets: (i) Instagram videos curated (IG-curated) [24] with hashtags similar to K400 classes, and (ii) random, uncrated Instagram videos (IG-uncurated). Both sets have million videos.
Table 3 (last row) reports transfer learning results on AVA and SSv2 using IG-uncurated pre-training. Notably, on AVA, MAE with IG-uncurated is betler than MAE with curated Kineties pre-training (e.g., by 3. 1..1/1//11111.. % over K400/600/700 pre-training); on SSv2, MAE with IG-uncurated is among the best, par with the K700 counterpart.
'The INIK pre-trained model is from https // / tuuth....... com//accebbokreeeercc/ee..

---

data
# videos
200-ep.
400-ep.
800-ep.
K400
240k
81.5
83.3
84.4
IG-curated
240k
79.0
81.6
83.2
IG-curated
512k
81.9
83.5
83.9
IG-curated
IM
83.5
84.1
84,2
IG-uncurated
IM
83.2
84.5
84.4
Table 4: Real-world Instagram data for MAE pre-training. We pre-train MAE each individual set for 200, 400, and 800 epochs. We compare fine-tuning accuracy on K400. The model is ViT-L.
Table 4 presents more results on the dataset size and training epochs. Pre-training on 240k subset of IG-curated (the same size K400) performs worse on K400 classification, which can be caused by the domain shift of data. However, increasing the dataset size of IG-curated 512k and 1M shows good gains: under the same number of pre-training epochs (200 and 400), it can outperform K400 pre-training even when evaluating on K400. IG-uncurated performs similarly well IG-curated, although the videos are randomly sampled and unrelated to K400 classes. This behavior is nor observed on contrastive learning methods for videos: e,g., in [22] is empirically shown that data curation has major impact contrastive learning [32, 10, 28] performance.
We believe that our exploration with real-world data has encouraging results. It is more realistic case of unsupervised learning at scale. We hope this exploration will shed light on future study,
5.4 System-level Comparisons
We provide system-level comparisons with the leading results on K400, AVA, and SSv2. The detailed tables are in the appendix (Table 7, 8. 9). These results are multifaceted, involving architecture designs, computational complexity, model sizes, input resolution, pre-training data and methods, etc., summarize in the tables. Our results are competitive and are close the leading entries. In particular, our results are based only on vanilla ViT architectures, while the leading methods are hierarchical specialized for videos. Our results demonstrate the potential of using fewer inductive biases and learning more from data, which is a pursuit of self-supervised learning.
5.5 Video Pre-training for Image Recognition
Finally, we report preliminary results on video pre-training for image recognition. The usage of vanilla VIT allows to convert to 2D easily: we only "deflate" patch embeddings by summing in time. Using ViT-L pre-trained by MAE on K400/ IG-uncurated, we obtain 83.7% / 84. 1% accuracy on IN1K image classification. This is better than training VIT-L from scratch on INIK (82.6% [31]), though lower than MAE pre- vre-training on INIK (85.9% [31]). Considering the large domain gap, we believe this result is decent and its improvement over training from scratch is encouraging. We hope will motivate the community to explore vidco pre-training for general visual representation learning.
Conclusion
We have explored simple extension of MAE [31] to video data. We have drawn several interesting observations. (i) We find that is possible to learn strong representations with minimal domain knowledge or inductive biases. This follows the spirit of the VIT paper [18]. Similar to BERT [15] and MAE [31], we show that self-supervised learning on videos can be tackled in a conceptually unified framework. (ii) We empirically show that the masking ratio an important factor for general masked autoencoding methods [69], and its optimal values may depend on the nature of the data (language, images, videos, etc.). (iii) We report encouraging results of pre-training uncurated data. It achieves strong performance, close to pre-training on controlled, curated data (e.g., Kinetics). To the best of our knowledge, promising results uncurated data rare in the literature.
real-world,
In spite of these observations, open problems remain. The scale data we have explored orders of magnitudes smaller than the language counterparts [52, 15, 53, 5]. While our method has largely improved the efficiency self-supervised learning, the high-dimensional video data still present major challenge for scaling up. We hope study will provide initial signals for future research.

---

References
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image [1] Pulkit Agrawal, João Carreira, and Jitendra Malik. Learning to see by moving. In ICCV, 2015. [2] Anurag Arnab, Mostafa Dehghami, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. ViVIT: A video vision transformer. In ICCV, 2021.
[3] Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image Transformers.
arXiv:2/06.08254 082544 2021.
[4] Gedas Bertasius, Heng Wang. and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, 2021.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
[6] João Carreira, Eric Noland, Andras Banki- Horvath, Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about Kinetics-600 600. arXiv: 1808. 100.00440, 2018.
[7] João Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. short note on the Kineties-
700 human action dataset. arXiv: /et::990779777 2019.
[8] João Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kineties dataset. In CVPR, 2017.
[9] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020.
[10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton, A simple framework for contrastive learning of visual representations. In ICML., 2020.
[11] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. ELECTRA:
Pre-training text encoders as discriminators rather than generators. In ICLR, 2020.
[12] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. RandAugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
[13] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR, 2005.
[14] Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
[15] Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional Transformers for language understanding. In NAACI, 2019.
[16] Ali Diba, Vivek Sharma, Luc Van Gool, and Rainer Stiefelhagen. DynamoNet: Dynamic Action
and Motion Network. In ICCV, 2019.
[17] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. PeCo: Perceptual codebook for BERT pre-training of Vision
Transformers. wwxiv:2111.12711 2021.
recognition at scale. In ICLR, 2021.
[19] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale Vision Transformers. In ICCV, 2021.
[20] Christoph Feichtenhofer. X3D: Expanding architectures for efficient video recognition. In CVPR, 2020.
[21] Christoph Feichtenhofer. Haoqi Fan, Jitendra Malik, and Kaiming He. SlowFast networks for video recognition. In ICCV, 2019.
[22] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In CVPR, 2021.
[23] Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learning with odd-one-out networks. In ICCV, 2017.
[24] Deepti Ghadiyaram, Matt Feiszli, Du Tran, Xueting Yan, Heng Wang, and Dhruv Mahajan. Large-scale weakly-supervised pre-training for video action recognition. In CVPR, 2019.
[25] Ross Goroshin, Joan Bruna, Jonathan Tompson, David Figen, and Yann LeCun. Unsupervised learning of spatiotemporally coherent metrics. In ICCV, 2015.
10

---

[26] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in 1 hour.
700.002677 2017.
[27] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag. et al. The "something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
[28] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent- a new approach to self-supervised learning. In NewrlPS, 2020.
[29] Chunhui Gu, Chen Sun, Sudheendra Vjjaynnaasimmann, Caroline Pantofaru, David A. Ross, George Toderici, Yeqing Li, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. AVA: A video dataset of neapo--tmpporlly localized atomic visual actions. In CVPR, 2018.
[30] Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In Workshop on Large Scale Holistic Video Understanding. ICCV, 2019. [31] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv:21 1eee2111.06677 1.06677. 2021.
[32] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[33] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry, Augment your batch: Improving generalization through instance repetition. In CVPR, 2020. [34] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
[35] Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, al. The Kinetics human action video dataset. 7055 2017.
[36] Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. MoviNets: Mobile video networks for efficient video recognition. In CVPR, 2021.
[37] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropugation applied to handwritten zip code recognition.
Neural computation, 1989.
[38] Hsin- Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming- Yang. Unsupervised represen- tation learning by sorting sequence. In ICCV, 2017.
[39] Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Improved multiscale vision transformers for classification and detection. arXiv:2 12. /2. 01566, 2021.
[40] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin Transformer v2: Scaling up capacity and resolution. arXiv:2111. 1.09883,. 2021.
[41] Ze Liu, Jia Ning. Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video Swin
Transformer. arXiv:2 106. 132300, 2021.
[42] Blya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2017.
[43] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[44] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video
prediction and unsupervised learning. In /CLR, 2017.
[45] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. In ICLR, 2016.
[46] Ishan Misra, C. Lawrence Zitnick, and Martial Hebert. Shuffle and Iearn: Unsupervised learning using temporal order verification. In ECCV, 2016.
[47] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurIPS, 2017.
[48] Junting Pan, Siyu Chen, Mike Zheng Shou, Yu Liu, Jing Shao, and Hongsheng Li. Actor- context-actor relation network for spatio-temporal action localization. In CVPR, 2021.
11

---

[49] Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In CVPR. 2017.
[50] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context
encoders: Feature learning by inpainting. In CVPR, 2016.
[51] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang. Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In CVPR, 2021.
[52] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
[53] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
[54] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
[55]
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
[56] Adria Recasens, Pauline Luc, Bean-Baapiste Alayrac, Luyu Wang. Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Päträucean, Florent Altché, Michal Valko, et al. Broaden your views for self-supervised video learning. In ICCV, 2021.
[57] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[58] Pierre Sermanet et al. Time-contrastive networks: Self-supervised learning from video. In ICRA, 2018.
[59] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. arXiv: /03.::500300255,, 2018.
[60] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 2014.
[61] N. Srivastava, E. Mansimov, and R. Salakhudinov. Unsupervised learning of video representa- tions using LSTMs. In ICML, 2015.
[62] Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Contrastive bidirectional transformer for temporal representation learning. 1906 055440 2019.
[63] Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
[64] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zhigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, 2016.
[65] Hao Tan, Jie Lei, Thomas Wolf, and Mohit Bansal. VIMPAC: Video pre-training via masked token prediction and contrastive learning. arXiv:2 1/250, 2021.
[66] Zhan Tong. Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training arXiv:2203. 12602, 2022.
[67] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[68] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML., 2008
[69] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. JMLR, 2010.
[70] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabelled video. In CVPR, 2016.
[71] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy.
Tracking emerges by colorizing videos. In ECCV, 2018
[72] Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Fore- casting from static images using variational autoencoders. In ECCV, 2016.
[73] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. BEVT: BERT pretraining of video transformers. In CVPR, 2022.
12

---

[74] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR. 2018.
[75] Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015.
[76] Xiaolong Wang, Allan Jabri, and Alexei A. Efros. Learning correspondence from the cycle- consistency of time. In CVPR, 2019.
[77] Chen Wei, Haoqi Fan, Saining Xie, Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. WXXi:2212.09113: 2021.
[78] Donglai Wei, Joseph Lim, Andrew Zisserman, and William T. Freeman, Learning and using
the arrow of time. In CVPR, 2018.
[79] Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invariances. In Neural Computation, 2002.
[80] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao. Qi Dai, and Han Hu. SimMIM: A simple framework for masked image modeling. arXiv:21 0022/11.0900 1.09886, 2021.
[81] Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In CVPR, 2019.
[82] Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. arXiv:2201.04285 2022.
[83] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang. Bin Xiao, Zhen Xiao, Jianwei Yang. Michael Zeng. Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision.
arXiv:211..11432. 2021.
[84] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[85] Bowen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, Andrew M Dai, Ruoming Pang, and Fei Sha. Co-training Transformer with videos and images improves action recognition. arXiv:2112.07175, 2021. 2021.
[86] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ICLR, 2018
13


Reviews:
[{"review_id": "rygbNz6H_JG", "reviewer": "Reviewer_esSt", "summary": "The paper extends the recently proposed MAE pipeline for images to the video domain. The method shows competitive performance on video datasets despite having minimum domain knowledge.", "questions": "1) It would be very interesting to see qualitative results for future predictions (when every token is masked in several last frames).\n2) Did you try to use masking while finetuning for a supervised task? (to obtain speed-up in computation on this stage as well). Maybe one can anneal mask ration in finetuning stage to obtain the best metrics with minimum computational resources.\n3) Is it possible to do Gibbs sampling with your pipeline? If yes -- did you try it and what is the quality of the samples? ", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Simple pipeline with competitive performance\n- Great ablation study\n- interesting possibility for computational speed-up (as the encoder is only applied on the sparse set of visible patches)\n- good performance on the uncurated dataset\n- good comparison with other video pretraining methods\n\nWeaknesses\n- I don’t see any\n\nAlthough the novelty seems limited to me as it is a straightforward generalization of MAE method to the video domain, in other aspects the paper seems seamless to me, the work has practical value to the community, and the experiments section is great. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OZ8ixolEw55", "reviewer": "Reviewer_VF6m", "summary": "This paper studies a simple extension of image MAE to video domain. The experiments are conducted on a set of standard video datasets, i.e., Kinetics, AVA, SSv2, etc., showing the proposed approach achieve appealing results. This paper also presents some interesting findings: (a) due to high information redundancy, video MAE requires higher mask ratio comparing to its image counterpart; (b) random masking works surprisingly well; (c) architectural findings such as video MAE requires a deeper decoder and a larger decoder hidden size. On the one hand, the ablation experiments are comprehensive which covers many aspects of the model design. On the other hand, these experiments are only conducted on a single dataset, K400, which is not convincing.  As the paper is mainly an analysis paper, instead of proposing a novel approach, the ablation studies are insufficient.", "questions": "- L192-194, Fig 6 (right). How do you vary the #encoded tokens when the mask ratio is fixed? By training longer?\n- L129 mentions it is sufficient to predict a single time slice (16x16) instead of the full space-time patch ($t$x16x16). Is there any experimental results that compares the two strategies?", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Simple, yet effective method for video self-supervised learning.\n- Comprehensive experimental comparison shows that the proposed approach has strong performance compare to prior work.\n- The paper is well-organized and is easy to follow. Most of the claims are supported with solid experimental results.\n\nWeaknesses:\n\n- The ablation experiments in Table 2 are only performed on a single dataset (K400), which may not provide a precise or comprehensive evidence of how these ablated items may affect the model performance. In TimeSformer [4] Table 1, it is shown that the conclusions aren’t exactly the same when looking at results from K400 and SSv2, as K400 is more static while SSv2 requires more temporal modeling. I strongly encourage the authors to also conduct ablation experiments on SSv2 to have more persuasive conclusions.\n- Overall the novelty is low, as it is almost a direct extension of image MAE. While there are few interesting conclusions/findings, e.g., (a) video modeling requires a higher mask ratio. (b) random masking works better than structual masking, and (c) decoder in video MAE needs to be more powerful, etc. But these are somewhat expected. I do value the experimental results that validate these findings, but the overall contribution is limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inadequate Data and Algorithm Evaluation"]}, "extra_text": {}}, {"review_id": "Ae10ynOoywz", "reviewer": "Reviewer_j1gc", "summary": "This paper proposes to extend MAE for videos. Extensive ablations show that MAE can be successfully extended to videos and the optimal masking ratio is higher than images. It achieves competitive results on popular video classification and detection datasets based on both large uncurated and well-aligned upstream datasets.", "questions": "1. Line 162: \"it takes K temporal clips (by default K=7 on Kinetics)\", any intuition why choose 7-view here?\n\n2. Line 130-131:  \"In practice, we find it sufficient to predict a single time slice of the patch (16x16)\", which time index?\n\n3. In Table 4, VideoMAE pretrained with IG-uncurated doesn't show big improvement over K400 when both are fine-tuned on in-domain K400. This raises a natural question in my mind: in MAE, does the accuracy continue to significantly improve with a bigger scale of data?\n\n4. A similar question regarding resolution is raised in Table 7,8,9: If VideoMAE with a larger resolution continues to show further gains?\n\n5. Random shuffle, slicing and reindexing causes randomly accessing the memory, which recomputes the cache. In practice, depending on the actual hardware, it may slow down the process (i.e. 90% cannot lead to 10x speed-up). Can authors share any knowledge about it?", "limitations": "Yes. The authors have adequately addressed the limitations and potential negative societal impact of their work", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Simple and effective idea: simple random masking with pixel-level reconstruction works well for videos.\n\n2. Clean design: Standard ViTs with a lightweight decoder is enough to achieve competitive results on video classification and detection.\n\n3. Comprehensive ablation studies on upstream/downstream data, decoder design choices, masking ratio/strategy, and  reconstruction targets. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept