Extending Video Masked Autoencoders to 128 frames
Authors: Nitesh Bharadwaj Gundavarapu
Luke Friedman
Raghav Goyal
Chaitra Hegde
Eirikur Agustsson
Sagar M. Waghmare
Mikhail Sirotenko
Ming-Hsuan Yang
Tobias Weyand
Boqing Gong
Leonid Sigal
Venue: NeurIPS 2024 poster (2024)
Keywords: MAE
long video understanding
masked autoencoder
adaptive masking
Primary area: machine_vision
TL;DR: A recipe to scale masked auto-encoders to long videos.

Abstract: Video understanding has witnessed significant progress with recent video foundation models demonstrating strong performance owing to self-supervised pre-training objectives;  Masked Autoencoders (MAE) being the design of choice. Nevertheless, the majority of prior works that leverage MAE pre-training have focused on relatively short video representations (16 / 32 frames in length) largely due to hardware memory and compute limitations that scale poorly with video length due to the dense memory-intensive self-attention decoding.  One natural strategy to address these challenges is to subsample tokens to reconstruct during decoding (or decoder masking). In this work, we propose an effective strategy for prioritizing tokens which allows training on longer video sequences (128 frames) and gets better performance than, more typical, random and uniform masking strategies. The core of our approach is an adaptive decoder masking strategy that prioritizes the most important tokens and uses quantized tokens as reconstruction objectives. Our adaptive strategy leverages a powerful MAGVIT-based tokenizer that jointly learns the tokens and their priority. We validate our design choices through exhaustive ablations and observe improved performance of the resulting long-video (128 frames) encoders over short-video (32 frames) counterparts. With our long-video masked autoencoder (LVMAE) strategy, we surpass state-of-the-art on Diving48 by 3.9 points and EPIC-Kitchens-100 verb classification by 2.5 points while relying on a simple core architecture and video-only pre-training (unlike some of the prior works that require millions of labeled video-text pairs or specialized encoders).

Extending Video Masked Autoencoders to 128 frames
Nitesh B. Gundavarapu Luke Friedman !
Raghav Goyal 2+1
Chaitra Hegde
Eirikur Agustsson'
Sagar Waghmare' Mikhail Sirotenko' Ming-Hsuan Yang'
Tobias Weyand Boqing Gong' Leonid Sigal"
Google Research University of British Columbia
(ngundavarapu, 1bfried, cvhegde, /irikurr/googgee.... (rgoyal14, 1sigal)@cs ubc.ca
Abstract
Video understanding has witnessed significant progress with recent video foun- dation models demonstrating strong performance owing to self-supervised pre- training objectives; Masked Autoencoders (MAE) being the design of choice. Nevertheless, the majority of prior works that leverage MAE pre-training have focused relatively short video representations (16 / 32 frames in length) largely due to hardware memory and compute limitations that scale poorly with video length due to the dense memory-intensive self-attention decoding. One natural strategy to address these challenges is to subsample tokens to reconstruct during decoding (or decoder masking). In this work, we propose an effective strategy for prioritizing tokens which allows training on longer video sequences (128 frames) and gets better performance than, more typical, random and uniform masking strategies. The core of our approach is an adaptive decoder masking strategy that prioritizes the most important tokens and uses quantized tokens as reconstruction objectives. Our adaptive strategy leverages a powerful MAGVIT-based tokenizer that jointly learns the tokens and their priority. We validate our design choices through exhaustive ablations and observe improved performance of the resulting long-video (128 frames) encoders over short-video (32 frames) counterparts. With our long-video masked autoencoder (LVMAE) strategy, we surpass state-of-the- art on Diving48 by 3.9 points and EPIC-Kitcheni 100 verb classification by 2.5 points while relying a simple core architecture and video-only pre-training (unlike some of the prior works that require millions of labeled video-text pairs or specialized encoders).
Introduction
Long video understanding has witnessed growing interest with various aspects of the problem having been explored by recent works [1, 2]. This includes incorporating (i) efficient attention mechanisms [3, 4], (ii) designing memory modules [5-7] to reason over context from the past, and (iii) approaching the problem from video-language perspective by proposing benchmarks [2, 8] and architectural choices that effectively leverage the interplay between the modalities [9-11, 8, 12].
The majority of recent models rely on foundational models pre-trained in self-supervisee manner based on videos and/or, for vision-language counterparts, video-text pairs; these foundational models, once pre-trained, can then be fine-tuned for specific task at hand (e.g., action classification). Masked
Equual primary contribution
i Work done Student Researcher Google Research Work done as Visiting Research Faculty at Google Research
Authhors now Google DeepMind
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Decoder
Decoddor
Deceder
Random masking encoder
Resonctruction
75 Decoder mask (ca)
25 Decoder mask (cu)
Figure I: Left: Proposed Long Video MAE Decoder Masking. We leverage a jointly trained adaptive tokenizer and importance module define decoder mask and token targets for video MAE pre-training strategy. The resulting sparsification in tokens (only 15%) allows pre-training with long videos (128-frames) and results in substantial performance gains. Right: Decoder masking and memory in long-video (128 frames) pre- pre-training. We report memory and FLOPs per-device for batch size of I using different decoder mask ratios and ViT-B architecture.
Autoencoders (MAE) have emerged as a simple and effective strategy to masked video modeling [13, 14] in this context. The standard MAE setup involves encoding a small fraction of visible / unmasked tokens (e.g., 10%) of an input video, and decoding the remaining masked tokens (e.g., 90%)
using dense space-time attention decoder blocks. However, for longer video sequences, reconstructing all the masked tokens at the decoding stage quickly leads to out-of-memory (OOM) due quadratic complexity of Transformers (feasible only for < 64 frames for moderate video resolution and current consumer hardware).
As result, most existing MAE-based video understanding approaches focus on learning representa- tions that encode only few frames (16 [15] 32 [14)) a time. This limits their ability understand actions and events spanning longer time horizons. Existing work has worked around the short context limitation by segmenting longer input video into short chunks and feeding them into the video model sequentially. Inference results from these chunks are then combined using late fusion. For example, "multi-crop" evaluations simply average-pool predictions over all chunks. However, this is clearly limiting. Long-con Long-context video models have the potential to understand longer actions (like complex diving routines), whole activities consisting of a sequence of actions and eventually entire story arcs.
Attempts to reduce computational cost and memory consumption, which bottleneck video MAE scalability, have recently started emerge. These efforts can be characterized by either (i) recon- structing subset of tokens at the decoding stage or decoder masking (VideoMAEv2 [14]), or by (ii) sub-sampling and focusing on subset of "informative" tokens instead of using the entire set of tokens (e.g., focusing The resulting gains have been shown to reduce training time and hardware requirement [17]. and can be reinvested to scale model size [14]. However, scaling the input video along the temporal dimension has received little-to-no attention besides scaling input frames to a higher spatial resolution [14).
measures of objectiveness [16] or motion [17] as proxies).
In this work, we build on [14] which performs masking at both the encoding and decoding stages. In particular, we focus on decoder masking considering its impact on memory and scalability, and propose a content-dependent adaptive masking strategy for videos in the MAE setup (see Figure 1). The core idea is token-importance (or saliency) scheme, learned jointly with adaptive quantization, that establishes a rank-order of video tokens, which is then used to select higher-ranking tokens for decoding. The jointly learned quantization shown also be effective, compared raw pixels, in defining the targets for reconstruction - further improving performance.
Our primary contributions are as follows:
1. We design content-dependent adaptive masking strategy and demonstrate that given low-token budget, it outperforms prior uniform and motion- based masking strategies.
2. The memory savings obtained from the low-token budget enables pre-training over long- videos (128 frames), allowing
ask the question of exactly how much benefit long videos bring in the context of MAE pretraining. We observe that the long-video MAE (128 frames) model consistently outperforms short-video MAE (32 frames), as measured using downstream fine-tuning performance; including when a short-video MAE is fine-tuned with longer context (e.g., 32 frame pre-trained MAE fine-tuned on 128 frames).
3. Leveraging these findings, we demonstrate state-of-the-art performance with our long-video MAE (LVMAFF) approach on conventional action classification benchmarks that are known to require longer-range motion understanding JEPIC-Kitchens- 100 [18] and Diving48 [19])

---

using standard ViT encoders and just one temporal crop without relying on any language supervision or labels during pre-training.
2 Related work
Masked Autoencoders (MAE). Masked autoencoders have initially been proposed as an effective mechanism for learning self-supervised representations in masked image modeling (MIM) [20]. Following their success in the image domain, several video MAE-based representation learning extensions have been proposed. Specifically, ST-MAE [13] extends MIM to spatio-temporal patches in clips - treating space-time blocks as tokens, while VAE [15] explores frame-level and tube-level masking strategies; both leverage an extremely high masking ratio (~ 90%) compared to image counterparts (~ 60%). Models that leverage joint image-video pre- pre-training have also been introduced (e.g., BEVT [21], OmniMAE [22].. However, scalability to high-resolution and long videos remains a substantial challenge, owing to the quadratic complexity of attention mechanisms.
MAE Reconstruction Objectives. While most video MAE models are trained to reconstruct the pixels of masked video patches [13, 15], there are a few notable exceptions. Mainly, BEVT [21] proposed to reconstruct discrete token targets obtained using a VQ-VAE [23] patch tokenizer, while MVD [24] focuses on masked feature modeling and distillation with high-level features as targets, obtained using leacher model. Compared to BEVT, we see even greater gains when reconstructing discrete token targets by using a more powerful tokenizer (MAGVIT [25]) and an improved, jointly trained, tokenization and masking scheme.
Efficiency by Token Reduction. Sub-sampling salient tokens, based on a learned or an off-the-shelf strategy during training. is shown to give faster convergence and require less resources [17, 16, 26]. VijectViViT [16] extracts object bounding boxes using an off-the-shelf object detector, and by placing emphasis on tokens belonging to objects was able to achieve lower token utilization and obtain competitive performance on the Epic Kitchens [18] and SSv2 benchmarks
[27]. Token Learner [26] observes and corrects for token redundancy in higher layers of ViT [28] using learned modules.
Similar to the above approaches, use token importance to inform our proposed masking. Differ- ently, we don't it sub-sample tokens, but selectively decode them during the training phase in a more generalizable dual masking setup. Concurrent EVEREST [17] is the most comparable to our work. attempts to learn what tokens in the spatio-temporal volume are more informative using a heuristic based on distance in feature space (as a proxy for motion). In the short 16-frame regime they find that sub-san such tokens in MAE setup leads to lower memory and resource consumption, while maintaining performance. Different from [17], our token saliency scheme is learnt separately and independently in a MAGVIT tokenizer [25] setup, while EVEREST learns token-saliency during MAE pre-training itself The latter approach can bias the method towards selecting easy tokens for the MAE reconstruction objective, undermining the learning, which is not an issue for our method. By learning token importance jointly with the tokenizer we incur a negligible extra cost, on top of the tokenizer computation, and noted earlier get significant additional performance improvements by using the tokens as targets as opposed to RGB in [17].
Motion in Video MAE. Building on the above, attempts to identify and attribute relevance to video tokens involved in motion compared to static background has seen growing interest [29, 30], since motion is fundamental to understanding videos. In particular, MGMAE [29] and MGM [30] show that an informed encoder masking based on motion- cuiis at patch-level makes the MAE reconstruction task account for motion, and found faster convergence and improved performance on motion-centric video datasets such as SSv2 [27]. Specifically, MGM [30] uses H.264 codec [31] to extract petel- motion vectors, and mask out tokens involved in motion during encoding, and MGMAE [29] uses online optical flow extractor (RAFT [32]), to warp and guide encoder masking using flow. Unlike these approaches, we do not model motion explicitly, and any apparent motion information in our mask is a byproduct of our data-driven token importance learning scheme.
Long Videos and Masking. As discussed above, masking tokens during training reduces memory requirement and this fact exploited in several video understanding works for scaling model size [33, 34]. However, expanding these techniques for long-videos videos has received limited attention. Recently, ViVi [9] leveraged random masking during contrastive pre-training on video-language tasks, and observed best -onormmance-mmory trade-off. Specifically, in order to convert short-video encoder to long- oingvi,eos, they fine-tune the last four layers of ViViT [35] on long videos (128 frames) along with 75% random masking. Similar to LongViViT
ViViT [9], we utilize masking to pre-train over long
3

---

videos. However, we focus on masked autoencoders instead of contrastive learning for long-videos and pre-train all the layers of our model, instead of partial freezing, in order to capture dense space- time correspondence in long-videos. Further, the overall training procedure of LongViViT [9] is considerably more complex requiring first pre-training with short videos and then further pre-training with long ones. We on the other hand, not only have simpler scheme that can directly pre-train MAE model with long videos (128 frames), but show that this is critical for improved performance.
Long Videos and Memory. Memory-based approaches [5, 6] attempt to form compressed rep- resentation of the past activations or memory; which is then incorporated into current window or time step, effectively prolonging the context length over which reasoning and predictions are formed. Orthogonal these works, we expand the local context of clips during pre-training from 16 128 frames; this can be combined with memory module to increase the global context window.
3 Approach
We first give the background on MAE and the dual masking, initially introduced in [14]. We then focus on describing our proposed adaptive importance masking strategy and discuss how it can be leveraged to train with up to a 128 frame context window.
3.1 Background: MAE and Dual Masking
input video V R33xPxXXW is first partitioned into non-overlapping spatio-temporal patches, and tokenized using patch embedding layer (typically a 3DConv) to give tokens T [T.),- where T; Rd is an 2th token with added positional encoding, N is total number of tokens, is the hidden dimension, and F, H. W are number of frames, height and width respectively of the input video. Using an encoder mask Me E 1), the unmasked / visible tokens are selected T" (T-MM.)) with N the total number of unmasked encoder tokens. vanilla ViT encoder [28] is applied the unmasked tokens to give encoded visible tokens Z = (T"). the usual case, we obtain input to the decoder Ze by combining the encoded tokens Z with learnable masked tokens M {M Me. where M, E _2 is [MASK] token embedding with positional embedding. The overall objective is to reconstruct the masked-out tokens using the unmasked encoded tokens. However, with Dual Masking [14], decoder mask Ma E f0, 1 is used to select tokens to reconstruct z Z (Mi)se(1-Ma>> with Nd the total number of unmasked decoder tokens where N << A' and N + NNd << N. Then, the combined tokens are reconstructed using a vanilla ViT decoder V = dec (Z)) Finally, the Mean Squared Error (MSE) loss computed between the original and reconstructed pixels,
C [M Ma
Σ
IV.-V,
(1)
sEMENMa
Alternatieely, if vector quantized tokenization is employed, the loss becomes,
L [.MenMe
Σ V. - VQ(V)
(2)
iEM.nMa
where VQ(.) is a vector quantization mapping typically trained separately using VQ- VA-VAE or variants. In the proposed approach VQ(()) takes the form of adaptive FSQg described in Section 3.3. Further, we want to highlight that encoder and decoder masking strategies need not be the same.
3.2 Masking Strategies
The above dual masking formulation requires both encoder and decoder masking, A number of strategies have been explored, but largely fall into two categories: content agnostic and informative.
Content agnostic masking strategies leverage either fixed or randomized scheme which is agnostic of the video content. Fixed strategies comprise of (i) grid-based uniform masking [15], which keeps every k-th row/column along spatial dimension; and (ii) frame masking [14] where all tokens from every k-th frame are kept an attempt to reduce temporal redundancy. The choice between the two depends on the assumptions regarding relative importance of spatial temporal information.

---

Pixel reconstruction, GAN and perceptual losses
MAGVIT Encoder
Top-K
MAGVIT Decoder
Mask
Token
Scorer
Adaptive tokenizer
Illustration of Adaptive FSQ-MagViT Training. PSQ-MagViT adaptive tokenizer includes MAGVIT encoder and CNN-based token scorer with differentiable top-k selection layer designating importance of tokens. During tokenizer training unselected tokens zeroed out and video reconstructed using MAGVIT decoder. We then freeze this adaptive tokenizer and use it to generate larget tokens for scalable pre-training of video MAE.
Figure 2:
(iii) Randomized strategies [13, 15] leverage sampling instead, which results in form of data augmentation, since the same clip encountered in different epochs would result in a different mask. Under low-token budget, the above mentioned approaches shown perform competitively, and at the same time, gain memory efficiency [17]. Masking approaches mentioned above are effective. but at their core are sub-optimal as they fail to take into account the content of the video itself. Our approach falls into an informative masking class of strategies, which instead of sub-sampling overall set of tokens, leverage an importance function to produce a relative rank-order (or priority) on input tokens and use this ordering when selecting which tokens to encode/decode. We denote the token-importance (or saliency) as S E [0, 1JN,, where a probability distribution over rank-order of tokens. Letting S be probability distribution allows both
N is the total number of tokens an input video. particular, tokens, which gives deterministic (k most probable) and stochastic forms of informed masking. Prior work has relied on optical flow [29, 31] proxy for importance, which in this work leverage We, on the other hand, propose a more general adaptive FSQ-MagViT scheme that learns importance of tokens concurrently with tokenization (see Section 3.3), leveraging MAGVIT [25] tokenizer.
of our baselines.
While the above mentioned strategies can used to generate both encoder and decoder masks in the dual masking MAE [14] (with the potential mild overhead of keeping the two sets of tokens disjoint), in this paper we focus on decoder masking in particular. This design choice motivated by the observation that decoder masking has much more significant impact on the memory usage and scalability
the overall MAE framework (see Figure 1) (right). Our focus on informative and effective decoder masking allows to pre-train long- videv sequences (128 frames) (left). To this end we keep the encoder masking for most experiments in this paper relatively simple.
Figure I
Encoder masking. Motivated by results in [15] for most experiments in the paper (unless otherwise stated) we use randomized tube masking with ratio of 90% (i.e., encoder only sees 10% of patches).
Decoder masking. We generate Adaptive decoder mask Ma based on the intuition that recon- structing high-salient tokens is more meaningful for the MAE pre-training and for downstream fine-tuning tasks. We select top-k salient tokens from S and set them as visible (indicated as O's) in the decoder mask Ma {0, 1).,, where k tio. In addition to selecting top-k tokens, we also include small fraction (p,) of tokens sampled randomly, which we observed to improve performance, similar to ObjectViViT [16]. Therefore, N' = (p) Pa) x N + x N. Illustration of resulted mask is shown in Appendix A.6.
- Pa) N and pa is decoder masking ra-
3.3 Adaptive Finite Scalar Quantized VAE for Saliency and Reconstruction Targets
Our adaptive token selection module, shown in Figure 2, combines an effective CNN-based tokenizer and differentiable top-k selection layer.
Tokenizer. For the tokenizer we combine MAGVIT [25], which a powerful 3D-CNN based video tokenizer and Finite Scalar Quantization (FSQ) [36], simple alternative VQ that easily scales
5

---

up to large codebooks. We refer to this combination as PSQ-MagViT and provide more details in Appendix 1.
Token Scorer For the token selection, we learn a CNN-based Token scoring module followed by a differentiable top-k layer, resulting in the mask which we apply the FSQ-Mag ViT tokens. More precisely, we embed the video to feature space with two CNN layers, strided both spatially and temporally to match the resolution of the tokens from FSQ-MagVii ViT encoder. From this CNN feature, we obtain pairwise Euclidean distances between spatial locations in adjacent frames, measuring how much each token in the i-th frame differs from the corresponding token the 1)-th frame. This distance is considered as token importance as it signifies the extent of underlying change in the video as done in [17].
Adaptive FSQ-MagViT We get the soft-tokens from FSQ-MagViT and token importance from the token scoring module. We keep the tokens with the top-k largest distances (k: 768) and mask the remaining N tokens, replacing their values with O. For the special case of the first frame we keep all the tokens during training. These masked tokens are then fed through FSQ and the MagViT decoder. Similar to MAGVIT [25]. we optimize for pixel reconstruction loss, GAN loss and perceptual loss by training the FSQ-Mag ViT and token scorer end-to-end.
3.4 Implementation details
First, we train our adaptive token selection module on Kinetics600 [37] data on 16 frame clips. Once trained, we keep this module frozen. Next, we follow standard MAE pre-training with a couple of changes (i) we reconstruct the top-k tokens selected by the token selection module (and small number of randomly selected tokens), and (ii) we reconstruct the latent embeddings from the tokenizer instead of RGB pixels. Note, we consciously choose to decouple the learning of token selection and MAE itself so that the gradients do not bias token selection towards selecting easily unmaskable tokens and vice-versa. Further implementation details are Appendix A.
4 Experiments
We experiment with LVMAE on conventional video action classification benchmarks that have potential benefit from long-video encoders: SPICC-Kichhens-ie 100 (EK100) [18] and Diving48 (D48) [19]. EK100 [18] contains ~ 90K video clips in total at 25 FPS with 3.7 secs avg. duration, and 15% videos with least 5 secs (=125 frames) duration. The task to classify nouns (=300) and verbs (=97) together. D48 [19] contains videos with 158 avg. number of frames, and vary from 24 to 822 frames, and task to classify among fine-grained dive categories [5]. Results on additional datasets are presented in Appendix A.2.
4.1 Decoder is the most memory intensive stage in long-video MAE
In Figure 1 (right), we show the memory and compute characteristics of VIT-B MAE architecture as we vary the decoder masking ratio for a video length 128 frames. We fix the encoder masking ratio to 90% for this comparison. Even though the encoder has 3X more layers, decoder starts dominating for long videos due to the high number of tokens and quadratic scaling. We find that the decoder masking ratio should be close to the encoder masking ratio to respect compute and memory constraints imposed by the long-vin video pre- pre-training regime.
4.2 Adaptive decoder masking strategy outperforms short videos
In Table 1, we compare the proposed Adaptive masking with several alternative decoder masking strategies short videos (32 frames) using both RGB pixels and FSQ-MagViT tokens reconstruc- tion targets on the EK100 dataset. We establish a baseline using the default MAE configuration which involves random tubes [14] encoder mask (with mask ratio 90%), and decoding all the masked tokens, denoted as decoder mask ratio None. Then we experiment with the low-budget setting,
1 We note that EVEREST does non- con-differentiable top-k selection on the low-level features of the MAE. In our case, we are learning the CNN top-k module in parallel with FSQ 2ag-MaVV VIT and need differentiable top-k be able to learn it end-to-end.

---

Table 1: Decoder masking strategies on short-videos (32 frames) on EK100. We report fine- tuning t action classification performance of MAE pre-trained models using pixel (RGB) or token (FSQ-MagViT) reconstruction targets. For decoder masking, we consistently use 15% as the token budget. We find that (1) compared to no decoder mask (None) with 100% budget, uniform masking scheme (Uniform [14]), random masking scheme (Random), and decoder masking using Optical Flow (Flow) perform competitively with the lower token budget, and (2) we obtain best results with our proposed Adaptive decoder masking scheme (Adaptive). Note that fine-tuning performance is
reported on I temporal crop.
Saliency scheme
Masking
RGB
FSQ-Mag VIT
None
39.87 10114
42.63 +0.07
Random
39.51 10.11
42.20 #0.07
Uniform [14]
V
39.92 10.14
41.73
#0.05
Flow
V
39.10 10.12
10.16
EVEREST [17]
V
36.15 10.14
39.24:10.0
Adaptive (Ours)
40.58 +.008
43.21 +0009
N 0.85N (i.e. 15% token budget), where tokens to be decoded are selected using different saliency schemes described below.
Random and Uniform. The Random importance scheme selects tokens randomly from the spatio- temporal volume. For the Uniform saliency scheme, we form decoder-visible tokens by picking frames uniformly using a step size of 7 to select 1/7th of frames or roughly 15% token budget for decoding. We observe competitive performance to the baseline, consistent with VideoMAEv2 [14].
Flow and EVEREST. For the Flow importance scheme, we identify parts of the input video that contain motion. First, we obtain pixel-level Optical Flow F E 1|1,122XTx8 xx for the video using RAFT [32], containing both and displacement fields. We then use the pixel-level Flow to obtain a probability distribution of motion in token space to generate masks. Specifically, following the same patchify operation used for RGB video, for each spatio-temporal patch in Flow, we take the mean of absolute value for all pixels in the patch across v and coordinates, and normalize across all patches to obtain S E [0, 1 where N is the total number of tokens. For the proposed Adaptive scheme and the Flow scheme, we choose Pa = 90% and we add a small amount of random tokens, Pr = 5% as we found it improves the results. For EVEREST we follow details of [17].
Although, Flow and EVEREST strategies provide more informed token prioritization based on explicit motion vectors or learned pixel distance, respectfully, they don't 't perform as well as the proposed Adaptive masking approach. This is in part due to inaccuracies prioritization that may result from optical flow when, for example, background motion is involved. Overall, we make few observations: (I) with only 15% decoder token budget, our proposed adaptive scheme bridges the gap with, and even improves on, vanilla VideoMAE which decodes all tokens with 100% token budget (None), and (2) in addition to random and uniform schemes, that are content agnostic, our proposed strategy outperforms over content informmedd approaches (Flow and EVEREST), demonstrating its effectiveness, and lastly (3) the above trends hold over both pixel and token reconstruction objectives. As a side note our findings corroborate with other related works [25, 33, 24] that reconstructing higher-level targets outperform RGB pixel reconstruction for MAE The masking strategy comparisons upon scaling 128 frames are presented in Table 4d and illustrate similar trends.
4.3 Adaptive decoder masking enables pre-training over long videos
Using the best performing adaptive decoder masking strategy and FSQ-Mag VIT reconstruction targets, we reinvest the memory efficiency gained using just 15% token budget towards extending the input number of frames to 128. Table 2 shows performance on long-videos (128 frames). We first highlight that default VideoMAE with no decoder masking encounters out of memory error due to large number of spatio-temporal tokens, demonstrating the need for decoder masking.
Overall, we observe that: (1) 128 frame pre-training outperforms 32 frames pre-training when fine-tuned on 128 frames; (2) 128 frame pre-training outperforms the typical 32 frames multi-crop
evaluation. These two findings establish the significant benefit unlocked by our proposed model owing to long-video MAE pre- pre-training, often ignored by previous models due to short context lengths.
7

---

Table Decoder masking enables training over long-video (128 frames). We report fine-tuning top-1 action classification performance of long-video MAE pre- pre-trained models using token (FSQ- MagVin as reconstruction target. Note that 128 frumes fine-tuning is performed using random-tube masking with 20% masking and evaluation is reported on I temporal crop. Refer to Appendix A.5.3 for the fine-tune fne-tuning details..
128 frames pre-training
32 frames pre-training
Saliency
Fine-tuning (128 frames) Eval (128 frames I crop)
Fine-tuning (128 frames)
Fine-tuning (32 frames)
scheme
Eval (128 frames x crop)
Eval (32 frames 4 crops)
EK100
D48
EK100
D48
EK 100
D48
None
out memory
44.5
85.7
44.1
76.8
Adaptive (Ours)
47.3
87.9
45.0
83.2
45.0
75.7
4.4
Comparison with prior state-of-the-art works
In Tables 3a and 3b, we compare our proposed method, dubbed LVMAE, against the state-of-the-art on OPIC-Kitchens-1O0 and Diving48 respectively. For these comparisons, we first pre-train our model on unlabeled videos from Kinetics710 [38] data using 128 frames and adaptive masking strategy. This followed by pre-training and fine-tuning on respective datasets 128 frames similar Sec.4.3. We additionally scale the encoder size to VIT-L. Further experiment details are in the Appendix. Note that existing SOTA methods use tailor-made architectures and/or pre-train their models using large-scale supervised pre- pre-training data.
In Table 3a, we show that our proposed model improves current SOTA EPIC-Kitchens Top-1 Verb classification by +2.5 points using standard ViT architecture and just a single crop. On the Top-1 Noun classification, our model lags behind MTV-B [39] pre-trained video clips and a specialized multi-view architecture, TAdaFormer [40] pre-trained on supervised Kinetics710 10 and recently published Avion [41] that pre-trains on large-scale egocentric data namely Ego4D. We would like to point out that large-scale labeled pre-training helps nouns more than verbs. The EPIC-Kitchens noun categories such as hands, gloves, spoon, knife etc. routinely appear as annotations in large-scale pre-training datasets such as ImageNet21k [42], Ego4D [43] etc. (e.g. 282/300 nouns from EPIC- CHtCC-Ktthees-111 100 also appear in ImageNet21k). As noted by Verbs-In-Action [44], verbs are relatively scarce in existing datasets. This explains why existing SOTA that use large-scale datasets excel at noun classification. On the other hand, our approach doesn't 'tt use large- scale pre-training datasets and learns long-range spatio-temporal dynamics to push SOTA on verb classification. Furthermore, Table 3a, we find that if we add supervised pre-training stage to our model using medium-scale dataset, can bridge the gap on Noun classification while maintaining SOTA on Verb classification.
60 million labeled
In Table 3b, we show that our proposed model improves the absolute state-of-the-arr on Diving48 dataset which contains complicated diving sequences by 3.9 points. It's worthwhile note that the current SOTA method, MC-V ViT [5] effectively uses 27M video-text pairs while we use 1M unlabeled videos and just 15K labeled videos.
4.5 Ablation study
In Table 4a, we ablate our decoder masking strategy by pre-training models at 32 frames and report the performance on SPIC-KKtchenns dataset using a ViT-B backbone. Similar to [16]. we find that reconstructing small amount of random tokens helps in improving performance. Since reconstructing random tokens adds stochasticity to the sampling process, we hypothesize it can help with overfitting, On the other hand, it is sample inefficient to only reconstruct random tokens and combining these two approaches strikes balance. We limit this ablation study to a sampling ratio of 15%, respecting memory constraints for extending 128 frame pre- trainin....
In Table 4c, we gradually increase the number frames and report the effect on EPIC-Kitchens- 100 dataset. We find significant improvements in accuracy as we increase the number of frames from 16 to 32 to 64. However, as we move from 64 frames to 128 frames, the marginal improvement small. This is expected such videos form the tail-end of the distribution.
8

---

Table 3: Comparison to State-of-the-art (SOTA). In these tables we compare broad set of approaches, many of which use additional (labeled) data in pre-training or specialized modules.
(a) Comparison to SOTA on EK-100.
Model
Extra Pre-training Data
Action
Verb
Noun
SlowFast [45]
K400
38.5
65.6
50.0
IPL (I3D) [46]
K400
41.0
68.6
51.2
ViVIT-L/16x2 [35]
IN21
44.0
66.4
56.8
MoViNet-A5 [47]
N/A
44.5
69.1
55.1
MeMViT-16, 16x4 [6]
K400
46.2
70.6
58.5
MeMviT-24, 32x3 [6]
K600
48.4
71.4
60.3
Omnivore (Swin-B) [48]
KK41KK+IKK4400+SSU
49.9
69.5
61.7
MTV-B [39]
IN21K
46.7
67.8
60.5
MTV-Br2sot [39]
WTS-60M
50.5
69.9
63.9
TAdaFormer-BBu4 [40]
K710
49.1
71.0
60.5
L [40]
K710
51.8
71.7
64.1
vision-language pre-training
LaViLa (TSF-B) [49]
WIT Ego4D
46.9
69.0
58.4
LaViLa (TSF-L) [49]
WIT + Ego4D
51.0
72.0
62.9
Avion (ViT-B) [41]
WIT Ego4D
49.1
70.0
59.8
Avion (ViT-L) [41]
WIT + Ego4D
54.4
73.0
65.4
LVMAE (VīT-B)
None
47.3
73.1
56.8
LVMAE (ViT-B)
Unlabeled K710
47.0
73.0
56.3
LVMAE (ViT-L)
Unlabeled K710
50.9
75.5
59.6
LVMAE (ViT-L)
K710
52.1
75.0
61.8
Comparison to SOTA Diving48.
Model
Pre-train
Top-1
TimeSformer-L. [50]
IN21K
81.0
VideoSwin-B [51]
IN21K
81.9
BEVT [21]
IN21K+K400
86.7
SIFAR-B-14 [52]
IN21K
87.3
ORViT [53]
IN21K
88.0
AIM ViT-B/16 [54]
CLIP
88.9
AIM ViT-L/14 [54]
CLIP
90.6
ALIGN+LTIP+JFT
MC-ViT-B [5]
89.7
+HT 100M
ALIGN+LTIP+JFT
MC-ViT-L [5]
91.0
#T100MM-TP
Video-FocalNet-B [55]
K400
90.8
LVMAE (ViT-B)
None
87.8
LVMAE (VIT-B)
Unlabeled K710
91.2
LVMAE (VIT-L)
Unlabeled K710
94.9
In Table 4c, we compare our model's performance with the current SOTA model, Avion [41], on videos of different lengths using the EPIC-Kitchens' Verbs benchmark. For this study, the Large version of our model. observe sustained performance improvements over SOTA with longer durations signifying
model's capability on longer sequences.
In Table 4b, we ablate the choice of targets for MAE pre-training. For this experiment, we first pre-train several models using 32 frames with no decoder masking. We find that when we shift from standard RGB targets MAGVIT targets, the Noun-Vers Top-1 accuracy improves by 3.3%. Further, we notice a small drop from switching from MAGVIT targets to Adaptive FSQ-Mag VIT targets. In the row, we show that when we use the adaptive decoder masking strategy with Adaptive FSQ-Mag ViT targets, recover the performance. In effect, our proposed adaptive masking strategy

---

Table 4: Ablation study. We ablate a number of important design choices (please see text for details).
MAE Pre-training (FSQ-MagViT)
Fine-Tuning
# of frames
Eval-Protocol
Noun-Verb Acc
Sampling ratio
Noun- Verb Accuracy
16
16 x 8 clips
41.7
Adaptive (1 Pd) Random (px)
32
32 4 clips
45.0
64
64 2 clips
47.1
None 15% + 0%
42.6
128
128 clips
47.3
42.5
10% + 5%
43.2
(c) Number of frames ablation.
0% 15%
42.2
(a) Effeet of masking ratios using adaptive scheme.
Saliency Scheme
EPIC-Kitchens
Diving48
None
OOM
OOM
Masking
Target
Noun- Merb-Verbaaan Accuracy
Random
46.4
86.3
None
RGB
39.9
Uniform
45.6
85.2
None
Standard MAGVIT
43.2
Flow
46.3
86.3
None
Adaptive FSQ Mag VIT
42.6
Adaptive (Ours)
47.3
87.9
Adaptive
Adaptive FSQ-Mag VIT
VTT
43.2
(d) Masking strategy ablation at 128 frames with Adaptive FSQ-MagViT targets.
(h) Varying targets for MAE at 32 frames.
Model
Os-4s
4s-8s
8s-16s
16s-32s
>32s
AVION [41]
75.6
66.0
64.7
66.3
51.9
LVMAE (Ours) 77.8
67.0
66.2
72.2
57.7
Relative Difference
+2.9%
+1.5%
+2.3%
+8.9%
+11.2%
(e) Comparison with SOTA on EPIC-Kitchens-100 Verbs at different video lengths.
allows us to retain the performance boosts from MAGVIT targets at very high masking ratios, and thereby scale these gains 128 frames effectively and surpass state-of-the-art. Unless otherwise mentioned, we always use Adaptive FSQ-Mag ViT as targets for all of our experiments.
In Table 4d, compare proposed adaptive decoder masking strategy with other strategies at 128 frames using PSQ as targets, We find that our proposed adaptive masking is best as we scale the number of frames from 32 to 128 both EPIC-Kitchens- 100 and Diving48 datasets.
5 Limitations and broader impact
In this work, we restrict ourselves to relatively small datasets and model sizes and leave the exploration about large-scale pre-trace pre-training, joint training of image and video datasets, higher-capacity models, etc. to future work. Furthermore, while the 128 frames in this work are big eapp from prior works' focus on 16 Using efficient decoders (and encoders) or combining our long local-context with memory offers an alternate path to scaling MAEs, which orthogonal our approach some degree.
frames, we anticipate more significant improvements to handle longer videos.
Improved long-video understanding can potentially revolutionize how users interact with video content since it can enable Al reason across complex events and nuances. This potential will benefit accessibility, efficient content creation, recommendation. moderation, etc. Mcanwhile, as with many machine learning models, our proposed method can be biased by the data. In addition, some applications (e.g., surveillance) might negatively impact society, and we urge users and researchers to deal with such use cases responsibly.
Conclusion
In this paper we present a relatively simple but highly effective adaptive masking strategy for video MAE pre-training that allows us to pre-train on long videos (128 frames). Our approach is based on novel MAGVIT-based tokenization strategy that also learns importance of tokens. Further, we are the first, to our knowledge, to show that long-video MAE pre-training is not only possible but leads to better encodings. Using our approach we are able to achieve state-of-the-a downstream performance, despite using simple architecture and video only pre-training.
10

---

7 Acknowledgements
We thank Chris Duvarney for finding better hyper-parameters to our models. We are grateful to Huisheng Wang, Nisarg Kothari, Philip Mansfield and Hartwig Adam for their continued support. We sincerely acknowledge Anurag Arnab and the Scenic team for the excellent framework.
References
[1] Chao-Yuan Wu and Philipp Krahenbuhl. Towards long- long-form video understanding. In CVPR, 2021. 1
[2] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-forru video language understanding. In NeurlPS, 2024.
[3] Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video models. In ECCV, 2022.
[4] Jue Wang. Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. Selective structured state-spaces for long-form video understanding. In CVPR, 2023.
[5] Ivana Balažević, Yuge Shi, Pinelopi Papalampidi, Rahma Chaabouni, Skanda Koppula, and Olivier J Hénaff. Memory consolidation enables long- cong-context video understanding, In ICML, 2024. 1, 1,4, 6, 8,9
[6] Chao- Chao-Yuan Wu. Yanghao Li. Karttikeya Mangalam, Haoqi Fan, Bo Xiong. Jitendra Malik, and Christoph Feichtenhofer. Memvit: recognition. In CVPR, 2022. 4,9
multiscale vision transformer for efficient long-term video
[7] Michael S Ryoo, Keerthana Gopalakrishnan, Kumara Kahatapitiya, Ted Xiao, Kanishka Rao, Austin Stone, Yao Lu, Julian Ibarz, and Anurag Arnab. Token turing machines. In CVPR, 2023, 1
[8] Md Mohaiminul Islam, Ngan Ho, Xitong Yang. Tushar Nagarajan, Lorenzo Torresani, and Gedas Bertasius. Video recap: Recursive captioning of hour-long videns. In CVPR, 2024.
Pinelopi Papalampidi, Skanda Koppula, Shreya Pathak, Justin Chiu, Joc Heyward, Viorica Patraucean, Jiajun Shen, Antoine Miech, Andrew Zisserman, and Aida Nematzdeh. A simple recipe for contrastively pre-training video-first encoders beyond 16 frames. In CVPR, 2024. 1, 3,3,4
[10] Ce Zhang, Taixi Lu, Md Mohaiminul Islam. Ziyang Wang, Shoubin Yu. Mohit Bansal, and Gedas Bertasius. In EMNIP, 2024.
A simple Ilm framework for long-range video
[11] Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang. and Jianlong Fu. Long-form video- language pre-training with multimodal temporal contrastive learning. In NeurIPS, 2022. 1
[12] Reuben Tan, Ximeng Sun, Ping Hu, Jui-hsien Wang, Hanieh Deilamsalehy, Bryan A Plummer, Bryan Russell, and Kate Saenko. Koala: Key frame-conditioned long video-llm. In CVPR, 2024.
[13] Christoph Feichtenhofer, Yanghao Li, Kaiming Hc, et al. Masked autoencoders as spatiotemporal leamerr. In NeurlPS, 2022. 2, 5,5
[14] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang. Yali Wang. and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In CVPR, 2023. 2, 4, 14, 15
[15] Zhan Tong, Yibing Song, Jue Wang. and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre- trainiin In NeurlPS, 2022. 2, 2, 4, 14
[16] Xingyi Zhou, Anurag Arnab, Chen Sun, and Cordelia Schmid, How can objects help action recognition? In CVPR, 2023. 2, 2,
[17] Sunil Hwang, Jachong Yoon, Youngwan Lee, and Sung Ju Hwang. Everest: Efficient masked video autoencoder by removing redundant spatiotemporal tokens. In ICMI., 2024. 2, 6,
[18] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-10 In IJC, 2022. 2, 2, 6
[19] Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representation bias. In ECCV, 2018. 2,

---

[20] K. He, X. Chen. S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In CVPR. 2022. 3
[21] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu- Yu-Gang Jiang, Luowei Zhou, and Yuan. Bevt: Bert pretraining of video transformers. CVPR, 2022. 3, 3.9
[22] R. Girdhar, A. El-Nouby, M. Singh, K.V. Alwala, A. Joulin, and I. Misra. Omnimae: Single model masked pretraining on images and videos. In CVPR, 2023. 3
[23] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In JCML., 2021. 3, 14
[24] Rui Wang. Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. In CVPR, 2023. 3,7
[25] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann. Ming-Hsuan Yang. Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In CVPR, 2023. 3, 5, 6, 7, 14, 15, 17
[26] Michael Ryoo, Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova, Tokenlearner: What can 8 learned tokens do for images and videos? In NeurlPS, 2021. 3
[27] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim. Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Queller-Freitage et al. The" something something" video database for lcarning and evaluating visual common sense. In ICCV, 2017. 3, 14
[28] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. In ICLR, 2021. 3,4 4
[29] Bingkun Huang, Zhiyu Zhao, Guozhen Zhang, Qiao, and Limin Wang. Mgmae: Motion guided masking for video masked autoencoding. In ICCV, 2023. 3.5
[30] David Fan, Jue Wang. Shuai Liao, Yi Zhu, Vimal Bhat, Hector Villal-o, Rohith MV, and Xinyu Li. Notion-guided masking for spatiotemporal representation learning. In ICCV, 2023. 3
[31] lain E Richardson. Video codec design: developing image and video compression systems. John Wiley & Sons, 2002. 3,5
[32] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020. 3,7
[33] Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand. Yue Zhao, al. Videoprism: A foundational visual encoder for video understanding. In ICML, 2024. 3.7
[34] Yi Wang, Kunchang Li, Xinhao Jiashuo Yu, Yinan He, Guo Chen. Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding. In ECCV, 2024. 3
[35] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, 2021. 3,
[36] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization:
VQ- VAE made simple. In ICLR, 2024. 5, 14, 17
[37] Joao Carreira, Eric Noland, Andras about kinctics-600. arXiv preprint arXiv: 1808.01340, 2018. 6, 15
Chloc Hillier, and Andrew Zisserman. A short note
[38] Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note the kinetics-700 human action dataset. arXiv preprint /9X7::/907.000 06987, 2019.
[39] Shen Yan, Xuehan Xiong, Anurag Arna, Zhichao Lu. Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. In CVPR, 2022. 8, 9
[40] Ziyuan Huang. Shiwei Zhang. Liang Pan, Zhiwu Qing. Yingya Zhang, Ziwei Liu, and Marcelo H. Ang Jr. Temporally models for efficient video understanding, arXiv preprint arXiv:2308.05787 2023. 8,
12

---

[41] Yue Zhao and Philipp Krähenbühl. Training large video model on a single machine in a day. arXiv preprini arXiv:2309. 2023. 8, 9, 10
[42] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. In NeurlPS Datasets and Benchmarks Track, 2021. 8
[43] Kristen Grauman, Andrew Westhury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022. 8
[44] Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in video-language models. In ICCV, 2023. 8
[45] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for vidco recognition. In ICCV, 2019,
[46] Xiaohan Wang, Linchao Zhu, Heng Wang, and Yi Yang. Interactive prototype learning for egocentric action recognition. In ICCV, 2021. 9
[47] Dan Kondratyuk, Liangzhe Yuan, Yandong Li. Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recognition. In CVPR, 2021. 9
[48] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens Van Der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In CVPR, 2022. 9
[49] Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar, Learning video representations from large language models. In CVPR, 2023. 9
[50] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. space-time attention all you need for video understanding? In JCML, 2021.
[51] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022. 9
[52] Quanfu Fan, Rameswar Panda, et al. Can an image classifier suffice for action recognition? In ICLR, 2022.
[53] Roci Herzig, Elad Ben-Avraham, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Object-region vidco transformers. In CVPR, 2022. 9
[54] Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Li. Aim: Adapting image models for efficient video action recognition. ICLR, 2023. 9
[55] Syed Talal Wasim, Muhammad Uzair Khattak, Muzammal Naseer, Salman Khan, Mubarak Shah, and Fahad Shahbaz Khan. Video-focalnets: Spatio-temporal focal modulation for video action recognition, In ICCV, 2023. 9
[56] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion tokeenizzr key to visual generation. In ICLR, 2024. 14
[57] Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hierarchical video dataset for fine-grained action understanding. In CVPR, 2020. 14
[58] Zhihao Liu. Yi Zhang, Wenhui Huang, Yan Liu, Mengyang Pu, Chao Deng, and Junlan Feng. Learning temporal for fine-grained action recognition. In ICIP, 2024. 14, 16
[59] Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In ICCV, 2019. 16
[60] Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Temporal query networks for fine-grained video understanding. In CVPR, 2021. 16
[61] Mci Chee Leong, Haosong Zhang, Hui Li Tan, Liyuan Li, and Joo Hwee Lim. Combined enn transformer encoder for enhanced fine-grained human action recognition. In CVPRW, 2022. 16
[62] Hugo Touvron, Mathiiu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, 2021. 16
[63] Mostafa Dehghani, Alexey Gritsenko, Anurag Arnab, Matthias Minderer, and Yi Tay. Scenie: A jax library for computer vision research and beyond. In CVPR, 2022. 16
13

---

A Appendix
A.1 Choice of tokenizer & quantizer
We choose MAGVIT [25, 56] (which has a 3D CNN encoder) because it is a strong video tokenizer architecture that has been shown to work both for generation and understanding [56] in prior works. For the choice of quantizer, prior works show that Lookup Free Quantizer (LFQ) [56] and Finite
Scalar Quantizer (FSQ) [36] outperform traditional Vector Quantization (VQ) [23]. To study the effect of quantizer on our long-context MAE task, we first train several MAGVIT tokenizers with LFQ and FSQ quantizers at different codebook sizes, and then train MAE at 64 frames using our proposed decoder masking strategy. We present the corresponding results in Table. 5, We find that there is not much difference on the final EPIC- Kitchenst 100 accuracy with these choices. However, FSQ with 18 bit codebook size shows the highest PSNR, the highest EK-100 accuracy and the second best FVD. Based on this result, we choose FSQ as our quantizer.
A.2 Performance on additional datasets
In this section, we share supplementary results two additional benchmarks, Sonething-Somethiin V2 [27] and FineGym288 [57].
A.2.1 Something-Sometthnn V2 benchmark
In Table we test our proposed method on SonethingSomething- [27] dataset, which has an average clip length of 3.8s at 12fps i.e. 46 frames on average, by training Base sized model using our proposed decoder masking strategy with FSQ-MagViT as targets. We gradually increase the number of frames from 16 to 96 for this experiment.
The results match the trends we have seen on the other datasets in Table 4c and Table 2, with longer context providing significant boost in final accuracy. There are diminishing returns from using more than 64 frames of context, only small number of examples from this dataset have clip length this long. Note that our performance exceeds both VideoMAE VI and V2 [14, 15] for the base sized model, despite pre-training for fewer epochs (1600 vs. 2400) and using only a single crop at evaluation time.
A.2.2 FineGym288 benchmark
FineGym288 [57] is a video classification benchmark that, similar to Diving48, tests the ability to categorize multi-second sports action sequences consisting of fine-grained motion, although in this case focusing on gymnastics. The original dataset has ~99 training examples and ~00 validation examples with video length ranging from 13 frames to 877 frames (average 47 frames). During dataset creation process. we add a Is margin to the temporal action boundaries and increase the average video length to 107 frames, in order to exploit cong-context.
In Table 7, we compare against SOTA when pre-training 128 frame model with a VIT-L backbone and using additional unlabeled Kinetics-710 10 data (see section A.5.2). We outperform the current SOTA model, Temporal Cues Transformer [58], despite not using any additional temporal cues as in this work.
A.3 Varying decoder budget
We ablate the effect of the decoder mask ratio on Diving48 top-1 accuracy while keeping the encoder mask ratio fixed at 90% in Table 8. For this experiment we our adaptive masking strategy, 128 frame context for pre-training and fine-tuning, and use adaptive 25Q-MagVin VIT tokens as pre- training targets. For each experiment, we sample adaptive tokens and random tokens in 2:1 ratio. 15% token budget in the decoder yields the highest accuracy, significantly outperforming even the much more expensive setting with 50% token budget that requires >4x more memory than the 15% setting. This aligns with our observations in Table 1. Note that for this experiment, we used improved hyper-parameters across all models, resulting in better results than presented in Table 4d.
Why does lower token budget outperform higher budget? Videos contain redundant information, and we hypothesize that when we reconstruct a lower number of high importance (or higher rank
14

---

Table 5: LFQ vs FSQ. We compare tokenizers trained with different quantization schemes and report their reconstruction quality (PSNR, FVD) on Kinetics600 [37] benchmark and the corresponding MAE model's s EPIC- Kitchens 100 top-1 accuracy.
Quantizer
PSNR[K600] FVD[K600]
Top- Opp-I[EPC--HTT Op--IEEPC-Kiichenss--- 100] f
LFQ 14 bit codebook
23.2
19.4
46.3
LFQ 18 bit codebook
22.6
14.7
46.2
FSQ 14 bit codebook
24.3
24.3
44.1
FSQ 18 bit codebook
25.1
19.1
46.4
Table 6: Samething-Something benchmark. We report top-1 performance of our proposed MAE pre-training with decoder masking & FSQ-Mag ViT as targets while varying the number of frames.
Number of frames
Decoder masking
Eval-Protocol
Top-1 Accuracy
16
None
16 4 clips
67.4
32
None
32 2 clips
69.9
64
85%
64 I clips
71.0
96
85%
96 I clips
70.6
order) tokens during pre-training, the gradients can potentially be stronger than when give equal importance to all tokens, and this would explain the stronger pre-trained encoder. However, we didn't notice this behavior consistently across datasets, namely on Somnething-Somehinnge
A.4 Scaling model size number of frames
Given our decoder efficiency improvements, one can also scale model size instead scaling on the number of frames axis. To study this effect, we first fix the memory budget and token budget to reference Base sized model trained on 128 frames with 15% token budget. Then we vary the model size, and for each model size, maximize the number of frames that can fit in memory budget. In Table 9, we present top-1 accuracy, GFLOPs and the maximum frames on EPIC-Kitchens 100 dataset. We find that with larger model size and fixed memory budget, the accuracy improves despite lower frames but at the cost of significantly increased compute. We reiterate that all the three settings above only made possible due to the memory savings from proposed adaptive masking strategy. We leave further exploration in this direction to future work.
A.5
Implementation details
A.5.1 Adaptive tokenizer
We train our adaptive token selection module following the training recipe from MAGVIT [25] on Kinetics600 dataset training on 16 frame clips, (model card in Table 10), For inference, we produce 8 x 14 x 14 tokens along with the mask that determines the selected tokens.
Once trained, we keep this module frozen and use it across all our experiments. Note, we consciously choose to decouple the learning of token selection and MAE itself, so that the gradients do not bias token selection towards selecting easily unmaskable tokens and vice-versa. To obtain the adaptive mask for long-vin ooggvi,eoss, we simply slide window of 16 frames with a stride of 16 through the tokenizer and token scorer module, and concatenate the resulting tokens and importance masks.
A.5.2 Pre-training on long-videos (128 frames)
For our MAE pre-training we follow VideoMAE architecture [14] and employ a vanilla ViT-B encoder with full space-time attention and 12 layers and a layer decoder with full space- time attention. This architecture will also produce 8 x 14 x 14 tokens, matching the masks and latents produced by the adaptive tokenizer. Given a long video, we first compute the importance mask and select the k = (1 pa) N tokens based on their importance from the video (plus #r x N random tokens). This step can computed offline and only needs be run once per dataset. Next, we use tube masking strategy and encode 10% tokens from the video along with their positional embeddings.
15

---

Table 7: SOTA comparison on FineGym288. We report results on the FineGyy228 benchmark compared to current state-of-the-eee methods.
Method
Per-video Accuracy
TSM [59]
83.1
TQN [60]
89.6
VT-CE [61]
90.1
TCT [58]
92.6
LVMAE
92.8
Table 8: Varying decoder budget. We report Diving48 top-1 performance and relative memory
usage of our proposed MAE pre- pre-training with decoder masking & ASQ0-MaVil1 as targets.
Decoder Budget
Top-1 Accuracy
Memory
5%
88.1
0.6x
15%
89.7
1x
25%
88.7
1.25x
50%
87.5
4.2x
Then, we decode the chosen Na = + x N tokens from these encoded tokens and learnable M ASK tokens along with respective positional embeddings. We make sure that no loss is applied on mecoder-visible tokens. Unless otherwise mentioned, initialize the MAE model from scratch and pre-train on respective downstream datasets for 1600 epochs. Full pre-training hyper-parameters parameters presented in Table 11.
For the case where we pretrain using additional K710 unlabeled data (see Tables 3a and 3b), we first pretrain MAE model on K7 10 using our exact same recipe and hyperparameters as used elsewhere this paper, except only for 800 epochs. We then pretrain a new MAE model on the downstream dataset (e.g. EPICC-Kitchens-100 1000) initializing from the final checkpoint of this first model. We use identical hyperparameters when initializing from pretrained checkpoint in this way as we do when training from scratch.
A.5.3 Fine-tuning on long-videos (128 frames)
Finally, we fine-tune the pre-trained models on respective datasets using standard recipes detailed in Table 11 and report the performance metrics. In all cases we evaluate the final checkpoint of finetuning and average over runs with three different random seeds when reporting metries to minimize variance. Fine-tuning batch size and epochs are typically much smaller than pre-training and hence we can afford fine-tune our models with lower masking ratios (drop-token ratios) fit memory. In particular, we use masking ratio of 20% in the encoder for 128 frame fine-tuning and 0%% for 32 and 64 frame fine-tuning.
When fine-tuning the ViT-L backbone models on Diving48 and Epic- Shiic-Kicchens-i -000 we make the following slight adjustments to hyperparameters:
We use a single layer of class attention [62] as the aggregation method when generating pre- logits opposed to mean pooling. We found this slightly improved accuracy (+0.55 points EPIC-Kitchens- 100 Verbs and +1.2 points on Diving48).
2. We use 25% encoder masking instead of 20% encoder masking to avoid going OOM with the larger model.
3. For Diving48 we finetune for less steps (50 epochs instead of 200 epochs).
A.5.4 Codebase and resources
We implement the code in Scenie [63] and run our pre-training experiments on 128 TPUv5e chips and fine-tuning experiments on 64 TPUv5e chips. Pre-training takes 24hrs and fine-tuning 16hrs in this setting for 128 frame models for PPCC-Kitchens-100.
16

---

Table 9: Model size vs frames. We report top-1 performance of our proposed MAE pre-training with decoder masking & PSQ-MagViT with different model sizes and maximum frames for that model size given fixed memory budget.
Model Size
GFLOPs (relative)
Max frames
EPIC-Kitchens 100 Top-1
Small
0.43x
144
39.1
Base
1x
128
47.3
Large
1.52x
80
48.9
Table 10: Adaptive Tokenizer Model Card
Config
Pre-training
Inference
Number of frames
16 frames, frame stride 1
Spatial resolution
128 x 128
112 x 112
Model size[25]
Base channels[25]
64
64
VQVAE channel multipliers(25]
1, 2,
Discriminator channel multipliers[25]
2, 4, 4, 4,4
Latent spatiotemporal shape
16
16
14 14
Vocabulary size
2 using FSQ[36] with (8,8,4.4.4.44
4] levels
Embedding dimension
top-k for token selection
768
15%
Batch size
256
256
Peak learning rate Learning rate schedule
10-4
linear warm up and cosine decay
Optimizer
Adam with 3 and 83 0.99 Non-saturating
Generator loss type
Generator adversarial loss weight
0.1
Perceptual loss weight
0.1
Discriminator gradient penalty
r1 with cost 0.999
EMA model decay rate
A.6 Visualization adaptive mask
In Figure 3 we visualize our adaptive tokenizer. We can see it focuses the tokens where the relevant motion happening. In Figure 4, we show qualitative comparisons of our adaptive mask with other alternative masking strategies. As expected, optical-flow based masks do very poorly with pure camera motion as they cannot distinguish camera motion from foreground motion. In addition, we find that our mask places more relevance to foreground.
17

---

Video
Adaptive tokens
Video
Adaptive tokens
Figure Our Adaptive tokenizer visualized. We visualize the tokens masks by masking the corresponding input video (repeating frames match the latent temporal dimension).
Table 11: Model Card with detailed model architecture and training setups for Base size experiments on Diving48 and Pitc-Kis
Dataset
Pre-training
Fine-tuning
inputs
pixels
targets
tokens
classes
encoder (layers, heads, MLP dim)
ViT-B (12, 12, 3072)
encoder input shape (xxwxhxc) tubelet dimensions (wxhxt) encoder output shape (txwxha xxxc) decoder (layers, heads, MLP dim) decoder output shape (c)c)
(1 Pe) 32 128xx2xx2222
16x16x2
(1 pe)
16/64x14x14xx66
VIT (4, 4, 1536)
None
(1 Pd + pr) #16 14 14x384
None
optimizer
Adam
Momentum
optimizer momentum
81 0.9, 82 0.95
8 = 0.9
weight decay
0.05
0.0
learning rate learning rate schedule
1.5e 4
0.5
cosine decay
warmup epochs
40
2.5
epochs
1600
EK: 50, Diving: 200 Jitter-Scale, Mixup, RandAug
augmentation
None
batch size
1024|512
64
label smoothing
0.1
0.2
dropout
0.1
0.0
18

---

Video
Flow (RAFT)
Flow
tokens
Random
tokens
Adaptive
tokens (ours)
Figure 4: Token selection strategies visualized. We can see that flow based token selection can be dominated by large background motion. Randomly selected masks unable to focus the tokens the interesting parts of the video. In contrast, we see that the adaptively selected tokens reflect well what is changing in the video.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Section 4 Experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Section 5, Limitations and Broader Impact
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
No theoretical claims and results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: Yes]
Justification: Refer to Appendix A.5 for implementation details and model cards for tok- enizer training (Table 10), MAE pre-training (Table 11) and MAE fine-tuning (Table 11).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [No]
Justification: We will strive to make the code open-source if the paper is accepted but at this point, we cannot share any code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips.coom httppscccppbbiic//ee gttp:/guieeeeeeeeeee /eee:e//oobebmmii ssi ssionoolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Refer to Appendix A.5 for implementation details and model cards for tok- enizer training (Table 10), MAE pre-training (Table 11) and MAE fine-tuning (Table 11),
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Reported for main ablation in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by eπror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources are provided in Section A.5.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpps:///eurips httppp////earas....cbabbb////// csGuidel ines?
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Section 5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We are not releasing any data model checkpoints.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The base codebase is cited in Section A.5.4.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "7XBZ2TEYqb", "reviewer": "Reviewer_LVuv", "summary": "This paper focuses on efficiently extending VideoMAE to much longer videos. It proposes an adaptive decoder masking strategy that utilizes a  MAGVIT tokenizer to localize the importance of each token, which becomes targets that reduce the memory/computation of decoders.\nThe motivation aims to scale the input video along the temporal dimension while maintaining pre-training efficiency. The key innovation is using a tokenizer to generate masks and targets, significantly reducing the training burden. The evaluation mainly focuses on long-video recognition tasks like (Epic-Kitchens-100 and Diving-48).", "questions": "Please see above weakness.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The idea of using tokenization to select important tokens seems novel to me and can significantly improve the decoder's training efficiency in VideoMAE.\nThe ablation of Table 1/2/4 clearly shows each component's effectiveness, while there is space for improvement. \nSoTA performance achieved without using a larger-scale dataset and under an efficient training budget setting seems promising.", "weaknesses": "My main concern is the fixed budget setting (128/32 frames and 15% masking ratio on two datasets), which limits the potential to become a standard baseline for further research. This method can work well with models of various sizes and longer videos, so it is important to show that it also performs well in standard VideoMAE settings (short-video datasets). As the model grows larger, the computation of the decoder increases not only with the number of frames but also with the model size. Therefore, I am eager to see the proposed method’s results (efficiency/performance) on different datasets, number of frames, and mask ratios. In short, I am not convinced by the current results that the proposed masking strategy can be a future research baseline.\n\nThe quality of the tokenizer is another important aspect that needs to be included. Table 4 shows that using MAGVIT as a target can significantly improve performance, indicating there are insights behind this choice. Moreover, the novel token selection module may influence the tokenization results or potentially benefit reconstruction. However, I cannot find tokenizer-related results. Specifically, why does utilizing such a tokenizer significantly reduce the number of tokens? Why choose MAGVIT instead of other options? The 3D tokenizer compresses the temporal dimension several times; could this lead to inferior performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review and for recognizing the novelty, effectiveness and impact of our proposed approach. While we designed our experiments to study the impact of long-video MAE pre-training as such has not been attempted before, we agree with the reviewer that the proposed model can work well across model sizes and video lengths and we also agree that our justification on the choice of tokenizer is lacking. We hope that our below response addresses your concerns:\n\n## Q1: Short-video datasets\nWe favored datasets containing longer action sequences as they can benefit the most from the increased context-length. In addition, we present a study on Something-Something-V2 dataset in the [common response](https://openreview.net/forum?id=bFrNPlWchg&noteId=nljvnUQeki) where we show the same trend as observed in Table 2 of the main paper.\n\n## Q2: Scale model size vs scale number of frames, given our decoder efficiency \nThis is an interesting direction that we haven’t considered in our study as we focused on how to scale the number of frames. Below, we present some preliminary findings where we study the effect of varying model sizes vs varying the number of frames. For this experiment, we first fix the memory budget and masking ratio to the reference base model trained with 128 frames at 15% decoder masking, and try to maximize the number of frames that we can fit in this memory budget. We report final accuracy and FLOPs relative to the base model. \n\n|Model Size|GFLOPs (relative)|Number of frames with constant memory budget| EK-100 Noun-Verb accuracy|\n|----|----|---|----|\n|Small|0.43x|144|39.1|\n|Base (reference)|1x|128|47.3|\n|Large|1.52x|80|48.9|\n\nWe find that with larger model size and fixed memory budget the accuracy improves despite lower frames but at the cost of significantly increased compute. We will include this discussion in our paper.  Finally, we’d like to reiterate that all the three settings above are only made possible due to the memory savings from our proposed strategy.\n\n## Q3: Different number of frames and mask ratios \nOur motivation is to study whether we can effectively pre-train MAE on long-videos and whether such pre-training helps. For this, we selected 128 frames as our target (as it is 8x the typical MAE pre-training frames already), and designed our method around achieving this target. As shown in Figure 1 of the main paper, at 15% masking ratio, the decoder memory becomes less than encoder memory and hence we chose it as our masking ratio. Post-facto, we agree that it’s interesting to study the effect of varying number of frames and mask ratios. To satisfy reviewer’s curiosity, we present some related findings:\n\n### Varying mask ratio\n\nWe ablate the effect of the decoder mask ratio on accuracy while keeping the encoder mask ratio fixed at 10%. For this experiment we use our adaptive masking strategy, a 128 frame context for pre-training and fine-tuning and use adaptive MAGVIT tokens as pre-training targets. A 15% mask ratio yields the highest accuracy, significantly outperforming even the much more expensive setting with 50% mask ratio that requires >4x more memory than the 15% setting.\n\n|Decoder Masking|Diving-48 Accuracy|Memory usage (w.r.t. 15% masking ratio)|\n|----|----|----|\n|5%|88.1|N/A|\n|15%|89.7|1|\n|25%|88.7|1.25|\n|50%|87.5|4.2|\n\n\n\n### Varying number of frames\nIn the table below, we ablate the effect of increasing the number of frames while keeping the mask ratio fixed at 15%. During inference, we ensure that the model sees the same number of frames and report Epic-Kitchens-100 Top1 Noun-Verb Accuracy. We clearly observe a trend of the accuracy improvement with the number of frames similar to what we presented in Table 2 of the main paper. \nNote that both 64 and 128 frame MAE models go OOM without our decoder masking strategy.\n\n|Pre-training frames|Masking|Eval protocol|Epic-Kitchens-100|\n|-----|---|---|---|\n|16|15%|16 x 8 clips|41.7|\n|32|15%|32 x 4 clips|45.0|\n|64|15%|64 x 2 clips|47.1|\n|128|15%|128 x 1 clip|47.3|\n\n\n\n## Q4: Tokenizer clarifications\n``Why does utilizing such a tokenizer significantly reduce the number of tokens?`` \n\nWe believe that the tokenizer can be used to reduce tokens due to two reasons: (1) When we compare with pixels as targets, tokens can encode more surrounding context, and thereby they can have more redundancy. This choice allows us to afford higher masking ratios than used by VideoMAE-V2 [1] with pixels as targets (2) Our adaptive tokenizer training process forces the tokenizer to learn a good importance weighting scheme that gets reflected in the adaptive mask. This lets us have a high masking ratio. In addition, since our mask is learnt independently of MAE, unlike EVEREST [2], our masking scheme outperforms all the other schemes as shown in Table 1 of the main paper.\n\n``Why choose MAGVIT instead of other options?``\nPlease refer to the common response where we discussed the tokenizer and quantizer.\n\n``The 3D tokenizer compresses the temporal dimension several times; could this lead to inferior performance?``\nTemporal downsampling is an inherent result of using the 3D-CNN MagViTv2 architecture (which reduces resolution but provides spatiotemporal features). We have not ablated the effect of temporal compression in the tokens on the downstream performance. However, we believe compression doesn’t necessarily lead to performance drop, as prior works [3] have shown that reconstructing high-level targets in MAE is better than reconstructing low-level targets and this is also corroborated by Table 1 of our main paper.\n\n[1] Wang, Limin, et al. \"Videomae v2: Scaling video masked autoencoders with dual masking.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\\\n[2] Hwang, Sunil, et al. \"EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens.\"\\\n[3] Yu et al. “Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation”. In ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nHauqmCurG", "reviewer": "Reviewer_9WAZ", "summary": "This video understanding paper extends the video mae idea to a longer 128 frames. They use the MAGVIT tokenizer to achieve this and test the approach on Diving-48 and epic kitchens. Both achieved an improved score despite using a pretty standard network and pre-training", "questions": "Why does the approach only make sota for the verbs?\nWhat is the limit on the length of tokens", "limitations": "The idea of focusing on more extended encodings is interesting, but the dataset performance doesn't seem to make it worth it.", "rating": 3, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The use of a MAGVIT encoder masking strategy within a masking strategy of videomae\nThe use of FSQ instead of VQ for the codebook encoding", "weaknesses": "The performance for EK-100 isn't that great.\nThe approach needs to be tested on longer sequence videos, maybe to make a more significant difference than the current two.\nThere is limited innovation on the actual network used", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Differentiating EK-Noun SoTA with EK-Verb SoTA\nWe appreciate the reviewer’s feedback and agree that our EK-100 noun accuracy is below SoTA. However, we would like to humbly point out that SoTA methods for EK-Noun rely on large-scale pretraining, while ours does not. The EK-100 noun categories such as hands, gloves, spoon, knife etc. routinely appear as annotations in large-scale pre-training datasets such as ImageNet21k [1], Ego4D [2] etc. (282/300 nouns from EK-100 also appear in ImageNet21k). As noted by Verbs-In-Action paper [3], verbs are relatively scarce in existing datasets. This explains why methods [4, 5] that use large-scale datasets excel at noun classification. On the other hand, our approach doesn’t use large-scale pre-training datasets and learns long-range spatio-temporal dynamics to achieve absolute SOTA on verb classification. Hence, we argue that our contribution is orthogonal to existing works per Reviewer UhsZ, and we make progress on a different aspect of the problem than data-scaling, namely long-context spatiotemporal actions.\n\n\n### Q1. “Why does the approach only make sota for the verbs?”\nPlease see above discussion, and also find related discussion in lines 297-304 in the main paper. Performing well on nouns requires understanding object semantics which are benefited by large-scale pre-training data, while performing well on verbs requires spatiotemporal understanding. The focus of our paper is on the second question.\n\n\n### Q2. “... The approach needs to be tested on longer sequence videos”\nWe extend the temporal context of MAE encoders from typical 16 frames to 128 frames. We found that reconstructing <15% of tokens can lead to performance drop, and with 15% we are already at the limit of the high-bandwidth memory (HBM) of the TPU accelerators at 128 frames. Orthogonal works use sliding window inference [6, 7], non-differentiable external memory [8, 9], frame filtering etc [10] to scale 16 frame video features to longer sequence tasks. In this paper, we demonstrate the utility of pretraining MAEs with longer context and provide novel techniques for doing so efficiently.  It is exciting future work to combine what we have discovered with these other orthogonal approaches to further scale up to even longer videos.\n\n### Q3. “There is limited innovation on the actual network used”\nOn the encoder part, it is our intended choice to use a well-known model architecture as vision transformers are the model of choice for video foundation models [11, 12, 13], allowing potential adoption of our method. While MAE architecture has minimal modification, our dual innovation is in (1) designing an adaptive tokenizer and (2) designing a methodology of using importance score from adaptive tokenizer for latent reconstruction to achieve the required decoder efficiency while maintaining / improving performance.\n\nWe refer the reviewer to the qualitative results in Figure 3 and Figure 4 of our Appendix that shows that our method of selecting tokens is indeed superior to random selection or flow based methods, and this is corroborated by our empirical results from Table 1 of the main paper.\n\n### Q4. “What is the limit on the length of tokens”\nWe limit the number of decoder tokens to be in the same ballpark as the number of encoder tokens in our MAE pre-training setup, so that the memory, FLOPs and training time are not disproportionately affected by the decoder. Typically, MAE encoders use only 10% of the video tokens and are trained for 1600 to sometimes 3200 epochs [14]. However, as shown in Figure 1 (right) of our main paper, decoder memory and FLOPs disproportionately dominate at 128 frames making such training inefficient. Hence, it is crucial to limit the number of tokens to be approximately same as the number of encoder tokens during pre-training.\n\n\n\n### Q5. “... dataset performance doesn't seem to make it worth it.”\nIn Tables 3 (a) and 3 (b), we show SOTA performance on EK-100 verbs and Diving48. We further improved these numbers leading up to the rebuttal as shared in the common response. Moreover, we pre-train in an unsupervised manner without relying on large-scale datasets compared to related approaches.\n\n\n[1] Ridnik, Tal, et al. \"Imagenet-21k pretraining for the masses.\" arXiv preprint arXiv:2104.10972 (2021). \\\n[2] Grauman, Kristen, et al. \"Ego4d: Around the world in 3,000 hours of egocentric video.\" In CVPR. 2022. \\\n[3] Momeni et al., “Verbs in Action: Improving verb understanding in video-language models”. In ICCV 2023. \\\n[4] Zhao, Yue, and Philipp Krähenbühl. \"Training a large video model on a single machine in a day.\" arXiv preprint arXiv:2309.16669 (2023). \\\n[5] Xiong, Xuehan, et al. \"M&m mix: A multimodal multiview transformer ensemble.\" arXiv preprint arXiv:2206.09852 (2022). \\\n[6] Wang, Xiang, et al. \"Proposal relation network for temporal action detection.\" arXiv preprint arXiv:2106.11812 (2021). \\\n[7] Chen, Guo, et al. \"Video mamba suite: State space model as a versatile alternative for video understanding.\" arXiv preprint arXiv:2403.09626 (2024). \\\n[8] Wu et al., “Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition”. In CVPR 2022. \\\n[9] Balazevic et al., “Memory Consolidation Enables Long-Context Video Understanding”. arXiv 2402.05861. \\\n[10] Tan et al., “Koala: Key frame-conditioned long video-llm”. In CVPR 2024. \\\n[11] Wang, Yi, et al. \"Internvideo: General video foundation models via generative and discriminative learning.\" arXiv preprint arXiv:2212.03191 (2022). \\\n[12] Li, Kunchang, et al. \"Unmasked teacher: Towards training-efficient video foundation models.\" In ICCV 2023. \\\n[13] Zhao, Long, et al. \"Videoprism: A foundational visual encoder for video understanding.\" arXiv preprint arXiv:2402.13217 (2024). \\\n[14] Tong, Zhan, et al. \"Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.\" Advances in neural information processing systems 35 (2022): 10078-10093.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ckadNuQOya", "reviewer": "Reviewer_UhsZ", "summary": "The paper proposes a novel MAGVIT-based adaptive tokenizer & masking module to extend VideoMAE to 128 frames. The tokenizer & masking module is individually trained and applied offline, making it possible for VideoMAE to reconstruct sparser (but important) and more semantically informative targets. The experimental results show that the approach allows for significant memory savings, enabling pre-training on longer video clips and leading to improved performance on downstream long video understanding tasks.", "questions": "1. In Table 4a, the 10%+5% masking even exceeds the case without decoder masking, and perhaps the authors could provide more explanation for this phenomenon.", "limitations": "The author discusses the limitations of the work, which I guess are largely due to resource limitations.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow, with clear explanations of the proposed method and experimental results.\n2. The paper addresses an important issue in video understanding, namely the scalability of video masked modeling for longer sequences. The approach in this paper is relatively orthogonal to prior work in the field.\n3. The proposed module is individually trained and applied offline, which avoids the training difficulties and extra overhead in videomae pretraining that come with the complex designs.\n4. The proposed module provides both decoder masking and high semantic information tokens, which saves computational overhead and improves model performance. The effectiveness of each part is verified through ablation experiments.", "weaknesses": "1. The major weaknesses of this paper include the limited dataset sizes, small model sizes, and narrow range of task types, as mentioned in Section 5. I'm interested in the model's performance on action detection and temporal action detection tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers comments on our work being well-written, its importance in the field of video understanding, orthogonality to existing works, and that the effectiveness of our approach is verified through ablation experiments. Below, we address the additional questions raised by the reviewer.\n\n### Q1. Limited dataset sizes and small model sizes\n\nWe tested our approach on conventionally used model sizes that include Base (B) and Large (L) transformer variants on two of the widely used datasets, namely EPIC-Kitchens-100 and Diving48. Although there exists literature [1, 2, 3, 4] on large-scale pre-training and large model sizes for video understanding, we focussed our study on answering the question of how we can effectively pre-train MAEs on 128 frames and whether such pre-training helps. Extending our approach and adopting it in large-scale models is definitely an exciting prospect that we would like to explore, however it is beyond the reach of the current work.\n \n### Q2. Limited and narrow range of tasks\n\nDuring the rebuttal period, we additionally tested our method on Something-Something-v2 dataset. We presented the corresponding results in the [common response](https://openreview.net/forum?id=bFrNPlWchg&noteId=nljvnUQeki) and observe the same trend as seen in Table 2 of the main paper. Due to time/resource constraints of the rebuttal period we were not able to run similar ablations for models with a VIT-L backbone and for different task types including temporal action localization, but we hope to include these experiments as well in a future version.\n\n### Q3. “10%+5% masking even exceeds the case without decoder masking” \nWe appreciate that the reviewer noticed this. Videos contain redundant information, and we hypothesize that when we reconstruct a lower number of high importance (or higher rank order) tokens during pre-training, the gradients can potentially be stronger than when we give equal importance to all tokens, and this would explain the stronger pre-trained encoder. However, we didn’t notice this behavior consistently across datasets, namely on SSv2. We will include this discussion in the paper.\n\n\n[1] Wang, Yi, et al. \"Internvideo: General video foundation models via generative and discriminative learning.\" arXiv preprint arXiv:2212.03191 (2022). \\\n[2] Li, Kunchang, et al. \"Unmasked teacher: Towards training-efficient video foundation models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. \\\n[3] Zhao, Long, et al. \"Videoprism: A foundational visual encoder for video understanding.\" arXiv preprint arXiv:2402.13217 (2024). \\\n[4] Wang, Yi, et al. \"Internvideo2: Scaling video foundation models for multimodal video understanding.\" arXiv preprint arXiv:2403.15377 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bszp8vdxHd", "reviewer": "Reviewer_EcJb", "summary": "This paper studies the MAE pretraining of long videos. To address the challenges of hardware memory and compute limitations, this paper propose an effective strategy of decoder masking, subsampling tokens as reconstruction targets. This strategy leverage the MAGVIT-based tokenizer, prioritizes the most important tokens and uses quantized tokens as reconstruction objectives. The proposed decoder masking strategy outperforms previous masking strategies on EK-100 and Diving48. With 128 frames as input for pretraining and finetuning, the video encoder obtain better performance compared with the short-video (32 frames) pretraining.", "questions": "1. Why does this work convert MAGVIT to FSQ-MAGVIT for reconstruction targets and adaptive token selection?\n\n2. Results on other long-form video downstream tasks.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Study the MAE-based pretraining of long videos. The proposed adaptive decoder masking strategy solves the OOM problem of long-video training and beats some previous masking strategy on downstream tasks.\n\n2. Compared to 32 frames MAE-based pretraining, the proposed method with 128 frames obtains better performance on EK-100 and Diving48.", "weaknesses": "1. This paper only shows the results of 32 frames input and 128 frames input. To figure out the best setting of input frame number, it would be better to show experimental results when increasing the frame number from 16 to 128 gradually for pretraining and finetuning (while keeping the similar computational cost at inference). \n\n2. This paper does not include any experiments or theory analysis to explain why to convert MAGVIT to FSQ-MAGVIT for reconstruction targets and adaptive token selection.\n\n3. Only 2 downstream tasks are used for the main experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable feedback on our work, and for noticing the effectiveness of our method in improving both pre-training efficiency and downstream accuracy. Below, we address some of the concerns and questions raised by the reviewer. \n\n### Q1. Gradually increasing the number of frames:\nOur motivation is to study whether we can effectively pre-train MAE on long-videos and whether such pre-training helps downstream tasks. For this, we selected 128 frames as our target (as it is 8x the typical MAE pre-training frames already), and designed our method around achieving this target. Post-facto, we agree that it’s interesting to study the effect of varying number of frames. To satisfy reviewer’s curiosity, we present some related findings:\n\nIn the table below, we ablate the effect of increasing the number of frames while keeping the mask ratio fixed at 15%. During inference, we ensure that the model sees the same number of frames and report Epic-Kitchens-100 action Top1 Accuracy.  We clearly observe a trend of the accuracy improvement with the number of frames similar to what we presented in Table 2 of the main paper. \n\n|Pre-training frames|Masking|Eval protocol|Epic-Kitchens-100|\n|-----|---|---|---|\n|16|15%|16 x 8 clips|41.7|\n|32|15%|32 x 4 clips|45.0|\n|64|15%|64 x 2 clips|47.1|\n|128|15%|128 x 1 clip|47.3|\n\n\nBelow, we repeat the same experiment without decoder masking and observe OOM for 64 and 128 frames, whereas our proposed decoder masking method unlocks additional gains in the previous table. This further demonstrates the effectiveness of our method.\n\n|Pre-training frames|Masking|Eval protocol|Epic-Kitchens-100|\n|----|---|---|----|\n|16|None|16 x 8 clips|41.3|\n|32|None|32 x 4 clips|44.1|\n|64|None|64 x 2 clips|OOM|\n|128|None|128 x 1 clip|OOM|\n\n\n### Q2. \"Why does this work convert MAGVIT to FSQ-MAGVIT for reconstruction targets and adaptive token selection?\"\nWe thank the reviewer for raising this question. We have now added a justification in the [common response](https://openreview.net/forum?id=bFrNPlWchg&noteId=nljvnUQeki) on our choice of tokenizer and the quantizer, and we request the reviewer to consult the same. Our studies showed that our choice of FSQ achieved the highest PSNR and the highest downstream EK100 accuracy, however the differences in downstream EK100 accuracies are not much with different choice of quantizers. \n\n\n### Q3. Only 2 downstream tasks are used for the main experiments.\n\nDuring the rebuttal period, we additionally tested our method on Something-Something-v2 dataset. We presented the corresponding results in the [common response](https://openreview.net/forum?id=bFrNPlWchg&noteId=nljvnUQeki) and observe the same trend as seen in Table 2 of the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Summarizing feedback
We thank all the reviewers for their valuable time and feedback. We are encouraged by the positive feedback from all the reviewers who found our work
- Novel (Reviewer LVuv, Reviewer UhsZ), orthogonal to prior work (Reviewer UhsZ), and addresses an important issue in video understanding (Reviewer UhsZ), 
- Solves the OOM problem of long-video training and beats previous masking strategies (Reviewer EcJb), saves computational overhead and improves model performance (Reviewer UhsZ), significantly improves the decoder’s training efficiency in VideoMAE (Reviewer LVuv)
- Achieves SoTA without using large-scale dataset (Reviewer LVuv), improved scores using a pretty standard network and pretraining (Reviewer 9WAZ), Compared to 32 frames, the proposed method with 128 frames obtains better performance (Reviewer EcJb)
- Verifies the effectiveness of each part through ablations (Reviewer UhsZ, Reviewer LVuv)
- Is well-written and easy to follow, with clear explanations of the proposed method and experimental results (Reviewer UhsZ)

## Further improvements to results:

Leading up to the rebuttals, we discovered that some of our hyper-parameter choices such as learning rate, whether to use cls token during fine-tuning etc. were not optimal, and we have obtained even better results than reported in the main paper (93.7 → 94.9 on Diving48 +3.9 over SoTA, and 75.0 → 75.5 on EpicKitchens-100 Verbs +2.5 over SoTA). 


## Addressing common concerns:
We appreciate the feedback and the questions, and we address some common concerns below and address specific questions and concerns of the reviewers in the respective rebuttals. We hope that these adequately answer the reviewers’ concerns and we are happy to continue discussions, and delve into any details regarding our proposed method.

### Limited number of datasets and task types:

During the rebuttal period, we additionally tested our method on a short-clip dataset, Something-Something-v2, which has an average clip length of 3.8 seconds at 12fps, i.e. 46 frames using a Base sized model.

|Number of frames|Decoder Masking|Eval protocol|SSv2 val accuracy|
|-------|-----|-----|-----|
|16|None|16 x 4 clips|67.4|
|32|None|32 x 2 clips|69.9|
|64|15%|64 x 1 clips|71.0
|96|15%|96 x 1 clips|70.6

The results match the trends we have seen on the other datasets in Table 2 of the main paper, with longer context providing a significant boost in final accuracy. There are diminishing returns from using more than 64 frames of context, as only a small number of examples from this dataset have clip length this long.  Note that our performance exceeds both VideoMAE v1 [1] and VideoMAE v2 [2] for the base sized model, despite us pre-training for fewer epochs (1600 vs. 2400) and using only a single crop at evaluation time.  

Due to time/resource constraints of the rebuttal period we were not able to run similar ablations for models with a VIT-L backbone and other downstream tasks, but hope to include these experiments as well in a future version. We will include this discussion in the final version of the paper.

### Choice of tokenizer and quantizer:

We chose MagViTv2[3, 4] (which has a 3D CNN based encoder) because it is a strong tokenizer architecture which has been shown to work both for generation and understanding [4] compared with prior works. On the choice of quantizer, prior works show that Lookup Free Quantizer (LFQ) [4] and Finite Scalar Quantizer (FSQ) [5] outperform traditional Vector Quantization (VQ) [6]. To study the effect of quantizer on our long-context MAE task, we performed several experiments comparing LFQ vs FSQ with MagViT as the tokenizer, and found that there is not much difference on the final EK-100 accuracy with these choices. However, FSQ with 18 bit codebook size showed the highest PSNR, the highest EK-100 accuracy and the second best FVD. So, we chose it as our quantizer. The corresponding study is presented below and we will include this in our final submission.

|Tokenizer|PSNR [K600]|FVD [K600]|Epic-Kitchens-100|
|---|---|---|----|
|LFQ 14 bit codebook|23.2|19.4|46.3|
|LFQ 18 bit codebook|22.6|14.7|46.2
|FSQ 14 bit codebook|24.3|24.3|45.1
|FSQ 16 bit codebook|24.7|21.9|46.0
|FSQ 18 bit codebook|25.1|19.1|46.4



[1] Tong, Zhan, et al. "Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training." Advances in neural information processing systems 35 (2022): 10078-10093. \
[2] Wang, Limin, et al. "Videomae v2: Scaling video masked autoencoders with dual masking." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. \
[3] Yu et al. “Magvit: Masked generative video transformer.” In CVPR 2023. \
[4] Yu et al. “Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation”. In ICLR 2024. \
[5] Mentzer, Fabian, et al. "Finite scalar quantization: Vq-vae made simple." arXiv preprint arXiv:2309.15505 (2023). \
[6] Yu, Jiahui, et al. "Vector-quantized image modeling with improved vqgan." arXiv preprint arXiv:2110.04627 (2021).

Meta-review: The reviewers praised the paper's novelty, writing style, and clear explanations of the proposed method and experimental results. They also acknowledged the importance of addressing the scalability of video masked modeling for longer sequences.

While some reviewers initially expressed concerns about the limited scope of the experiments, the authors have since provided additional results on an additional dataset (SSv2), number of frames, and mask ratios, demonstrating the robustness of their approach. The authors have also clarified the quality of the tokenizer used and its impact on performance. Furthermore, the authors have addressed the concern about testing on longer video sequences, providing new results that show the effectiveness of their approach on longer sequences.

Overall, the authors have successfully addressed the concerns raised by the reviewers, providing additional experiments and explanations that strengthen the paper's contributions. The paper has undergone significant improvements since the initial submission, and the authors have demonstrated a thorough understanding of the reviewers' concerns. 

After carefully reviewing the reviews and rebuttal, this meta-reviewer concluded that this paper is ready for publication and will make a valuable addition to the research community.
Decision: Accept (poster)