Extending Video Masked Autoencoders to 128 frames

Authors: nitesh b gundavarapu
luke  friedman
raghav  goyal
chaitra  hegde
eirikur  agustsson
sagar  waghmare
mikhail  sirotenko
ming-hsuan  yang
tobias  weyand
boqing  gong
leonid  sigal
google  research
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 00:19:23

Abstract: Video understanding has witnessed significant progress with recent video foundation models demonstrating strong performance owing to self-supervised pre-training objectives;  Masked Autoencoders (MAE) being the design of choice. Nevertheless, the majority of prior works that leverage MAE pre-training have focused on relatively short video representations (16 / 32 frames in length) largely due to hardware memory and compute limitations that scale poorly with video length due to the dense memory-intensive self-attention decoding.  One natural strategy to address these challenges is to subsample tokens to reconstruct during decoding (or decoder masking). In this work, we propose an effective strategy for prioritizing tokens which allows training on longer video sequences (128 frames) and gets better performance than, more typical, random and uniform masking strategies. The core of our approach is an adaptive decoder masking strategy that prioritizes the most important tokens and uses quantized tokens as reconstruction objectives. Our adaptive strategy leverages a powerful MAGVIT-based tokenizer that jointly learns the tokens and their priority. We validate our design choices through exhaustive ablations and observe improved performance of the resulting long-video (128 frames) encoders over short-video (32 frames) counterparts. With our long-video masked autoencoder (LVMAE) strategy, we surpass state-of-the-art on Diving48 by 3.9 points and EPIC-Kitchens-100 verb classification by 2.5 points while relying on a simple core architecture and video-only pre-training (unlike some of the prior works that require millions of labeled video-text pairs or specialized encoders).

Main paper: 
 introduction: 
 Long video understanding has witnessed growing interest with various aspects of the problem having been explored by recent works [1, 2] . This includes incorporating (i) efficient attention mechanisms [3, 4] , (ii) designing memory modules [5] [6] [7] to reason over context from the past, and (iii) approaching the problem from video-language perspective by proposing benchmarks [2, 8] and architectural choices that effectively leverage the interplay between the modalities [9-11, 8, 12] .The majority of recent models rely on foundational models pre-trained in a self-supervised manner based on videos and/or, for vision-language counterparts, video-text pairs; these foundational models, once pre-trained, can then be fine-tuned for a specific task at hand (e.g., action classification). Masked Figure 1 : Left: Proposed Long Video MAE Decoder Masking. We leverage a jointly trained adaptive tokenizer and importance module to define a decoder mask and token targets for a video MAE pre-training strategy. The resulting sparsification in tokens (only 15%) allows pre-training with long videos (128-frames) and results in substantial performance gains. Right: Decoder masking and memory in long-video (128 frames) pre-training. We report memory and FLOPs per-device for a batch size of 1 using different decoder mask ratios and ViT-B architecture.Autoencoders (MAE) have emerged as a simple and effective strategy to masked video modeling [13, 14] in this context. The standard MAE setup involves encoding a small fraction of visible / unmasked tokens (e.g., 10%) of an input video, and decoding the remaining masked tokens (e.g., 90%) using dense space-time attention decoder blocks. However, for longer video sequences, reconstructing all the masked tokens at the decoding stage quickly leads to out-of-memory (OOM) due to quadratic complexity of Transformers (feasible only for < 64 frames for moderate video resolution and current consumer hardware).As a result, most existing MAE-based video understanding approaches focus on learning representations that encode only a few frames (16 [15] / 32 [14] ) at a time. This limits their ability to understand actions and events spanning longer time horizons. Existing work has worked around the short context limitation by segmenting longer input video into short chunks and feeding them into the video model sequentially. Inference results from these chunks are then combined using late fusion. For example, "multi-crop" evaluations simply average-pool predictions over all chunks. However, this is clearly limiting. Long-context video models have the potential to understand longer actions (like complex diving routines), whole activities consisting of a sequence of actions and eventually entire story arcs.Attempts to reduce computational cost and memory consumption, which bottleneck video MAE scalability, have recently started to emerge. These efforts can be characterized by either (i) reconstructing a subset of tokens at the decoding stage or decoder masking (VideoMAEv2 [14] ), or by (ii) sub-sampling and focusing on a subset of "informative" tokens instead of using the entire set of spatio-temporal tokens (e.g., focusing on measures of objectiveness [16] or motion [17] as proxies). The resulting gains have been shown to reduce training time and hardware requirement [17] , and can be reinvested to scale model size [14] . However, scaling the input video along the temporal dimension has received little-to-no attention besides scaling input frames to a higher spatial resolution [14] .In this work, we build on [14] which performs masking at both the encoding and decoding stages.In particular, we focus on decoder masking considering its impact on memory and scalability, and propose a content-dependent adaptive masking strategy for videos in the MAE setup (see Figure 1 ). The core idea is a token-importance (or saliency) scheme, learned jointly with adaptive quantization, that establishes a rank-order of video tokens, which is then used to select higher-ranking tokens for decoding. The jointly learned quantization is shown to also be effective, compared to raw pixels, in defining the targets for reconstruction -further improving performance.Our primary contributions are as follows:1. We design a content-dependent adaptive masking strategy and demonstrate that given a low-token budget, it outperforms prior uniform and motion-based masking strategies. 2. The memory savings obtained from the low-token budget enables pre-training over longvideos (128 frames), allowing us to ask the question of exactly how much benefit long videos bring in the context of MAE pretraining. We observe that the long-video MAE (128 frames) model consistently outperforms short-video MAE (32 frames), as measured using downstream fine-tuning performance; including when a short-video MAE is fine-tuned with longer context (e.g., 32 frame pre-trained MAE fine-tuned on 128 frames). 3. Leveraging these findings, we demonstrate state-of-the-art performance with our long-video MAE (LVMAE) approach on conventional action classification benchmarks that are known to require longer-range motion understanding (EPIC-Kitchens-100 [18] and Diving48 [19] ) using standard ViT encoders and just one temporal crop -without relying on any language supervision or labels during pre-training. 
related work: 
 Masked Autoencoders (MAE). Masked autoencoders have initially been proposed as an effective mechanism for learning self-supervised representations in masked image modeling (MIM) [20] .Following their success in the image domain, several video MAE-based representation learning extensions have been proposed. Specifically, ST-MAE [13] extends MIM to spatio-temporal patches in clips -treating space-time blocks as tokens, while VideoMAE [15] explores frame-level and tube-level masking strategies; both leverage an extremely high masking ratio (∼ 90%) compared to image counterparts (∼ 60%). Models that leverage joint image-video pre-training have also been introduced (e.g., BEVT [21] , OmniMAE [22] ). However, scalability to high-resolution and long videos remains a substantial challenge, owing to the quadratic complexity of attention mechanisms.Objectives. While most video MAE models are trained to reconstruct the pixels of masked video patches [13, 15] , there are a few notable exceptions. Mainly, BEVT [21] proposed to reconstruct discrete token targets obtained using a VQ-VAE [23] patch tokenizer, while MVD [24] focuses on masked feature modeling and distillation with high-level features as targets, obtained using a teacher model. Compared to BEVT, we see even greater gains when reconstructing discrete token targets by using a more powerful tokenizer (MAGVIT [25] ) and an improved, jointly trained, tokenization and masking scheme.Efficiency by Token Reduction. Sub-sampling salient tokens, based on a learned or an off-the-shelf strategy during training, is shown to give faster convergence and require less resources [17, 16, 26] .ObjectViViT [16] extracts object bounding boxes using an off-the-shelf object detector, and by placing emphasis on tokens belonging to objects was able to achieve lower token utilization and obtain competitive performance on the Epic Kitchens [18] and SSv2 benchmarks [27] . Token Learner [26] observes and corrects for token redundancy in higher layers of ViT [28] using learned modules.Similar to the above approaches, we use token importance to inform our proposed masking. Differently, we don't sub-sample tokens, but selectively decode them during the training phase in a more generalizable dual masking setup. Concurrent EVEREST [17] is the most comparable to our work. It attempts to learn what tokens in the spatio-temporal volume are more informative using a heuristic based on distance in feature space (as a proxy for motion). In the short 16-frame regime they find that sub-sampling such tokens in MAE setup leads to lower memory and resource consumption, while maintaining performance. Different from [17] , our token saliency scheme is learnt separately and independently in a MAGVIT tokenizer [25] setup, while EVEREST learns token-saliency during MAE pre-training itself. The latter approach can bias the method towards selecting easy tokens for the MAE reconstruction objective, undermining the learning, which is not an issue for our method. By learning token importance jointly with the tokenizer we incur a negligible extra cost, on top of the tokenizer computation, and as noted earlier get significant additional performance improvements by using the tokens as targets as opposed to RGB in [17] .Motion in Video MAE. Building on the above, attempts to identify and attribute relevance to video tokens involved in motion compared to static background has seen growing interest [29, 30] , since motion is fundamental to understanding videos. In particular, MGMAE [29] and MGM [30] show that an informed encoder masking based on motion-cues at patch-level makes the MAE reconstruction task account for motion, and found faster convergence and improved performance on motion-centric video datasets such as SSv2 [27] . Specifically, MGM [30] uses H.264 codec [31] to extract patch-level motion vectors, and mask out tokens involved in motion during encoding, and MGMAE [29] uses online optical flow extractor (RAFT [32] ), to warp and guide encoder masking using flow. Unlike these approaches, we do not model motion explicitly, and any apparent motion information in our mask is a byproduct of our data-driven token importance learning scheme.Long Videos and Masking. As discussed above, masking tokens during training reduces memory requirement and this fact is exploited in several video understanding works for scaling model size [33, 34] . However, expanding these techniques for long-videos has received limited attention. Recently, LongViViT [9] leveraged random masking during contrastive pre-training on video-language tasks, and observed best performance-memory trade-off. Specifically, in order to convert short-video encoder to long-videos, they fine-tune the last four layers of ViViT [35] on long videos (128 frames) along with 75% random masking. Similar to LongViViT [9] , we utilize masking to pre-train over long videos. However, we focus on masked autoencoders instead of contrastive learning for long-videos and pre-train all the layers of our model, instead of partial freezing, in order to capture dense spacetime correspondence in long-videos. Further, the overall training procedure of LongViViT [9] is considerably more complex requiring first pre-training with short videos and then further pre-training with long ones. We on the other hand, not only have a simpler scheme that can directly pre-train a MAE model with long videos (128 frames), but show that this is critical for improved performance.Long Videos and Memory. Memory-based approaches [5, 6] attempt to form a compressed representation of the past activations or memory, which is then incorporated into current window or time step, effectively prolonging the context length over which reasoning and predictions are formed.Orthogonal to these works, we expand the local context of clips during pre-training from 16 to 128 frames; this can be combined with a memory module to increase the global context window.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See Section 5, Limitations and Broader Impact Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
approach: 
 We first give the background on MAE and the dual masking, initially introduced in [14] . We then focus on describing our proposed adaptive importance masking strategy and discuss how it can be leveraged to train with up to a 128 frame context window.An input video V ∈ R 3×F ×H×W is first partitioned into non-overlapping spatio-temporal patches, and tokenized using a patch embedding layer (typically a 3DConv) to give tokens T = {T i } N i=1 , where T i ∈ R d is an i th token with added positional encoding, N is total number of tokens, d is the hidden dimension, and F , H, W are number of frames, height and width respectively of the input video. Using an encoder mask M e ∈ {0, 1} N , the unmasked / visible tokens are selected T u = {T i } i∈(1-Me) , with N e the total number of unmasked encoder tokens. A vanilla ViT encoder [28] is applied to the unmasked tokens to give encoded visible tokens Z = Φ enc (T u ). In the usual case, we obtain input to the decoder Z c by combining the encoded tokens Z with learnable masked tokens M = {M i } i∈Me , where M i ∈ R d is [MASK] token embedding with positional embedding. The overall objective is to reconstruct the masked-out tokens using the unmasked encoded tokens. However, with Dual Masking [14] , a decoder mask M d ∈ {0, 1} N is used to select tokens to reconstruct Z c = Z ∪ {M i } i∈(1-M d ) , with N d the total number of unmasked decoder tokens where N e << N d and N e + N d << N . Then, the combined tokens are reconstructed using a vanilla ViT decoder V = Φ dec (Z c ). Finally, the Mean Squared Error (MSE) loss is computed between the original and reconstructed pixels,L = 1 |M e ∩ M d | i∈Me∩M d | Vi -V i | 2 .(1)Alternatively, if vector quantized tokenization is employed, the loss becomes,EQUATIONwhere V Q(•) is a vector quantization mapping typically trained separately using VQ-VAE or variants.In the proposed approach V Q(•) takes the form of adaptive FSQ-MagViT described in Section 3.3. Further, we want to highlight that encoder and decoder masking strategies need not be the same.The above dual masking formulation requires both encoder and decoder masking. A number of strategies have been explored, but largely fall into two categories: content agnostic and informative.Content agnostic masking strategies leverage either a fixed or randomized scheme which is agnostic of the video content. Fixed strategies comprise of (i) grid-based uniform masking [15] , which keeps every k-th row/column along spatial dimension; and (ii) frame masking [14] where all tokens from every k-th frame are kept in an attempt to reduce temporal redundancy. The choice between the two depends on the assumptions regarding relative importance of spatial vs. temporal information. (iii) Randomized strategies [13, 15] leverage sampling instead, which results in a form of data augmentation, since the same clip encountered in different epochs would result in a different mask. Under low-token budget, the above mentioned approaches are shown to perform competitively, and at the same time, gain memory efficiency [17] .Masking approaches mentioned above are effective, but at their core are sub-optimal as they fail to take into account the content of the video itself. Our approach falls into an informative masking class of strategies, which instead of sub-sampling overall set of tokens, leverage an importance function to produce a relative rank-order (or priority) on input tokens and use this ordering when selecting which tokens to encode/decode. We denote the token-importance (or saliency) as S ∈ [0, 1] N , where N is the total number of tokens in an input video. In particular, S is a probability distribution over tokens, which gives us a rank-order of tokens. Letting S be a probability distribution allows both deterministic (k most probable) and stochastic forms of informed masking. Prior work has relied on optical flow [29, 31] as proxy for importance, which in this work we leverage as one of our baselines. We, on the other hand, propose a more general adaptive FSQ-MagViT scheme that learns importance of tokens concurrently with tokenization (see Section 3.3), leveraging a MAGVIT [25] tokenizer.While the above mentioned strategies can be used to generate both encoder and decoder masks in the dual masking MAE [14] (with the potential mild overhead of keeping the two sets of tokens disjoint), in this paper we focus on decoder masking in particular. This design choice is motivated by the observation that decoder masking has a much more significant impact on the memory usage and scalability of the overall MAE framework (see Figure 1 ) (right). Our focus on informative and effective decoder masking allows us to pre-train on long-video sequences (128 frames) -see Figure 1 (left). To this end we keep the encoder masking for most experiments in this paper relatively simple.Encoder masking. Motivated by results in [15] for most experiments in the paper (unless otherwise stated) we use randomized tube masking with ratio of 90% (i.e., encoder only sees 10% of patches).Decoder masking. We generate Adaptive decoder mask M d based on the intuition that reconstructing high-salient tokens is more meaningful for the MAE pre-training and for downstream fine-tuning tasks. We select top-k salient tokens from S and set them as visible (indicated as 0's) in the decoder mask M d ∈ {0, 1} N , where k = (1 -ρ d ) × N and ρ d is decoder masking ratio. In addition to selecting top-k tokens, we also include a small fraction (ρ r ) of tokens sampled randomly, which we observed to improve performance, similar to ObjectViViT [16] . Therefore,N d = (1 -ρ d ) × N + ρ r × N .Illustration of resulted mask is shown in Appendix A.6.Our adaptive token selection module, shown in Figure 2 , combines an effective CNN-based tokenizer and a differentiable top-k selection layer.Tokenizer. For the tokenizer we combine MAGVIT [25] , which is a powerful 3D-CNN based video tokenizer and Finite Scalar Quantization (FSQ) [36] , a simple alternative to VQ that easily scales up to large codebooks. We refer to this combination as FSQ-MagViT and provide more details in Appendix A.5.1.Token Scorer For the token selection, we learn a CNN-based Token scoring module followed by a differentiable top-k layer, resulting in the mask which we apply to the FSQ-MagViT tokens. 1 More precisely, we embed the video to a feature space with two CNN layers, strided both spatially and temporally to match the resolution of the tokens from FSQ-MagViT encoder. From this CNN feature, we obtain pairwise Euclidean distances between spatial locations in adjacent frames, measuring how much each token in the i-th frame differs from the corresponding token in the (i -1)-th frame. This distance is considered as token importance as it signifies the extent of underlying change in the video as done in [17] .Adaptive FSQ-MagViT We get the soft-tokens from FSQ-MagViT and token importance from the token scoring module. We keep the tokens with the top-k largest distances (k = 768) and mask out the remaining N -k tokens, replacing their values with 0. For the special case of the first frame we keep all the tokens during training. These masked tokens are then fed through FSQ and the MagViT decoder. Similar to MAGVIT [25] , we optimize for pixel reconstruction loss, GAN loss and perceptual loss by training the FSQ-MagViT and token scorer end-to-end.First, we train our adaptive token selection module on Kinetics600 [37] data on 16 frame clips. Once trained, we keep this module frozen. Next, we follow standard MAE pre-training with a couple of changes (i) we reconstruct the top-k tokens selected by the token selection module (and a small number of randomly selected tokens), and (ii) we reconstruct the latent embeddings from the tokenizer instead of RGB pixels. Note, we consciously choose to decouple the learning of token selection and MAE itself so that the gradients do not bias token selection towards selecting easily unmaskable tokens and vice-versa. Further implementation details are in Appendix A.5.1.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The compute resources are provided in Section A.5.4. 
experiments: 
 We experiment with LVMAE on conventional video action classification benchmarks that have potential to benefit from long-video encoders: EPIC-Kitchens-100 (EK100) [18] and Diving48 (D48) [19] . EK100 [18] contains ∼ 90K video clips in total at 25 FPS with 3.7 secs avg. duration, and ∼ 15% videos with at least 5 secs (=125 frames) duration. The task is to classify nouns (=300) and verbs (=97) together. D48 [19] contains videos with 158 avg. number of frames, and vary from 24 to 822 frames, and task to classify among 48 fine-grained dive categories [5] . Results on additional datasets are presented in Appendix A.2.In Figure 1 (right), we show the memory and compute characteristics of ViT-B MAE architecture as we vary the decoder masking ratio for a video length of 128 frames. We fix the encoder masking ratio to 90% for this comparison. Even though the encoder has 3X more layers, decoder starts dominating for long videos due to the high number of tokens and quadratic scaling. We find that the decoder masking ratio should be close to the encoder masking ratio to respect compute and memory constraints imposed by the long-video pre-training regime.In Table 1 , we compare the proposed Adaptive masking with several alternative decoder masking strategies on short videos (32 frames) using both RGB pixels and FSQ-MagViT tokens as reconstruction targets on the EK100 dataset. We establish a baseline using the default MAE configuration which involves random tubes [14] as encoder mask (with mask ratio 90%), and decoding all the masked tokens, denoted as decoder mask ratio None. Then we experiment with the low-budget setting, Table 1 : Decoder masking strategies on short-videos (32 frames) on EK100. We report finetuning top-1 action classification performance of MAE pre-trained models using pixel (RGB) or token (FSQ-MagViT) reconstruction targets. For decoder masking, we consistently use 15% as the token budget. We find that (1) compared to no decoder mask (None) with 100% budget, uniform masking scheme (Uniform [14] ), random masking scheme (Random), and decoder masking using Optical Flow (Flow) perform competitively with the lower token budget, and (2) we obtain best results with our proposed Adaptive decoder masking scheme (Adaptive). Note that fine-tuning performance is reported on 1 temporal crop. , where tokens to be decoded are selected using different saliency schemes described below.Random and Uniform. The Random importance scheme selects tokens randomly from the spatiotemporal volume. For the Uniform saliency scheme, we form decoder-visible tokens by picking frames uniformly using a step size of 7 to select 1/7 th of frames or roughly 15% token budget for decoding. We observe competitive performance to the baseline, consistent with VideoMAEv2 [14] .Flow and EVEREST. For the Flow importance scheme, we identify parts of the input video that contain motion. First, we obtain pixel-level Optical Flow F ∈ [-1, 1] 2×T ×H×W for the video using RAFT [32] , containing both x and y displacement fields. We then use the pixel-level Flow to obtain a probability distribution of motion in token space to generate masks. Specifically, following the same patchify operation used for RGB video, for each spatio-temporal patch in Flow, we take the mean of absolute value for all pixels in the patch across x and y coordinates, and normalize across all patches to obtain S ∈ [0, 1] N , where N is the total number of tokens. For the proposed Adaptive scheme and the Flow scheme, we choose ρ d = 90% and we add a small amount of random tokens, ρ r = 5% as we found it improves the results. For EVEREST we follow details of [17] .Although, Flow and EVEREST strategies provide more informed token prioritization based on explicit motion vectors or learned pixel distance, respectfully, they don't perform as well as the proposed Adaptive masking approach. This is in part due to inaccuracies in prioritization that may result from optical flow when, for example, background motion is involved. Overall, we make a few observations:(1) with only 15% decoder token budget, our proposed adaptive scheme bridges the gap with, and even improves on, vanilla VideoMAE which decodes all tokens with 100% token budget (None), and (2) in addition to random and uniform schemes, that are content agnostic, our proposed strategy outperforms over content-informed approaches (Flow and EVEREST), demonstrating its effectiveness, and lastly (3) the above trends hold over both pixel and token reconstruction objectives. As a side note, our findings corroborate with other related works [25, 33, 24] that reconstructing higher-level targets outperform RGB pixel reconstruction for MAE. The masking strategy comparisons upon scaling to 128 frames are presented in Table 4d and illustrate similar trends.Using the best performing adaptive decoder masking strategy and FSQ-MagViT reconstruction targets, we reinvest the memory efficiency gained using just 15% token budget towards extending the input number of frames to 128. Table 2 shows performance on long-videos (128 frames). We first highlight that default VideoMAE with no decoder masking encounters out of memory error due to large number of spatio-temporal tokens, demonstrating the need for decoder masking.Overall, we observe that: (1) 128 frame pre-training outperforms 32 frames pre-training when fine-tuned on 128 frames; (2) 128 frame pre-training outperforms the typical 32 frames multi-crop evaluation. These two findings establish the significant benefit unlocked by our proposed model owing to long-video MAE pre-training, often ignored by previous models due to short context lengths. In Tables 3a and 3b , we compare our proposed method, dubbed LVMAE, against the state-of-the-art on EPIC-Kitchens-100 and Diving48 respectively. For these comparisons, we first pre-train our model on unlabeled videos from Kinetics710 [38] data using 128 frames and adaptive masking strategy. This is followed by pre-training and fine-tuning on respective datasets at 128 frames similar to Sec.4.3.We additionally scale the encoder size to ViT-L. Further experiment details are in the Appendix. Note that existing SOTA methods use tailor-made architectures and/or pre-train their models using large-scale supervised pre-training data.In Table 3a , we show that our proposed model improves current SOTA on EPIC-Kitchens Top-1 Verb classification by +2.5 points using standard ViT architecture and just a single crop. On the Top-1 Noun classification, our model lags behind MTV-B [39] pre-trained on 60 million labeled video clips and a specialized multi-view architecture, TAdaFormer [40] pre-trained on supervised Kinetics710 and recently published Avion [41] that pre-trains on large-scale egocentric data namely Ego4D. We would like to point out that large-scale labeled pre-training helps nouns more than verbs.The EPIC-Kitchens noun categories such as hands, gloves, spoon, knife etc. routinely appear as annotations in large-scale pre-training datasets such as ImageNet21k [42] , Ego4D [43] etc. (e.g. 282/300 nouns from EPIC-Kitchens-100 also appear in ImageNet21k). As noted by Verbs-In-Action [44] , verbs are relatively scarce in existing datasets. This explains why existing SOTA that use large-scale datasets excel at noun classification. On the other hand, our approach doesn't use largescale pre-training datasets and learns long-range spatio-temporal dynamics to push SOTA on verb classification. Furthermore, in Table 3a , we find that if we add a supervised pre-training stage to our model using medium-scale dataset, we can bridge the gap on Noun classification while maintaining SOTA on Verb classification.In Table 3b , we show that our proposed model improves the absolute state-of-the-art on Diving48 dataset which contains complicated diving sequences by 3.9 points. It's worthwhile to note that the current SOTA method, MC-ViT [5] effectively uses 27M video-text pairs while we use ∼ 1M unlabeled videos and just 15K labeled videos.In Table 4a , we ablate our decoder masking strategy by pre-training models at 32 frames and report the performance on EPIC-Kitchens-100 dataset using a ViT-B backbone. Similar to [16] , we find that reconstructing a small amount of random tokens helps in improving performance. Since reconstructing random tokens adds stochasticity to the sampling process, we hypothesize it can help with overfitting. On the other hand, it is sample inefficient to only reconstruct random tokens and combining these two approaches strikes a balance. We limit this ablation study to a sampling ratio of 15%, respecting memory constraints for extending to 128 frame pre-training.In Table 4c , we gradually increase the number of frames and report the effect on EPIC-Kitchens-100 dataset. We find significant improvements in accuracy as we increase the number of frames from 16 to 32 to 64. However, as we move from 64 frames to 128 frames, the marginal improvement is small. This is expected as such videos form the tail-end of the distribution. Unlabeled K710 94.9In Table 4e , we compare our model's performance with the current SOTA model, Avion [41] , on videos of different lengths using the EPIC-Kitchens-100 Verbs benchmark. For this study, we use the Large version of our model. We observe sustained performance improvements over SOTA with longer durations signifying our model's capability on longer sequences.In Table 4b , we ablate the choice of targets for MAE pre-training. For this experiment, we first pre-train several models using 32 frames with no decoder masking. We find that when we shift from standard RGB targets to MAGVIT targets, the Noun-Verb Top-1 accuracy improves by 3.3%. Further, we notice a small drop from switching from MAGVIT targets to Adaptive FSQ-MagViT targets.In the last row, we show that when we use the adaptive decoder masking strategy with Adaptive FSQ-MagViT targets, we recover the performance. In effect, our proposed adaptive masking strategy allows us to retain the performance boosts from MAGVIT targets at very high masking ratios, and thereby scale these gains to 128 frames effectively and surpass state-of-the-art. Unless otherwise mentioned, we always use Adaptive FSQ-MagViT as targets for all of our experiments.In Table 4d , we compare our proposed adaptive decoder masking strategy with other strategies at 128 frames using FSQ-MagViT as targets. We find that our proposed adaptive masking is best as we scale the number of frames from 32 to 128 on both EPIC-Kitchens-100 and Diving48 datasets. 
limitations and broader impact: 
 In this work, we restrict ourselves to relatively small datasets and model sizes and leave the exploration about large-scale pre-training, joint training of image and video datasets, higher-capacity models, etc. to future work. Furthermore, while the 128 frames in this work are a big leap from prior works' focus on 16 to 32 frames, we anticipate more significant improvements to handle longer videos. Using efficient decoders (and encoders) or combining our long local-context with memory offers an alternate path to scaling MAEs, which is orthogonal to our approach to some degree.Improved long-video understanding can potentially revolutionize how users interact with video content since it can enable AI to reason across complex events and nuances. This potential will benefit accessibility, efficient content creation, recommendation, moderation, etc. Meanwhile, as with many machine learning models, our proposed method can be biased by the data. In addition, some applications (e.g., surveillance) might negatively impact society, and we urge users and researchers to deal with such use cases responsibly. 
conclusion: 
 In this paper we present a relatively simple but highly effective adaptive masking strategy for video MAE pre-training that allows us to pre-train on long videos (128 frames). Our approach is based on a novel MAGVIT-based tokenization strategy that also learns importance of tokens. Further, we are the first, to our knowledge, to show that long-video MAE pre-training is not only possible but leads to better encodings. Using our approach we are able to achieve state-of-the-art downstream performance, despite using a simple architecture and video only pre-training. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: Yes.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Section 5Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We are not releasing any data or model checkpoints.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The base codebase is cited in Section A.5.4.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We choose MAGVIT [25, 56] (which has a 3D CNN encoder) because it is a strong video tokenizer architecture that has been shown to work both for generation and understanding [56] in prior works.For the choice of quantizer, prior works show that Lookup Free Quantizer (LFQ) [56] and Finite Scalar Quantizer (FSQ) [36] outperform traditional Vector Quantization (VQ) [23] . To study the effect of quantizer on our long-context MAE task, we first train several MAGVIT tokenizers with LFQ and FSQ quantizers at different codebook sizes, and then train MAE at 64 frames using our proposed decoder masking strategy. We present the corresponding results in Table . 5. We find that there is not much difference on the final EPIC-Kitchens-100 accuracy with these choices. However, FSQ with 18 bit codebook size shows the highest PSNR, the highest EK-100 accuracy and the second best FVD. Based on this result, we choose FSQ as our quantizer.In this section, we share supplementary results on two additional benchmarks, Something-Something-V2 [27] and FineGym288 [57] .In Table 6 , we test our proposed method on SomethingSomething-V2 [27] dataset, which has an average clip length of 3.8s at 12fps i.e. 46 frames on average, by training a Base sized model using our proposed decoder masking strategy with FSQ-MagViT as targets. We gradually increase the number of frames from 16 to 96 for this experiment.The results match the trends we have seen on the other datasets in Table 4c and Table 2 , with longer context providing a significant boost in final accuracy. There are diminishing returns from using more than 64 frames of context, as only a small number of examples from this dataset have clip length this long. Note that our performance exceeds both VideoMAE V1 and V2 [14, 15] for the base sized model, despite pre-training for fewer epochs (1600 vs. 2400) and using only a single crop at evaluation time.FineGym288 [57] is a video classification benchmark that, similar to Diving48, tests the ability to categorize multi-second sports action sequences consisting of fine-grained motion, although in this case focusing on gymnastics. The original dataset has ∼29K training examples and ∼10K validation examples with video length ranging from 13 frames to 877 frames (average 47 frames). During dataset creation process, we add a 1s margin to the temporal action boundaries and increase the average video length to 107 frames, in order to exploit long-context.In Table 7 , we compare against SOTA when pre-training a 128 frame model with a ViT-L backbone and using additional unlabeled Kinetics-710 data (see section A.5.2). We outperform the current SOTA model, Temporal Cues Transformer [58] , despite not using any additional temporal cues as in this work.We ablate the effect of the decoder mask ratio on Diving48 top-1 accuracy while keeping the encoder mask ratio fixed at 90% in Table 8 . For this experiment we use our adaptive masking strategy, a 128 frame context for pre-training and fine-tuning, and use adaptive FSQ-MagViT tokens as pretraining targets. For each experiment, we sample adaptive tokens and random tokens in 2:1 ratio. 15% token budget in the decoder yields the highest accuracy, significantly outperforming even the much more expensive setting with 50% token budget that requires >4x more memory than the 15% setting. This aligns with our observations in Table 1 . Note that for this experiment, we used improved hyper-parameters across all models, resulting in better results than presented in Table 4d .Why does lower token budget outperform higher budget? Videos contain redundant information, and we hypothesize that when we reconstruct a lower number of high importance (or higher rank order) tokens during pre-training, the gradients can potentially be stronger than when we give equal importance to all tokens, and this would explain the stronger pre-trained encoder. However, we didn't notice this behavior consistently across datasets, namely on Something-Something-V2.Given our decoder efficiency improvements, one can also scale model size instead of scaling on the number of frames axis. To study this effect, we first fix the memory budget and token budget to a reference Base sized model trained on 128 frames with a 15% token budget. Then we vary the model size, and for each model size, we maximize the number of frames that can fit in memory budget. In Table 9 , we present top-1 accuracy, GFLOPs and the maximum frames on EPIC-Kitchens-100 dataset. We find that with larger model size and fixed memory budget, the accuracy improves despite lower frames but at the cost of significantly increased compute. We reiterate that all the three settings above are only made possible due to the memory savings from our proposed adaptive masking strategy. We leave further exploration in this direction to future work.We train our adaptive token selection module following the training recipe from MAGVIT [25] on Kinetics600 dataset training on 16 frame clips, (model card in Table 10 ). For inference, we produce 8 × 14 × 14 tokens along with the mask that determines the selected tokens.Once trained, we keep this module frozen and use it across all our experiments. Note, we consciously choose to decouple the learning of token selection and MAE itself, so that the gradients do not bias token selection towards selecting easily unmaskable tokens and vice-versa. To obtain the adaptive mask for long-videos, we simply slide a window of 16 frames with a stride of 16 through the tokenizer and token scorer module, and concatenate the resulting tokens and importance masks.For our MAE pre-training we follow VideoMAE architecture [14] and employ a vanilla ViT-B encoder with full space-time attention and 12 layers and a 4 layer decoder with full space-time attention. This architecture will also produce 8 × 14 × 14 tokens, matching the masks and latents produced by the adaptive tokenizer. Given a long video, we first compute the importance mask and select the k = (1 -ρ d ) × N tokens based on their importance from the video (plus ρ r × N random tokens). This step can be computed offline and only needs to be run once per dataset. Next, we use a tube masking strategy and encode 10% tokens from the video along with their positional embeddings.Table 7 : SOTA comparison on FineGym288. We report results on the FineGym288 benchmark compared to current state-of-the-art methods. Method Per-video Accuracy TSM [59] 83.1 TQN [60] 89.6 VT-CE [61] 90.1 TCT [58] 92.6 LVMAE 92.8 Then, we decode the chosen N d = k + ρ r × N tokens from these encoded tokens and learnable M ASK tokens along with respective positional embeddings. We make sure that no loss is applied on encoder-visible tokens. Unless otherwise mentioned, we initialize the MAE model from scratch and pre-train on respective datasets for 1600 epochs. Full pre-training hyper-parameters are presented in Table 11 .For the case where we pretrain using additional K710 unlabeled data (see Tables 3a and 3b ), we first pretrain an MAE model on K710 using our exact same recipe and hyperparameters as used elsewhere in this paper, except only for 800 epochs. We then pretrain a new MAE model on the downstream dataset (e.g. EPIC-Kitchens-100), initializing from the final checkpoint of this first model. We use identical hyperparameters when initializing from a pretrained checkpoint in this way as we do when training from scratch.A.5.3 Fine-tuning on long-videos (128 frames)Finally, we fine-tune the pre-trained models on respective datasets using standard recipes detailed in Table 11 and report the performance metrics. In all cases we evaluate the final checkpoint of finetuning and average over runs with three different random seeds when reporting metrics to minimize variance. Fine-tuning batch size and epochs are typically much smaller than pre-training and hence we can afford to fine-tune our models with lower masking ratios (drop-token ratios) to fit in memory. In particular, we use a masking ratio of 20% in the encoder for 128 frame fine-tuning and 0% for 32 and 64 frame fine-tuning.When fine-tuning the ViT-L backbone models on Diving48 and Epic-Kitchens-100, we make the following slight adjustments to hyperparameters:1. We use a single layer of class attention [62] as the aggregation method when generating prelogits as opposed to mean pooling. We found this slightly improved accuracy (+0.5 points on EPIC-Kitchens-100 Verbs and +1.2 points on Diving48).2. We use 25% encoder masking instead of 20% encoder masking to avoid going OOM with the larger model.3. For Diving48 we finetune for less steps (50 epochs instead of 200 epochs).We implement the code in Scenic [63] Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).We note that EVEREST[17] does non-differentiable top-k selection on the low-level features of the MAE. In our case, we are learning the CNN top-k module in parallel with FSQ-MagViT and need differentiable top-k to be able to learn it end-to-end. 


Decision: The reviewers praised the paper's novelty, writing style, and clear explanations of the proposed method and experimental results. They also acknowledged the importance of addressing the scalability of video masked modeling for longer sequences.

While some reviewers initially expressed concerns about the limited scope of the experiments, the authors have since provided additional results on an additional dataset (SSv2), number of frames, and mask ratios, demonstrating the robustness of their approach. The authors have also clarified the quality of the tokenizer used and its impact on performance. Furthermore, the authors have addressed the concern about testing on longer video sequences, providing new results that show the effectiveness of their approach on longer sequences.

Overall, the authors have successfully addressed the concerns raised by the reviewers, providing additional experiments and explanations that strengthen the paper's contributions. The paper has undergone significant improvements since the initial submission, and the authors have demonstrated a thorough understanding of the reviewers' concerns. 

After carefully reviewing the reviews and rebuttal, this meta-reviewer concluded that this paper is ready for publication and will make a valuable addition to the research community.

Reviews: review: paper_summary: paper_summary: This paper focuses on efficiently extending VideoMAE to much longer videos. It proposes an adaptive decoder masking strategy that utilizes a  MAGVIT tokenizer to localize the importance of each token, which becomes targets that reduce the memory/computation of decoders.
The motivation aims to scale the input video along the temporal dimension while maintaining pre-training efficiency. The key innovation is using a tokenizer to generate masks and targets, significantly reducing the training burden. The evaluation mainly focuses on long-video recognition tasks like (Epic-Kitchens-100 and Diving-48).
strength_weakness: strength_weakness: My main concern is the fixed budget setting (128/32 frames and 15% masking ratio on two datasets), which limits the potential to become a standard baseline for further research. This method can work well with models of various sizes and longer videos, so it is important to show that it also performs well in standard VideoMAE settings (short-video datasets). As the model grows larger, the computation of the decoder increases not only with the number of frames but also with the model size. Therefore, I am eager to see the proposed method’s results (efficiency/performance) on different datasets, number of frames, and mask ratios. In short, I am not convinced by the current results that the proposed masking strategy can be a future research baseline.

The quality of the tokenizer is another important aspect that needs to be included. Table 4 shows that using MAGVIT as a target can significantly improve performance, indicating there are insights behind this choice. Moreover, the novel token selection module may influence the tokenization results or potentially benefit reconstruction. However, I cannot find tokenizer-related results. Specifically, why does utilizing such a tokenizer significantly reduce the number of tokens? Why choose MAGVIT instead of other options? The 3D tokenizer compresses the temporal dimension several times; could this lead to inferior performance?
questions: questions: Please see above weakness.
limitations: limitations: Yes.
paper_summary: paper_summary: This video understanding paper extends the video mae idea to a longer 128 frames. They use the MAGVIT tokenizer to achieve this and test the approach on Diving-48 and epic kitchens. Both achieved an improved score despite using a pretty standard network and pre-training
strength_weakness: strength_weakness: The performance for EK-100 isn't that great.
The approach needs to be tested on longer sequence videos, maybe to make a more significant difference than the current two.
There is limited innovation on the actual network used
questions: questions: Why does the approach only make sota for the verbs?
What is the limit on the length of tokens
limitations: limitations: The idea of focusing on more extended encodings is interesting, but the dataset performance doesn't seem to make it worth it.
paper_summary: paper_summary: The paper proposes a novel MAGVIT-based adaptive tokenizer & masking module to extend VideoMAE to 128 frames. The tokenizer & masking module is individually trained and applied offline, making it possible for VideoMAE to reconstruct sparser (but important) and more semantically informative targets. The experimental results show that the approach allows for significant memory savings, enabling pre-training on longer video clips and leading to improved performance on downstream long video understanding tasks.
strength_weakness: strength_weakness: 1. The major weaknesses of this paper include the limited dataset sizes, small model sizes, and narrow range of task types, as mentioned in Section 5. I'm interested in the model's performance on action detection and temporal action detection tasks.
questions: questions: 1. In Table 4a, the 10%+5% masking even exceeds the case without decoder masking, and perhaps the authors could provide more explanation for this phenomenon.
limitations: limitations: The author discusses the limitations of the work, which I guess are largely due to resource limitations.
paper_summary: paper_summary: This paper studies the MAE pretraining of long videos. To address the challenges of hardware memory and compute limitations, this paper propose an effective strategy of decoder masking, subsampling tokens as reconstruction targets. This strategy leverage the MAGVIT-based tokenizer, prioritizes the most important tokens and uses quantized tokens as reconstruction objectives. The proposed decoder masking strategy outperforms previous masking strategies on EK-100 and Diving48. With 128 frames as input for pretraining and finetuning, the video encoder obtain better performance compared with the short-video (32 frames) pretraining.
strength_weakness: strength_weakness: 1. This paper only shows the results of 32 frames input and 128 frames input. To figure out the best setting of input frame number, it would be better to show experimental results when increasing the frame number from 16 to 128 gradually for pretraining and finetuning (while keeping the similar computational cost at inference). 

2. This paper does not include any experiments or theory analysis to explain why to convert MAGVIT to FSQ-MAGVIT for reconstruction targets and adaptive token selection.

3. Only 2 downstream tasks are used for the main experiments.
questions: questions: 1. Why does this work convert MAGVIT to FSQ-MAGVIT for reconstruction targets and adaptive token selection?

2. Results on other long-form video downstream tasks.
limitations: limitations: Yes.

Comments: title: Thanks for your response
Please let us know whether we address all the issues by the end of discussion period
Additional task types
Additional clarifications on tokenizer choice
Addressing the length of video sequences
Response to rebuttal
Please let us know whether we address all the issues
Please let us know whether we address all the issues
Please let us know whether we address all the issues
Please let us know whether we address all the issues
comment: We hope that our response clarifies the rationale behind tokenizer choice and the choice of task types. We also hope that the break-down of our performance improvements along the dimension of video durations further substantiates our claims. We would like to request the reviewer to reevaluate our work in light of our response.
We hope that we have addressed your concerns and would like to request you to reevaluate our work in light of our rebuttal.
Dear reviewer

Thank you for responding to our rebuttal and we are glad to have addressed most of your concerns. We will incorporate your several feedbacks in the final version! Below, we answer your follow-up questions:

`What is the MagViTv2 inference cost regarding memory and speed?`

We benchmarked MagViTv2 inference cost using a single TPUv4 device. We are able to use a maximum batch size of 32 with each batch element containing one clip of 16 frames. The peak memory usage is 6.4Gi and average throughput is 195 videos/sec. For longer video clips, as mentioned in lines 812-813 of the main paper, we simply slide a window of 16 frames over the entire video with a stride of 16 frames. So, the inference costs would scale linearly with video lengths while the memory requirement stays the same.


`Will it become a bottleneck if we further extend the number of frames?`

No, MagViTv2 will not become a bottleneck for longer videos because MAE training costs scale quadratically with video length while MagViTv2 costs scale linearly. In addition, MagViTv2 tokens can be computed and stored offline for the entire dataset just once as we do not update the tokenizer during MAE training. Note that we typically train MAEs for thousands of epochs, so the amortized cost of MagViTv2 computations is an order of magnitude less if we precompute the tokens.
Dear reviewer,

Thank you for the comments on our paper.

We have submitted a response to your comments and a common response. As the other reviewers have participated in the discussions, we would like to ask you to let us know whether you have additional questions.  

We hope that you can consider raising the score after we address all the issues.

Thank you
We provided our results on short-clip (Something-Something-V2) and long-clip classification tasks (EPIC-Kitchens-100, Diving48) and demonstrated SoTA performance. In response to reviewer 9WAZ, we also provided a breakdown of our results according to duration of the videos, and found that [our method outperforms SoTA especially on the longer duration clips](https://openreview.net/forum?id=bFrNPlWchg&noteId=DvCiSrKkm7) i.e. >16s in length. 

One key issue that we faced when selecting datasets for evaluation is the lack of long-form video benchmarks. Many of the tasks that operate on long videos only require short video-contexts [1, 2, 3] and can be solved by sliding a short-clip window over the video lengths. For example, a single frame is shown to be enough to answer QA tasks in ActivityNet [2], which is also a widely used benchmark for temporal action localization. Recently, new benchmarks have come up that are shown to require a longer temporal certificate [1, 4, 5], but unfortunately they require multimodal understanding, which is beyond the scope of this paper, and we leave them to future works.

[1] Mangalam, Karttikeya, Raiymbek Akshulakov, and Jitendra Malik. "Egoschema: A diagnostic benchmark for very long-form video language understanding." Advances in Neural Information Processing Systems 36 (2024). \
[2] Lei, Jie, Tamara L. Berg, and Mohit Bansal. "Revealing single frame bias for video-and-language learning." arXiv preprint arXiv:2206.03428 (2022). \
[3] Buch, Shyamal, et al. "Revisiting the" video" in video-language understanding." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022. \
[4] Wang, Weihan, et al. "LVBench: An Extreme Long Video Understanding Benchmark." arXiv preprint arXiv:2406.08035 (2024). \
[5] Rawal, Ruchit, et al. "Cinepile: A long video question answering dataset and benchmark." arXiv preprint arXiv:2405.08813 (2024).
We thank the reviewer for such timely feedback. We agree with the reviewer that our exposition on the tokenizer can be improved further. However, we would like to point out that the focus of our work has been to find a way to effectively scale MAEs to 128 frames and we saw our tokenizer explorations as a means to achieve this goal. In addition, since tokenizer is the first stage of our pipeline, it is expensive to undertake ablations on tokenizer, especially, when we only see very limited gains when we switch tokenizers (as shared in common rebuttal). So, we chose the best tokenizer from literature [1] and picked one of the two best quantizers [1, 2]. Nevertheless, we are glad to see the reviewer's interest in our work and appreciate the experiment suggestions. We hope our response below addresses your concerns:


`the authors have not provided experiments between different tokenizers under various ratios of adaptive masking`

We thank the reviewer for this suggestion. We would first like to clarify that our tokenizer indeed sees different masking ratios during training as we fix the token budget at batch-level (on a batch size of 256) instead of at example level, and hence it can work at varying mask levels in the MAE training stage. For this reason, we treated the token budget of the tokenizer as a hyper-parameter and chose it to be 35% so that it can support a range around that value. We agree with the reviewer that both the choice of the tokenizer and the token budget can be further optimized to boost the performance of our method. We didn’t pursue this route as our focus was on enabling an effective strategy for long-video pre-training, and leave the exploration on tokenizers for understanding tasks to future works.

`I am still concerned about whether using FSQ in long-video MAE methods is necessary compared to a standard MAGVIT`

We chose FSQ because it's simple and performs well on reconstruction (see below for detailed results). However, we stress that the choice of quantizer is not critical for the downstream performance of our method, and we hence also do not make any claims encouraging FSQ in our work. It is likely that the crux of our paper and all the key findings may very well be unchanged had we picked LFQ.

Below, we evaluate our pre-trained FSQ and LFQ tokenizers by varying the masking ratio to see how they respond. We expect the PSNR to increase (and FVD to decrease) as the token budget increases. We find that FSQ adheres to this expected behavior better than LFQ does. Especially at a lower token budget, performance of FSQ is better than LFQ. This criteria is well-suited for decoder masking where we use very high masking ratios.


|Token budget|LFQ-PSNR | LFQ-FVD|FSQ-PSNR| FSQ-FVD|
|-------------|-----|------|-----|-----| 
|15%|19.2|174.3|23.0|60.6|
|20%|21.1|56.1|24.2|29.0|
|25%|22.8|25.1|24.9|18.3|
|30%|24.3|13.8|25.4|13.4|
|35%|25.4|9.2|25.8|10.7|
|40%|25.7|7.9|26.1|9.6|
|45%|25.0|8.9|26.3|9.0|
|50%|24.0|11.7|26.4|9.0|


However, despite the reconstruction capacity, in our previous ablations we have shown that the MAE pre-training is not highly sensitive to the choice of quantizer in the Magvit. 
|Quantizer|EK-100|
|----|---|
|LFQ |46.2|
|FSQ |46.4|



[1] Yu, Lijun, et al. "Language Model Beats Diffusion--Tokenizer is Key to Visual Generation." arXiv preprint arXiv:2310.05737 (2023). \
[2] Mentzer, Fabian, et al. "Finite scalar quantization: Vq-vae made simple." arXiv preprint arXiv:2309.15505 (2023).
Dear reviewer,

Thanks for acknowledging our response, and we are encouraged that we have addressed your concerns on nouns/verbs. To demonstrate our model’s capability on longer video sequences, below, we provide a break-down of the performance improvement of our model over previous SoTA [1] on EPIC-Kitchens-Verbs based on video durations. 

|Model|0s-4s|4s-8s|8s-16s|16s-32s|>32s|
|-------|------|-------|--------|---------|------|
|AVION - Large (previous SoTA)|75.6|66.0|64.7|66.3|51.9|
|Ours - Large|77.8|67.0|66.2|72.2|57.7|
|Relative Improvement (%)|+2.9|+1.5|+2.3|+8.9|+11.2|

We observe sustained performance improvements on videos with longer durations signifying our model’s capability on longer sequences. When we consider the number of frames, compared to contemporary video encoders [2, 3, 4], we can process 8X their number of frames. As stated in previous comments, we leave the exciting opportunity of using our video encoder as a backbone of an e2e long-video understanding system [5, 6] or adding a memory module to increase sequence length [7] further to the future works.

### More evidence on Nouns vs Verbs and further improvements on EK-Nouns

On a side note, to augment our earlier point that EK-100-Noun can indeed be improved by large-scale pre-training, we post-trained our best model on labeled Kinetics710 dataset containing approximately 1M videos, and found that we can improve EK-100 Noun accuracy from 59.5 → 61.8% and retain our SoTA on EK-100 Verbs at 75.0%. Overall, our action classification accuracy is now 52.1% which places us at Rank 3, only lagging behind AVION [1] and M&M [8]. AVION uses 4M Ego4D clip-text pairs and M&M uses 60M clip-text pairs in their pretraining. Note that our Verb accuracy didn’t improve with large-scale pre-training datasets and we achieve SoTA on Verbs despite using an order of magnitude less data than these two methods. This matches our intuition that EK-100 Nouns and EK-100 Verbs require different expertise. We will include add this to our discussion in the final submission and we thank the reviewer for pointing the difference out.

[1] Zhao, Yue, and Philipp Krähenbühl. "Training a large video model on a single machine in a day." arXiv preprint arXiv:2309.16669 (2023).
[2] Wang, Limin, et al. "Videomae v2: Scaling video masked autoencoders with dual masking." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
[3] Wang, Yi, et al. "Internvideo2: Scaling video foundation models for multimodal video understanding." arXiv preprint arXiv:2403.15377 (2024).
[4] Ryali, Chaitanya, et al. "Hiera: A hierarchical vision transformer without the bells-and-whistles." International Conference on Machine Learning. PMLR, 2023.
[5] Chen, Guo, et al. "Video mamba suite: State space model as a versatile alternative for video understanding." arXiv preprint arXiv:2403.09626 (2024).
[6] Sun, Yuchong, et al. "Long-form video-language pre-training with multimodal temporal contrastive learning." Advances in neural information processing systems 35 (2022): 38032-38045.
[7] Wu, Chao-Yuan, et al. "Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[8] Xiong, Xuehan, et al. "M&m mix: A multimodal multiview transformer ensemble." arXiv preprint arXiv:2206.09852 (2022).
Thanks for the authors' effort in the rebuttal. The response has addressed my first question (in the weaknesses). However, the authors have not provided experiments between different tokenizers under various ratios of adaptive masking, and the ablation studies include only a limited range of downstream tasks. I am still concerned about whether using FSQ in long-video MAE methods is necessary compared to a standard MAGVIT. Additionally, the experiments do not include various long-form video tasks, such as video action segmentation, temporal action localization, and spatiotemporal action detection. In summary, while the research topic is meaningful, I concerns that the experiments in this paper could not fully demonstrate the effectiveness and scalability of this method. Therefore, I would lower my rating slightly.
Thanks for the authors' rebuttal. It addressed most of my concerns. The scaling of model size seems promising. And additional results on Something-something are helpful. I have one concern: What is the MagViTv2 inference cost regarding memory and speed? Will it become a bottleneck if we further extend the number of frames?
ok thanks for this detailled response, especially about the nouns/verb issue, I still have a concern about the length of the video sequences that are possible to be used though
Dear reviewer,

Thank you for the comments on our paper. 

We have submitted the response to your comments and a common response. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues. 

Thank you
Dear reviewer,

Thank you for the comments on our paper. 

We have submitted the response to your comments and a common response. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues. 

Thank you
Dear reviewer,

Thank you for the comments on our paper. 

We have submitted the response to your comments and a common response. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues. 

Thank you
Dear reviewer,

Thank you for the comments on our paper. 

We have submitted the response to your comments and a common response. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues. 

Thank you