Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a self-supervised video representation learning method (VideoMAE) based on transformer design. The idea of this approach is inspired by the ImageMAE [22] which is based on the masking and reconstruction strategy. In VideoMAE, applying a high masking ratio and tube masking strategy lead to a challenging video reconstruction task which is the key to learn more representative features. Empirical results demonstrate the effectiveness of approach on 4 different video datasets in different settings. 

strengths_and_weaknesses: Strength:
- The paper is well-written and easy to follow.
- The motivation and the method are clearly elaborated.
- The experiments support most of the claims in the paper.
- The method outperforms existing approaches on challenging video benchmarks.
		

Weaknesses:
1. In my opinion, the claim of the first contribution (line 64) needs to be tone down as we have multiple works on the video SSL based on the idea of masking and predicting such as [53,50], in which the results are almost on par with the proposed approach. While there are some differences between the proposed approach and the existing ones which has been also mentioned by the authors in the submission, the claim of "the first masked video autoencoder that performs well for SSVP on small-scale video dataset" is not entirely precise.
2. While the proposed method presents an approach that performs on-par with SOTA, the novelty of videoMAE remains limited. In fact, (a) the idea of masking and reconstruction for video SSL has been explored before [53,50]. (b) the masking strategy (tube masking) has been already explored in [53] (c ) the idea of passing non-masked tokens to the transformer encoder which results in an efficient encoding has been explored in ImageMAE[22] paper.
3. In Table 5 of the main paper and Table 13 of supp, MaskFeat[53] results on SSV2 shown "supervised" labels, while based on [53], the pre-training is done on K400 and K600 unlabaled data and then they have a fine-tuning step on SSV2. It would be good if the authors can clarify this. To me, [53] follows the standard SSL paradigm for these downstream tasks.
4. While the feature transferability is investigated in Table 4, as far as I understood, the main set of experiments of comparison to the SOTA in Table 5 for SSV2 dataset is based on the pre-training step on SSV2. However, in the most of the previous works, the common practice is to consider a single pre-trained model which has been trained on a large-scale dataset such as K400/K600 and then evaluate on downstream tasks such as SSV2. I'm curious to see the result of VideoMAE on SSV2 when pre-trained on K400 with the same design and training setting as in Table 5 which makes the comparison of feature transferability comparable to SOTA. 

questions: 1. Regarding weakness 1 and 2, I would like authors to clearly state the contribution of this work to video SSL and the important differences to previous approaches.
2.  Regarding weakness 3, I would like authors to clarify the reason behind stating that method [53] uses extra labels during pre-training.
3. Regarding weakness 4, I would like to see the results of VideoMAE on SSV2 when pre-trained on K400 with the model architecture and training setting of the final best model. This allows us to have a fair comparison to previous methods in the literature that use only K400 for pre-training.
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the Masked AutoEncoder in the context of video self-supervised learning. It adapts the MAE technique to the video domain and propose tube masking along with a high masking ratio for videos. The experiments are conducted on a set of standard video action recognition tasks, showing the effectiveness of the approach. In addition, with this model, the authors also find that VideoMAE is data efficient — when sufficiently trained, the model achieves strong performance with a fraction of the training data. The main concerns of this paper are novelty, lack of comparison to a similar prior work and insufficient ablation studies.  Overall, if these concerns are properly address, I am happy to see this work accepted.
strengths_and_weaknesses: Strengths:

- The proposed model is simple yet quite effective. As the core task is video classification, it may benefit a lot of down-stream video tasks as a “foundation” model.
- This work shows that VideoMAE is very data efficient, this seems to be not studied in the image MAE paper. Specifically, the authors show that, when sufficiently trained, the model achieves strong performance with a fraction of the training data. Similarily, they also show the model performs much better than alternative approaches on small-scale dataset such as HMDB51.

Weaknesses:

- The proposed Tube Masking has been studied in VIMPAC [1] with a name “block masking”, this undermines the novelty of the approach. I suggest the authors cite/discuss and compare with this work.
- The novelty is limited in the sense that the work seems to simply combine MAE modeling techniques with the tube masking in VIMPAC with additional minor adjustments, e.g., increase masking ratio of tube masking. But please note, I only put novelty as a minor concern, as I do value the engineering effort of this paper and its potential benefits to the video understanding community.
- The ablation study is conducted on a single dataset SSv2. However, as shown in TimeSformer [2], the conclusion may vary a lot when looking at a different dataset, e.g., Kinetics-400 (K400). As SSv2 requires more temporal modeling while K400 videos are mostly stationary. I would imagine many conclusions will change. I strongly suggest the authors add additional ablation results on K400. I also commented in the question section about a specific ablation study where I believe the dataset property may affect the conclusion a lot.

[1] Tan, H., Lei, J., Wolf, T. and Bansal, M., 2021. Vimpac: Video pre-training via masked token prediction and contrastive learning. *arXiv preprint arXiv:2106.11250*

[2]Bertasius, G., Wang, H. and Torresani, L., 2021, July. Is space-time attention all you need for video understanding?. In *ICML.*
questions: L255-258, and Table 1c, the lower SSv2 performance of 16x1 compared to 16x2 I believe is mainly due to the shorter temporal support of 16x1 model (~1.33secs vs. 2.67secs for 12FPS SSv2 videos). It is quite well-known SSv2 dataset requires temporal modeling and shorter temporal support will hurt the model results (see [2] TimeSformer). In my view, this has less to do with the reconstruction target during pre-training. A better experiment will be using 16x1, and 16x2 to pre-train the model, and then fine-tune with the same 16x2 config, this removes the difference caused by temporal support. Though there is a slight input mismatch between pre-train and fine-tune for 16x1 model. I also encourage the authors to find alternatives setups to compare them fairly.
limitations: I don't see potential negative societal impact being discussed in this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a self-supervised learning method to learn representations of videos. The method is based on being able to decode masked elements from an encoded video, similar to the masked autoencoder approaches to image representation learning. The authors perform extensive experiments on different video classification datasets and ablation studies over the different design choices. As a result, the model obtains state-of-the-art performance and is able to perform well on smaller datasets without needing additional data, which is not common for self-supervised methods. 
strengths_and_weaknesses: **Strengths:**
- Comprehensive ablation studies that help understand which design choices work best
- Very strong results on multiple datasets, especially on smaller scale datasets without using additional data
- Simple idea that should be reproducible with the information in the paper and the submitted code

**Weaknesses:**
- The work is incremental and mostly consists in extending Masked AutoEncoders to videos. The small design choices required to adapt ViT methods for images to videos (such as temporal striding, temporal size of the input tokens, etc.) are well understood and the idea that the same spatial regions are correlated over time are well known in the video literature.
- The analysis of the model result is shallow and limited to showing the performance of the model with a certain configuration, without analyzing its mistakes.

**Detailed comment:**

This paper extends the idea of Masked AutoEncoders to video and runs multiple benchmarks to understand the effect of different design choices. While the paper is relatively incremental, there is a significant engineering effort and the authors are able to obtain strong results on multiple video classification benchmarks. One important insight is that this model is able to perform well on small datasets, contrary to other self-supervised learning methods. However, the analysis of the strengths and weaknesses of the model is quite limited.

Overall it is a good submission with a simple idea and impactful experimental results, therefore I argue for its acceptance.
questions: - How do the representations learned on one dataset transfer to other equally large scale datasets? Is there a best way to apply the model beyond training from scratch on a particular dataset? If so, how can we determine it?
- What kind of mistakes does the model do? Is there a pattern there that could help us understand better how the model works?
limitations: The discussion on limitations and potential negative societal impacts of the submission is quite limited. However, there are no major ethical considerations with video representation learning in general. The discussion could be extended by mentioning potential negative applications when applying the method to sensitive data.
review_title: null

Meta-review: metareview: This paper studies application of masked autoencoders to video data. It is a very empirical paper with lots of ablations and experiments. All three reviewers lean toward the acceptance of the paper. Reviewer C915 has a slight concern regarding the novelty of the paper over concurrent works including [50,53]. The reviewers believe that the ablation study is exhaustive and the paper has a good reproducibility. The authors are encouraged to add new experiments with kinetics pretraining in the final version.
confidence: Certain
Decision: Accept