Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper extends the recently proposed MAE pipeline for images to the video domain. The method shows competitive performance on video datasets despite having minimum domain knowledge.
strengths_and_weaknesses: Strengths

- Simple pipeline with competitive performance
- Great ablation study
- interesting possibility for computational speed-up (as the encoder is only applied on the sparse set of visible patches)
- good performance on the uncurated dataset
- good comparison with other video pretraining methods

Weaknesses
- I don’t see any

Although the novelty seems limited to me as it is a straightforward generalization of MAE method to the video domain, in other aspects the paper seems seamless to me, the work has practical value to the community, and the experiments section is great. 
questions: 1) It would be very interesting to see qualitative results for future predictions (when every token is masked in several last frames).
2) Did you try to use masking while finetuning for a supervised task? (to obtain speed-up in computation on this stage as well). Maybe one can anneal mask ration in finetuning stage to obtain the best metrics with minimum computational resources.
3) Is it possible to do Gibbs sampling with your pipeline? If yes -- did you try it and what is the quality of the samples? 
limitations: Yes
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to extend MAE for videos. Extensive ablations show that MAE can be successfully extended to videos and the optimal masking ratio is higher than images. It achieves competitive results on popular video classification and detection datasets based on both large uncurated and well-aligned upstream datasets.
strengths_and_weaknesses: 1. Simple and effective idea: simple random masking with pixel-level reconstruction works well for videos.

2. Clean design: Standard ViTs with a lightweight decoder is enough to achieve competitive results on video classification and detection.

3. Comprehensive ablation studies on upstream/downstream data, decoder design choices, masking ratio/strategy, and  reconstruction targets. 
questions: 1. Line 162: "it takes K temporal clips (by default K=7 on Kinetics)", any intuition why choose 7-view here?

2. Line 130-131:  "In practice, we find it sufficient to predict a single time slice of the patch (16x16)", which time index?

3. In Table 4, VideoMAE pretrained with IG-uncurated doesn't show big improvement over K400 when both are fine-tuned on in-domain K400. This raises a natural question in my mind: in MAE, does the accuracy continue to significantly improve with a bigger scale of data?

4. A similar question regarding resolution is raised in Table 7,8,9: If VideoMAE with a larger resolution continues to show further gains?

5. Random shuffle, slicing and reindexing causes randomly accessing the memory, which recomputes the cache. In practice, depending on the actual hardware, it may slow down the process (i.e. 90% cannot lead to 10x speed-up). Can authors share any knowledge about it?
limitations: Yes. The authors have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies a simple extension of image MAE to video domain. The experiments are conducted on a set of standard video datasets, i.e., Kinetics, AVA, SSv2, etc., showing the proposed approach achieve appealing results. This paper also presents some interesting findings: (a) due to high information redundancy, video MAE requires higher mask ratio comparing to its image counterpart; (b) random masking works surprisingly well; (c) architectural findings such as video MAE requires a deeper decoder and a larger decoder hidden size. On the one hand, the ablation experiments are comprehensive which covers many aspects of the model design. On the other hand, these experiments are only conducted on a single dataset, K400, which is not convincing.  As the paper is mainly an analysis paper, instead of proposing a novel approach, the ablation studies are insufficient.
strengths_and_weaknesses: Strengths

- Simple, yet effective method for video self-supervised learning.
- Comprehensive experimental comparison shows that the proposed approach has strong performance compare to prior work.
- The paper is well-organized and is easy to follow. Most of the claims are supported with solid experimental results.

Weaknesses:

- The ablation experiments in Table 2 are only performed on a single dataset (K400), which may not provide a precise or comprehensive evidence of how these ablated items may affect the model performance. In TimeSformer [4] Table 1, it is shown that the conclusions aren’t exactly the same when looking at results from K400 and SSv2, as K400 is more static while SSv2 requires more temporal modeling. I strongly encourage the authors to also conduct ablation experiments on SSv2 to have more persuasive conclusions.
- Overall the novelty is low, as it is almost a direct extension of image MAE. While there are few interesting conclusions/findings, e.g., (a) video modeling requires a higher mask ratio. (b) random masking works better than structual masking, and (c) decoder in video MAE needs to be more powerful, etc. But these are somewhat expected. I do value the experimental results that validate these findings, but the overall contribution is limited.
questions: - L192-194, Fig 6 (right). How do you vary the #encoded tokens when the mask ratio is fixed? By training longer?
- L129 mentions it is sufficient to predict a single time slice (16x16) instead of the full space-time patch ($t$x16x16). Is there any experimental results that compares the two strategies?
limitations: Yes.
review_title: null

Meta-review: metareview: This paper presents an interesting simple representation learning approach by extending masked autoencoders to videos. The reviewers have unanimously recognized the simplicity of approach, clarity of writing, and extensivity of experiments. Although there are some minor concerns about the novelty of the proposed method, the findings in this paper are of interest to the community. Given these, we are happy to recommend acceptance for this submission. 
confidence: Certain
Decision: Accept