Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a few new techniques: 1) A new video modeling architecture that uses a pre-trained VQ-VAE to tokenize frames, followed by a transformer encoder that aggregates the features and produces the final action class label. 2) Pre-training such an architecture using self-supervision by a) Masked prediction: Authors mask out blocks of tokens and predict them using the context (akin to BERT) and b) contrastive learning: Authors use the representation for two clips from same video as a positive match and otherwise negative match. The final model is trained with a linear combination of the masked prediction and contrastive losses, and finally finetuned for downstream tasks. The model is pretrained on HowTo100M dataset, and finetuned on multiple downstream datasets, where it obtains gains on more temporal datasets like SS-v2.
main_review: ## Strengths

1. The proposed model is a fresh approach to video modeling and performs well for temporal reasoning tasks like SSv2. The proposed pre-training methods leverage ideas from BERT and shows it can be effective for computer vision tasks too (similar to BEIT). 

2. The ablations are interesting, eg table 1 which shows pre-training is imperative for a model like this and lends significant gains.

## Weaknesses

1. Missing/incorrect details
- The paper in Table 1 compares their method to prior work and notes the pre-training dataset used. Authors report their method only uses "HT" for pretraining. However, the tokenizer is using a pre-trained DALL-E model, the training data for which should also be mentioned here? Given this is a paper about learning from unlabeled data, it is important authors are very explicit about *all* the data being used.

2. Continuing from 1), it is unfortunate that no results in Table 1 are comparable to each other; authors use a different pre-training dataset (HT+DALL-E), model architecture (DALL-E + Transformer) and modalities (video only) which makes it impossible to perform any apples-to-apples comparisons. The standard setting in Feichtenhofer et al (2020) seems to be to perform all comparisons with unlabeled Kinetics dataset as pre-training. It would have been ideal if authors kept at least some of the axes of variation fixed. In absence of that, it is only fair to compare the proposed method to state of the art. In that regard it struggles on spatially heavy datasets, and on temporal datasets it performs better although is comparable to more recent SOTA (eg MVIT -- 68.7 without large scale pretraining or HowTo100M pretraining, though it does use K600 labels). 

3. The architecture seems quite constrained as it relies on tokenization which looses spatial information. As authors themselves point out, this is likely the reason it struggles on spatial datasets. However, this also means this limitation would limit the impact of the model as it might be very useful for spatial tasks (video detection/segmentation etc). Hence, even though it might be good at learning representations, it is limited to classification. Using a BEIT style framework (as authors also recommend for future work) would make this paper much stronger in that regard.

4. Missing related work: Authors should cite MVIT and other recent transformer based video models. Also while it is not technically published work, I would encourage the authors to cite and compare with video swin transformer.
summary_of_the_review: The proposed model is interesting and in-line with recent work like BEIT etc, and obtains some decent results. However I find it hard to draw meaningful conclusions from the paper due to lack of proper apples-to-apples comparisons. Moreover, given the architecture is likely to have limited impact in the field (as I discuss in my weaknesses), I am borderline on this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new pre-training method named VIMPAC for video understanding, which is a combination of specially-designed masked token prediction objective and contrastive learning objective. VIMPAC is well-motivated and shows strong empirical results on several video understanding tasks.
main_review: ## **Strengths**
* VIMPAC achieves strong empirical results on temporally-heavy datasets.
* Although the mask token prediction task has been proven to be an effective pretraining objective for image tasks, the block masking method designed by authors is well motivated.
* Tokenizing all videos without spatial augmentation is a good way to reduce storage and io cost.

## **Weaknesses**
* This paper is utilizing a transformer as the main architecture, with a pretrained VQ-VAE as the tokenizer. I think it's great to have the baselines with the same arch for fair comparison. Bert-large is a relatively large model comparing to previous CNN models, and large models tend to work better in pretraining with big datasets.
* The contrastive learning objective part doesn't make a lot of sense to me. It seems adding contrastive learning objective only provides marginal improvement, and might even hurt the performance on temporally-heavy datasets. Ablation studies on contrastive learning also lead to some counter-intuitive conclusions so I would expect more analysis and insights from this part.
* I'm also interested if VIMPAC can work for tasks beyond action classification (might be hard due to the architecture design).
summary_of_the_review: With strengths and weaknesses I listed in previous section, I think this paper makes some reasonable contributions but is marginally below the iclr acceptance threshold.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new video pretraining method by combining masked token prediction and contrastive learning. Off-she-shelf VQ-VAE is used in this paper for discrete video tokens generation. In order to make the masked token prediction more effective, the authors proposed a block mask strategy where spatial-temporal neighboring video tokens are masked together. Experimental results are shown to validate its effectiveness.  
main_review: Strength:
+ the paper is well written, main ideas are clear enough, it is also easy to follow for readers.
+ the proposed block masking scheme makes sense in terms of avoiding information leakage from nearby tokens and it is also validated to be quite better than i.i.d.
+ extensive experiments are carried out

Weakness:
- novelty: 
（1）Combining masked token prediction (MP) and contrastive learning (CL) is quite straightforward. Both MP and CL are studied previously, simpling combining these techniques is kind of incremental. Yes, the proposed block masking strategy is a plus, but it will have problem by directly combining it to CL. In my opinion, global information loss will be caused by block masking, it is actually harmful to the CL task. If better combination mechanism of the two tasks can be invented, it will be a good work.
  (2) Discrete video token generation is also off-the-shelf technique. VQ-VAE is not new.

- performance and evaluation: 
  (1) the performances on "spatially-heavy" datasets, such as UCF101, K400 and HMDB51, are far from the state-of-the-arts, rather than comparable. Such results show that the proposed pretraining method is only good at "local" modeling, its global representation is not strong enough. This may not only be caused by the discretization of VQ-VAE, but also has something to do with the harmful blocking masking which can harm the CL.
 (2)  Now that VQ-VAE can be harmful to the spatial information, why it should still be leveraged. There is also no evaluation on this part. What if VQ-VAE is replaced by other feature encoder and we regress the masked contiguous video feature points? Why not using video frame patches as discrete input tokens? 
summary_of_the_review: Given the aforementioned weakness of this paper, I think the novelty of this paper is currently limited, and its performance is not good.
review_title: null

Meta-review: None of the reviewers recommended this paper. There were concerns that it is hard to draw meaningful conclusions from the experimental work due to the comparisons provided.  While the design of the block masking + contrastive learning proposed in this paper was rated as potentially being quite important, there remained some concern that subsequent tokenization steps could be problematic for "spatial heavy" datasets.

The AC recommends rejection.
Decision: Reject