Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a video self-supervised pretraining pipeline called VITO. It made several modifications over existing contrastive learning frameworks including larger crop size, improved temporal sampling scheme, and multi-scale attention feature pooling for the projector. The authors also investigated the data domain mismatch and propose a video curation procedure to further improve the model pretraining performance. Experimental results on semantic segmentation and object detection show that the proposed framework can achieve better results among methods pretrained on video datasets, and be competitive with the models pretrained on ImageNet.
strength_and_weaknesses: Pros:

This paper is well-organized and easy to follow. The technical contribution and the way to do data curation make sense to me. The ablation is comprehensive and the final results compared with other video-based model looks good, which shows the effectiveness of the proposed framework.

Cons:

The motivation of this paper is not very clear, as the results using video pretraining do not show superior performance compared with image pretraining based on the benchmark the authors chose. If the authors believe video is beneficial maybe choose another benchmark (like video segmentation or tracking) or at least combine ImageNet with the proposed VideoNet data to showcase better performance compared with the ImageNet baseline.
For other misc comments see the section below.

clarity,_quality,_novelty_and_reproducibility: 1. it's not clear what `we sample from a uniform distribution over the entire video segment of length T = 2.56s` means, and I don't quite get how the newly proposed temporal sampling works actually. Would be better if the authors could consider rephrasing it for a better understanding.
It's also not clear to me why it is good that the mode of the distribution is at 0 (according to Figure A.1), wouldn't it be a better choice to make the distribution of time delays uniform?

2. Using an ImageNet-pretrained model to do data curation is less preferable from my perspective. Although I agree that it might be necessary to do this in order to compete with the already-curated ImageNet data, this inevitably obscures the boundary between semi-supervised and self-supervised learning.

3. [1] should also be discussed in related work as it is close to both topics the authors covered.

[1] Xiong, Yuwen, et al. "Self-supervised representation learning from flow equivariance." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
summary_of_the_review: Overall I believe this paper makes novel technical contributions (using learnable attention gates to align features from different frames, pointing out that the video data quality actually matters a lot for existing models on several benchmarks, etc.). However, the evaluation part is my major concern, would be great if the authors could better motivate why video is important (although this might be shown in other works) and show either video+image could perform better than image only, or video pretraining learn better representations on some tasks compared with image-based and other video-based baselines.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors adapt existing contrastive learning approaches for self-supervised image representation learning to learning from video frames instead of static images. In particular, they first adapt the data augmentation steps, which are key to contrastive learning, to the video domain. Two main adaptations are proposed: first, the crop size is increased to account for the fact that, unlike ImageNet, in-the-wild videos are not centered on a single object, hence very aggressive cropping can make the task unnecessary hard. Second, they utilize a multi-scale attention pooling module to account for the fact that the frame content can shift significantly over time. Finally, they collect a new dataset which is automatically curated (using an ImageNet classifier) to match the ImageNet category distribution. They then experimentally demonstrate that, by combining these model and data improvements, self-supervised pre-training on video frames can come close to pre-training on ImageNet in terms of downstream performance on the tasks of semantic segmentation and object detection. 
strength_and_weaknesses: Strengths:

The paper is well written and is easy to follow.

The proposed approach is sound and its effectiveness is convincingly confirmed by an ablation analysis.

The final model trained on curated video frames archives similar performance to models trained on ImageNet in terms of downstream semantic segmentation and object detection accuracy.


Weaknesses:

This is a purely empirical study, there is no technical contribution (all that techniques that are used have been introduced in the past).

Related work overview is incomplete. In particular, the authors ignore the whole line of work on space-time walks for self-supervised image representation learning from videos  which is highly relevant (e.g. Jabri et al., NIPS'21 but also prior works and follow ups).

In the experiments the authors do not control for the amount of training data. In particular, they have collected 1,180,042 videos each of which is at least 10 seconds long. How many frames is that? There is indeed a lot of redundancy between frames from the same video, but the redundancy is by far not 100%, especially in relatively long videos. For a fair comparison, the authors need to report results when only one pair of frames is sampled from each vides in a pre-processing step, so that the overall amount of data is comparable to ImageNet.

Most importantly, the value of using videos as a source of training data for image-level contrastive learning is not convincingly demonstrated. The authors have to go to great lengths to simply match the ImageNet performance. The downsides of this approach are clear (e.g. much higher storage requirements) but the benefits are not obvious from the paper.
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is well written and easy to follow.

Originality: the paper is novel in an experimental sense (the first to match ImageNet pre-training using video frames), but there is no technical novelty. 

Reproducibility: the authors are planning to release the code, but not the data (which is collected from the internet). Hence, reproducing their results will be very challenging, but, strictly speaking, not impossible.
summary_of_the_review: This is a (for the most part) solid empirical study on adapting image-level contrastive learning methods to video frames. That said, a crucial factor is missing from the ablation analysis (control for the amount of training data) and the overall value of using video frames in this framework is not well justified. Moreover, related work overview has significant omissions. I hope the authors can address these concerns in the rebuttal.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method to learn image-based representations from video. The paper makes two contributions: (1) on the network side, the authors introduce simple modifications to the way augmentations are applied (e.g. change the scale and uniform temporal sampling ) and a simple attention head for attention pooling to replace average pooling before the features are passed to the MLP. (2) on the data side, a new video dataset is proposed using a data curation process which tries to address the domain gap between imagenet and other video datasets. They evaluate on semantic segmentation and object detection after fine-tuning.
strength_and_weaknesses: On the positive side:
+ The paper reports some results that might be useful for the community. 
On the negative side:
- The methodological contributions are quite thin, mainly comprising minimal changes to existing strategies for augmentation and attention pooling (which is also very standard).
- The method is shown to work on the proposed curated dataset. Although the point of using a video dataset close to imagenet for a fair comparison is valid, in my opinion it also weakens the paper's contribution in a sense that learning from uncurated video data is a much more challenging and impactful goal. 
- Experiments: from the experiments (table 1) it is not clear how VITO performs when trained on other video datasets (like Kinetics or Youtube 8M). Similarly, it is not obvious how other previously proposed methods would perform when trained on the newly proposed curated dataset. I suppose a strong baseline would be also to train DINO on the new dataset.
clarity,_quality,_novelty_and_reproducibility: The papers reads very well. I have my concerns about novelty and significance of experimental validation as explained above.
summary_of_the_review: The paper reports some results that might be useful for the community, but I believe more work is required to increase the impact of the proposed method.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper describes a series of hyper-parameter changes (i.e. scale of the crop, frame sampling) and data filtering for learning representation from videos. The changes are implemented on top MoCLR. On the dataset and tasks tested the paper results show that video pretraining can match imagenet pretraining. However, the technical novelty is limited and the experimental results insufficient.
strength_and_weaknesses: Strengths:

- Overall, the paper is well written
- The results confirm that with careful considerations (by filtering the data and tuning the augmentations) video based pretraining can match in certain conditions imagenet pretraining.

Weaknesses and questions:

Low technical novelty:

- the method itself is a direct adaptation/implementation of MoCLR.

- The study of adapting the augmentation to videos, while reasonable, is straight-forward and its more of a hyper-parameter tuning than an actual contribution (e.g. setting the augm scale and frame sampling).

- The data filtering pipeline addresses the distribution mismatch between ImageNet and the video dataset. It's not surprising that matching the data distribution will improve the performance and such pipelines, are common, albeit perhaps not for this very specific use case.

Insufficient evaluation, ablations and comparisons:

- As multiple frames are present in a video, counting all the frames, the filtered dataset is still 1 order of magnitude higher than imagenet. What is the performance when the same number o frames as that of the imagenet images are used?

- Expanding on the prior question, what is the performance of a model trained with a single frame (most representative according to the imagenet classifier used for filtering)? This should hopefully emphasize how much the additional dynamic present in video helps.

- No results and ablation with other backbones: how does the performance vary when the backbone is larger (e.g: ResNet-50-2x, ViT-B/L) and respectively smaller?

- Current state-of-the-art for video recognition is achieved by initializing from pre-training on static images. How does the performance vary?

- Insufficient evaluations: only object detection and segmentation considered. What is the performance on classification for example? What is the few-shot performance of such models?

- How does this compare with imagenet-22k pretraining? How would the filtering perform based on these categories?

clarity,_quality,_novelty_and_reproducibility: The paper is well written generally and the hyper parameters alongside the implementation details are listed in the appendix.
summary_of_the_review: While the result itself is interesting, the evaluation is very limited, with insufficient ablation and comparisons. Moreover, on the technical side, the novelty is very low as the approach is a direct implementation of an existing technique with appropriate hyper parameter tuning for video data.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was reviewed by four experts in the field and received a mixed score. The main concerns are the limited novelty, unconvincing experiments, and lack of clarity. The authors did a good job of rebuttal and addressed many of the concerns. However, the reviewers (including all positive ones) still feel that more work is needed to get it to the best version. AC also agrees that this work can be much stronger with additional experiments. While this paper clearly has merit, the decision is not to recommend acceptance. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.


justification_for_why_not_higher_score: 1. Limited technical contribution 
2. Unconvincing experiments
justification_for_why_not_lower_score: NA
Decision: Reject