Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigates whether using multiple frames for training is necessary for video-language downstream tasks. The authors propose a single-frame framework for video-language understanding. Results indicate that with large-scale pre-training and a proper frame ensemble strategy at inference time, a single-frame trained model that does not consider temporal information can achieve better performance than existing methods that use multiple frames for training. This result reveals strong static appearance bais in current video-langauge datasets. To allow for a more comprehensive evaluation of video-and-language models, the authors propose two new retrieval tasks based on existing fine-grained action recognition datasets that encourage temporal modeling. 
strength_and_weaknesses: Strengths:
1. The paper is clearly written and easy to follow.
2. It reveals the single-frame bias for current video-language datasets.
3. The proposed approach is simple and effective, and the authors prove that with the simple, single-frame model, the video-language models can already achieve good performance.

Weaknesses:
1. The novelty is limited. The single-frame model applies the basic image-language architectural design with self-attention and cross-attention.
2. The single-frame bias for current video datasets is not new. Researchers have realized that most video datasets (except for SSv2) do not require much temporal information for understanding the contents.
3. The SSv2 retrieval tasks are only naive extensions of the SSv2 dataset.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The novelty is limited. The authors provide details of the proposed method to reproduce the results.
summary_of_the_review: My main concern is the limited novelty, as it has been investigated by previous works that most video datasets do not need too much temporal information for understanding. The proposed single-frame architecture is a basic structure with self-attention and cross-attention, and the novelty is limited.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the domain of video-and-language tasks and analyses the methods and datasets in the space. Through empirical studies, authors show that using a single frame (randomly sampled) from the entire video is enough to reach similar performance as existing SOTA methods on these datasets. The claim is that such tasks do not need temporal understanding, merely static appearance/image understanding. Further, two new datasets are proposed in this space that require more temporal understanding.
strength_and_weaknesses: **Strengths**

-- The empirical studies presented in the work are insightful and will help propel more understanding in this relatively new domain of text-video understanding.


**Weaknesses**

-- While the empirical studies are certainly insightful, the paper's contributions are limited to just that. There are no other significant technical contributions.

-- While the training is done with single frame, inference is still conducted using multiple sampled frames, which makes the setup a bit hard to fairly evaluate.

-- The claim in the paper that existing datasets in the space (ActivityNet, MSRVTT) do not require any temporal understanding is only partially correct. Given that R1 results are still ~50%, it means that these are not solved datasets. The results point more towards the limitation of existing methods in fully learning/exploiting the temporal information in the videos rather a strict limitation of the datasets.

-- The two new proposed datasets SSv2-{label|template} also have very high (and comparable) R1 scores with just sampling single frame which means those 2 new datasets are roughly comparable with other existing datasets.

-- Overall, since the technical contribution in the paper is quite limited, the Sec 5 (Analysis) does not present any interesting studies and felt forced in the paper.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written for the most part. There is limited novelty in this work.
summary_of_the_review: Overall, not very convinced the paper is ready in its current form, especially as a long-form paper. The technical contributions are limited. The empirical studies do point to limitations of existing methods (and datasets to some extent) but do not propose any effective solutions to overcome such limitations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies a video-language model trained using only a single frame, finding that is can do quite well on many existing benchmarks. The paper then proposes new benchmarks on something-something, where more temporal understanding is required.
strength_and_weaknesses: The paper is well written and easy to understand. The experiments are well done. The finding that many current tasks can be done with a single frame training is interesting and useful. For the proposed task, it would be good to add more details. For example, how many unique labels are there?  It also be valuable to evaluate more existing methods on this benchmark to better understand how existing approachs perform on it.

Minor details: 
The bolding is misleading in Tables 1 and 2.
Should use citep command for references to make it easier to read.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and well written. It has details needed to reproduce the results. It isn't really novel, but has important findings. The proposed new benchmark is interesting and valuable, however the new benchmark is totally clear how useful it is.
summary_of_the_review: The paper studies an important and interesting problem, finding that single frames are good enough for many video language tasks and proposes a solution to it.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper explores single-frame training for video-and language tasks. While simple, this approach can achieve state-of-the-art performance on a range of datasets. This paper also proposes an early-fusion strategy at inference which boosts performance. 
strength_and_weaknesses: Strengths:
1. The single-frame training strategy is highly efficient while effective, which only needs to train one frame during pre-training and fine-tuning. 
2. The proposed model achieves state-of-the-art among the video-text retrieval tasks on VideoQA tasks.
3. This paper is clearly written and easy to follow.

Weaknesses:
1. The single-frame pre-training strategy (see Figure 6) is not fully convincing to me. Although the author claim that “when pre-trained on a sufficient amount of data, the performance of models trained with single frames might be very close to models trained with multiple frames”. However, there only exists 2.5M video-text data among the full 17M pre-training dataset. I think the reason is that with the growth of the #PT images/videos, the large-scale image-text dataset is enough for the model to achieve a good performance on downstream tasks (Note that a lot of recent works [1] have demonstrated that a pure image-text dataset is enough for video-text downstream tasks).
2. Meanwhile, with the growth of the pre-training video-text dataset, there inevitably exists more noisy information and irrelevant frames in the video. Will the single-frame strategy still work on these large-scale web video datasets? I think an additional experiment on a pure large-scale web video dataset (instead of a pre-training dataset mix of videos and large portion images) will further enhance the reliability of this paper (e.g., HowTo100M).
3. The proposed early-fusion strategy is simple and effective, which I do appreciate. However, compared with late-fusion, early-fusion strategy seems to need to input an Nx longer frame feature sequence into the multi-modal encoder. And this may bring an Nx higher memory/inference times cost compared with the late-fusion strategy. The author could further provide a memory/time cost comparison in Figure 4. 

clarity,_quality,_novelty_and_reproducibility: This paper is clearly written and easy to follow. The author also provides the code of the paper. 
summary_of_the_review: Although a lot of paper has proposed the single-frame pre-training strategy (on the pure image-text pre-training dataset), the single-frame finetuning strategy is still novel to me. My main concerns are the reliability of the single-frame strategy on the pure large-scale web video datasets and the memory/time cost of the early-fusion strategy. I will raise my score if these concerns are addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the domain of video-and-language tasks and analyses the methods and datasets in the space. Through empirical studies, authors show that using a single frame (randomly sampled) from the entire video is enough to reach similar performance as existing SOTA methods on these datasets. The claim is that such tasks do not need temporal understanding, merely static appearance/image understanding. Further, two new datasets are proposed in this space that require more temporal understanding.

- I agree with reviewers' point the novelty is limited. The single-frame model applies the basic image-language architectural design with self-attention and cross-attention.
- Lack of significant technical contributions, and the experimental setting is hard to fairly evaluate compared with other methods. 
-  the superior performance in this paper seems to come from a large portion image-text dataset instead of "the single-frame strategy". 

The draft in its current form, with its claimed novelty, is not above the acceptance threshold of ICLR. I suggest rejecting. 
justification_for_why_not_higher_score: The draft in its current form, with its claimed novelty, is not above the acceptance threshold of ICLR. I suggest rejecting.
Decision: Reject