Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for video-language retrieval. The method is based on pretraining a general still image detector and feeding the pooled output of the detector plus position+time features into a transformer to produce a video embedding. A global loss on cls video embedding, as well as local individual token losses (reminiscent of ColBert) are used to train the model. The approach is benchmarked on several datasets and exhibited better performance compared to the baselines and used less pretraining time (though that probably does not include the detector pretraining).
strength_and_weaknesses: Strengths:
1) The approach allows getting a higher performance compared to the baselines. Pretrained detector allows to directly inject information from the labeled detection datasets, plus word alignment helps improve the model convergence.
2) The general idea and solution are presented quite clearly in the paper.

Weaknesses: 
1) No instructions on how to run the code and reproduce the results. This is critical as the paper claims "democratization" of the video retrieval. 
2) The paper writing might be improved. There are several references to the equation (10) in the paper, while it is actually in the supplementary, which does not make much sense. The tables 4,5,6 are not clear, particularly what is "base". The section 3.3. is hard to follow, the equations lack a word explanation of why they intend to do, there is also no corresponding illustration for those.
3) The detector network is old and not trained end-to-end. It is mentioned in the text there are no bounding-boxes provided for training, but there is also an option to just backpropogate the final training loss back to the backbone on some architectures.

clarity,_quality,_novelty_and_reproducibility: See "Strength And Weaknesses" for clarity, quality and reproducibility.

In terms of novelty, I think most of the components have been used before, though for the practical types of papers I guess it does not matter that much.


summary_of_the_review: It does not seem like the paper offers a big breakthrough in terms of ideas, but the focus of the paper is pretty clear, so novelty does not seem be an issue.
I think for democratization of the video retrieval the major ingredient is the reproducible code that is easy to run. The submission does have a code associated with it, but it lacks instructions on how to run it, which creates artificial difficulties for a review. 
The paper also has issues with presentation discussed in the weaknesses, so overall I vote for slightly below the acceptance threshold.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an efficient video-language pre-training method. The key idea is to employ region-text pairs during the pre-training. Different from recent works that use end-to-end transformers, the authors revisit the object region features extracted from pre-trained object detector. The idea is somehow similar to OSCAR, but extending such use of region-text pairs for video-language problem. Experimental results show that their method achieves better performance on downstream image-text retrieval.
strength_and_weaknesses: Strengths:
1. The paper is well-written and easy to understand. The presentation is clear.
2. The authors proposed an efficient pre-training approach for video-language tasks. Some discussions regarding video temporal and spatial redundancy are interesting.
3. The authors provide thorough experiments and show that their pre-training is faster than prior works.
4. The pre-training objective, including video-sentence alignment and region-word alignment, is technically valid.

Weaknesses:
1. The proposed method heavily relies on pre-trained object detector. It seems very challenging and not affordable to extract/store all the region features, especially for large-scale video datasets. Such solution seems not appealing as the object detectors are keep improving. The region features need to be re-extracted if newer object detector is available. The feature extraction is very time consuming. Feature storage is very expensive. All video files need to be stored in the system for feature extraction if new object detector is available.  
2. The proposed method use mixture of video-text and image-text data for pre-training. But it is unclear what is the pre-training objective for the image-text inputs. I wonder if the proposed pre-training mainly learns from image-text inputs or video-text inputs. 
3. The authors only conduct experiments on retrieval tasks. It is unclear if the pre-training is still useful when it comes to video QA and video captioning.
clarity,_quality,_novelty_and_reproducibility: The idea is clear and easy to understand. 
summary_of_the_review: Overall, the proposed method achieves nice improvements compared to previous SOTA on image-text retrieval. This shows that prior end-to-end methods have a lot more spaces to improve. However, the proposed method seems very expensive in terms of feature extraction and feature storage. It may have difficulty to scale. In addition, as a pre-training paper, it would be good to verify their method on multiple downstream tasks. Now, the paper only considers retrieval, which is a bit narrow.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new video-language pre-training (VLP) method which is computationally efficient. The key idea is to reduce (1) temporal visual redundancy by frame sampling and (2) spatial visual redundancy by using region features that are extracted by a pre-learned object detector. To align region features and text tokens, this paper also proposes a bidirectional region-word alignment regulation. The efficiency and performance of the proposed method is empirically proved. 

strength_and_weaknesses: Strength:
1. VLP requires a huge amount of computational resources, which is not affordable by a lot of researchers. Therefore, it is important to reduce the training time without sacrificing the model performance.
2. The performance is good.
3. The paper is well-organized and easy to follow.
4. Thanks for proving the code which can help readers to reproduce the results.

Weakness:
1. On page 5, what’s the motivation of this Region-Word Alignment design? Why is it better than existing local alignment methods, such as FILIP? There are lots of work that utilize optical transport (OT) to align regions and words (e.g., UNITER: UNiversal Image-TExt Representation Learning), what’s the advantage of the proposed Region-Word Alignment compared to OT? 
2. To reduce the pre-training time, this paper proposes two strategies, i.e., (M1) reduce temporal visual redundancy by frame sampling and (M2) reduce spatial visual redundancy by using region features. M1 is widely used in exiting VLP methods (e.g., Align and Prompt). In terms of M2, there are some works (e.g., MAE and VideoMAE) that mask 90%-95% patches/tubes/cubes in image/frame/video to reduce the spatial visual redundancy, what’s the advantage of using region features from a pre-trained object detector? Of course, the pre-trained object detector can be used multiple times.
3. This paper uses a well-trained Faster-RCNN to extract region features then uses ViT to learn interactions between region features. Is that fair to compare with other methods which only use ViT as the video encoder?

clarity,_quality,_novelty_and_reproducibility: This paper is clear enough to understand. The quality is good, but the novelty is marginal as mentioned in weakness. Since the code is provided by the authors, its reproducibility can be guaranteed. 

summary_of_the_review: The method proposed in this work demonstrates significant performance improvement in retrieval tasks. However, the technical novelty of this paper is marginal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an approach for video-language retrieval (and in supplementals Video QA). The approach uses a detector features from which are leveraged in a transformer with a novel bidirectional region-word alignment regularization. Paper was reviewed by three reviewers and received the following ratings:

* 1 x reject, not good enough
* 1 x marginally below the acceptance threshold
* 1 x marginally above the acceptance threshold

No reviewer strongly argued for the paper. Concerns of reviewers were multi-fold: (1) limited novelty [yXUD, nWq8, ymrc], (2) issues with presentation and exposition [yXUD] and (3) limited evaluation on only one/few downstream tasks [nWq8]. It was also unclear if the pre-traded object detector time was counted in benchmarking. 

Authors have provided a rebuttal that tried to address some of these concerns. Reviewers have not commented on these argument and have not upgraded the scores. AC has carefully looked at the reviews, rebuttal and paper itself. Overall, AC agrees with limitations stated by the reviewers and does not believe rebuttal addresses aforementioned limitations well. Specifically:

* The novelty of proposed approach is indeed very limited. Other methods have proposed similar region-word alignment objectives. Even though proposed formulation is different, it does not fundamentally add any deep insights. 

* The performance while is indeed faster than prior works, cannot match the computational performance of more recent architectures (e.g., MERLOT, VIOLET and a number of others). Also with respect to those architectures, performance is shown on fewer tasks. 

* The issues with presentation and exposition, while improved in the revision, are not completely relinquished. 

Based on these and other factors mentioned by the reviewers, the paper does not appear to be ready for publication at this time. 


justification_for_why_not_higher_score: While the paper does propose a computationally efficient model, it does not appear to be able to provide competitive performance compared to the most recent VLP models (this field is very dynamic). Given this and the relative lack of novelty, I do not see a lot of insights this paper can bring to the already densely populated, in terms of publishing, field.
Decision: Reject