Venue: ICLR 2023 Conference

Reviews:

final_score: rating: null
review_content: It looks like authors did not provide a rebuttal or they decided to make their response to other reviewers only visible by those reviewers. Is this the case?
review_title: No response from authors?

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method for adapting image-text pre-trained models to video-language alignment tasks (specifically text-to-video retrieval). It argues there is a domain gap between text used in video-language pre-training versus downstream tasks, and uses an image captioning model to caption middle frames of the video and use them alongside with the subtitle text in training. It also adds additional global video tokens to the visual encoder and uses an attention mask to mask attention from tokens of each frame to those of other frames. Experiments on text-to-video retrieval show that the proposed approach outperforms competing methods. 
strength_and_weaknesses: **Strengths**:

* The paper is well-written and the proposed approach is described clearly. 

* The authors motivate the problem well with the experiments on the impact of data scale and language domain gap.  

* CLIP-ViP outperforms baselines on various datasets such as MSR-VTT, DiDeMo and ActivityNet. 

**Weaknesses**:

* The authors only evaluate their approach on the text-to-video retrieval task. It is unclear whether the method also provide benefits in other tasks such as video action recognition. Given that we can formulate video classification as video-text alignment (using text prompts), does the method perform well on this task as well?

* The authors argue that there is a discrepancy between language data in pre-training and downstream tasks, and propose their captioning method accordingly to bridge this gap. As the downstream task is limited to text-to-video retrieval, it is questionable how generalizable the captioning approach is. Providing results on other downstream tasks can clarify this. 

* Using the additional captioning model adds extra compute to the process that needs to be discussed. 
 
* Regarding the plots in Figure 1, does this overfitting hold for different values of learning rate and optimization schedule? 

* There are a few errors in the manuscript, e.g. section 4.3. "we definite" -> "we define"
clarity,_quality,_novelty_and_reproducibility: * The method includes new components for video-language training from CLIP-like models such as video proxies and proper attention masks, and it has moderate novelty. 

* Experimental results are convincing and show superior performance compared to the baselines. 

* Presentation is clear and well-motivated. 

summary_of_the_review: Overall, the proposed approach is sound and shows reasonable performance gain in the experiments. However, it is mainly evaluated on a single task which makes it questionable to what extent the method is generalizable to other tasks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an adaptation of the ViT based CLIP model for video-text alignment that learns the spatio-temporal structure of the videos during the alignment. The authors start by analysis the problems with current approaches that straight forwardly apply CLIP on video for video-text alignment. They find that data scale and domain gap are big factors in the performance of models. Based on these findings, the authors design a video proxy to allow the model to understand spatio-temporal aspects of video while also combined captioning and subtitle language information from videos for alignment. In the experiments, the authors show multiple ablations to validate the different components of their method and also outperform that state of the art in many datasets.
strength_and_weaknesses: Strengths:
+ New adaptation of ViT for video-text alignment
+ Captioning strategy for full videos using middle frame.
+ Info-NCE adaptation to include video subtitles and video caption.
+ Outperforms baselines and informative ablations


Questions/comments:
- How are negatives chosen?
There are two different text inputs being contrasted, subtitles and captions. I may have missed it from the text, but I couldn't find any details on the strategies used for getting the negatives for Info-NCE. Could the authors provide these details?


- Using DSL posprocessing on all baselines?
The best reported results in this paper are after using the DSL posprocessing. The author mention that they use this for fairness against other baselines. However, I am not sure if this was used in all the baselines. Can the authors comment on whether this makes sense doing or not?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and of high quality. There is good novelty, and I believe the work could be easily reproduced.
summary_of_the_review: All-in-all, I like this paper and I think it should be accepted in the current form. There is analysis of problems from previous methods while tackling this problem. There is novelty in the proposed method and objective strategies, and the method outperforms previous baselines.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper attributes the weakness of post-pretraining of text-video models (based on the text-image model) to the domain gap between video subtitles and image captions. The authors develop several methods to improve the effectiveness of post-pretraining: data augmentations via video captions, video proxy tokens, and omnisource cross-modal learning.

strength_and_weaknesses: Strength:
The paper is well organized.
The performance is very impressive.


Weaknesses:

1. One primary concern: is the domain gap between language sources really the underlying problem? Do the authors really solve this problem in this work?

1.1) In Table 1, we can see $HD-VILA_{cap}$ has a far smaller NMI than $HD-VILA_{sub}$. If the domain gap is the real problem and $HD-VILA_{cap}$ has the same amount of data as $HD-VILA_{sub}$, in Table 4, $HD-VILA_{cap}$ should have much better performance than $HD-VILA_{sub}$, but we can this is not the truth. Indeed, $HD-VILA_{cap}$ has worse performance than $HD-VILA_{sub}$, and $HD-VILA_{cap}$ is even worse than w/o post-pretrain. This violates the domain gap assumption of the authors.

1.2) In Table 1, CC12M has a smaller NMI than $HD-VILA_{sub}$. In Table 4, we can see only ImageCaption pre-training does not help fine-tuning. There are two possibilities: a) the domain gap assumption does not hold; b) we should also consider the domain gap of visual signals. It is also weird why $HD-VILA_{sub}$ + ImageCaption get such good performance as the two work ordinarily without each other.

1.3) Following 1.2, could the authors provide analyses of domain gaps of visual signals across different datasets?

1.4) I think there is a publication that can explain the improvement in performance when using $HD-VILA_{sub}$ + $HD-VILA_{cap}$ [1]. I purely consider $HD-VILA_{cap}$ as a bootstrapping of captions or data augmentations. Hope the authors can turn my mind around. Besides, I think one way to verify this idea is by adding HowTo100M + HD_VILA result in Table 4. If this does not work, I may lean toward the domain gap assumption.

2. What is the difference between video proxy tokens and video prompts? I think they are just the same thing with various names. 


[1] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. https://arxiv.org/abs/2201.12086


clarity,_quality,_novelty_and_reproducibility: The quality, clarity, and originality of the work meet the requirement of an ICLR publication.

summary_of_the_review: The paper does an excellent work in the post-pretraining of text-video models. I hope the authors can resolve my concern about the domain gap assumption.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new framework called CLIP-VIP to transfer the image-text model to video-language alignment. It has been found that the scale of dataset is crucial for post-pretraining. Therefore, the authors apply HD-VILA-100M due its large scale and diverse category. The other factor is the language domain gap measured by the NMI score in Table 1. Therefore, CLIP-VIP introduces an additional captioning model (OFA-Caption) to augment text data as another source. To enable cross-model post-pretraining based on video-text pairs, the authors invent video proxy tokens to summarize the content of the whole video temporal-spatially. The model is trained by the info-NCE loss in a self-supervised way.


strength_and_weaknesses: Pros:
+ The proposed CLIP-VIP model successfully extends a pre-trained CLIP model to the video-text tasks with only limited revisions, which marginally increases negligible parameters and computations.

+ This paper has systematically explored the crucial factors for model transfer between image-text and video-text tasks, which may shed light on the following works.

+ It has achieved very competitive results on video-retrieval tasks on many benchmarks. Moreover, there are solid ablation analyses for different components as a thorough evaluation.

Cons:
- This paper is not written and organized well. Without extra reading efforts, I cannot easily understand the motivation and the proposed method. Some of the sentences are pretty long and unfriendly to read. The introduction section focuses too much on algorithms/methods instead of its motivation. As a result, I am still confused about the major contributions of this paper.

- The major technical contribution of this paper seems to be proxy video tokens which are simple and trivial. Moreover, how to assign the number of proxy video tokens needs some manual effort, which degenerates its value in practice.

- The post-training is costly, considering the 32 GPUs in use. So, I have not seen the extraordinary benefits of model transfer in efficiency. The authors are suggested to compare the cost of training from scratch and post-training to highlight the advantages of model transfer.

Questions/Other Concerns:

1. How to assign the M (number of proxy video tokens) in practice? What will be if M is small or large?

2. How many GPU hours are needed for post-training? Is it more efficient than other methods?

3. What is the meaning of single frames (F) in page 5? What is the difference between F and video sequences (V)?

clarity,_quality,_novelty_and_reproducibility: The quality, clarity and originality are around average.
summary_of_the_review: Based on all the comments above, I am leaning toward weak rejection. I will consider upgrading the score after a strong rebuttal from the authors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper starts with two questions that try to answer what and how to apply CLIP to video-text tasks. They build a model based on the observations and propose new cross-modal learning methods for video-text retrieval. Their method was tested and showed promising results on those datasets. Adapting the image-caption pretarined model to video is a challenge. The findings in this paper can serve as a good baseline for the bridge of image-caption pretrained models and vision-text tasks.
strength_and_weaknesses: - As for Further pretraining on small data, some details are missing; for example, did the authors try to freeze the backbone and optimize the head only? 

- The adopted OFA was powerful and could generate diverse and well captions. However, such a strong captioning model also brings another problem, what if we do not use OFA, but use previous models such as BUTD, SCST, etc. Then what are the effects of image captions on the final performance?

- The proxy-guided attention looks new and interesting, and the results shown in Table 2 look promising. Showing some visualized attention samples will help the understanding of what the proxy learned.

- The cross-modal learning part is straightforward and combines several contrastive learning tasks. The novelty of this part is somehow limited, especially considering that in table 3, some losses do not bring a big performance. They simply combine some possible losses together and lack strong motivation.

- Given that they have already adopted OFA as the caption generation module and they have already got a large amount of data, then what if they train the model without the initial weight from CLIP? I think such a comparison can also show how much benefit their model leverages from the CLIP model.
clarity,_quality,_novelty_and_reproducibility: The writing of this paper is easy to read, and their proposed model shows good results on multiple datasets. The major concerns are the use of the powerful image captioning model and the contribution of model design and losses. The current major contribution is proxy-guide attention (the combination of contrastive loss is still limited).
summary_of_the_review: The proposed model achieved good performance on test datasets. The additional computation and proxy-guided attention need to be discussed more.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary

This paper studies how to best apply the image-based CLIP model to video-text tasks, and how to close the domain gap between text used in image-language pre-training versus downstream tasks, specifically text-to-video retrieval. Authors find that amount of data is a major factor. The paper uses the image captioning model to caption middle frames of the video and use them together with "global" video tokens alongside with the subtitle text in cross-modal multi-source training. The spatio-temporal structure of video is learned with a masking structure and alignment, and the authors show multiple ablations to validate the different components of their method. They outperform the state of the art on many datasets. 

Strengths

- The paper starts from a well-defined scientific question, and provides ablations that support the authors hypotheses and conclusions. The vast majority of reviewer questions were answered during the discussion and taken into account. Results are found to be "convincing"
- The paper is well-written and the proposed approach is described clearly. One reviewer finds that the paper contains long sentences, but the overall consensus is that the paper is well written.
- CLIP-ViP outperforms baselines and comparable work on various relevant datasets such as MSR-VTT, DiDeMo and ActivityNet.
- The paper presents a new adaptation of ViT for video-text alignment, a captioning strategy for full videos using middle frame, and adapts Info-NCE loss to include video subtitles and video caption. It outperforms baselines and informative ablations

Weaknesses

- The authors only evaluate their approach on the text-to-video retrieval task. It is unclear whether the method also provide benefits in other tasks such as video action recognition. 
- The authors argue that there is a discrepancy between language data in pre-training and downstream tasks, and propose their captioning method is able to bridge this gap. As the downstream task is limited to text-to-video retrieval, it is not clear how geheralizeable the method is
- The method can become computationally expensive, and ablations on pre-training would be useful, as they would help reproducibility. 



justification_for_why_not_higher_score: The model has strong performance against relevant benchmarks, but all reviewers mark that the overall novelty of the paper as "Aspects of the contributions exist in prior work." A higher score would also require more detailed and insightful ablations, specifically on the existence and effect of the "domain gap", which is a central part of the motivation of the work. Two (of five) reviewers cite this as their main concern, with one still recommending acceptance.
justification_for_why_not_lower_score: The authors showed strong performance on the video retrieval task. Reviewers appreciate the quality of the authors' feedback and the preparation of an updated paper. Only one reviewer (out of 5) does not explicitly recommend acceptance of the paper on the basis that the computational complexity needs to be discussed more. However, authors provide analysis responsive to the reviewer's comments, which the AC thinks should address the concern.
Decision: Accept: poster