Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper utilizes recently proposed CLIP-ResNet/ViT visual encoders instead of the standard backbones to conduct a large-scale empirical study for several vision and language tasks. The experimental setup aims to answer the question in the title of this paper by conducting 1) the task-specific fine-tuning of existing V&L models with CLIP-based visual backbone and 2) performing full-scale V&L pre-training.
The resulting performance in VQA, Visual Entailment, V&L Navigation suggests that CLIP is a viable alternative to the existing visual representations (e.g., ResNet-based models pre-trained on ImageNet).
main_review: Strengths: 
The paper revisits a broad range of vision and language tasks (VQA, VLN, SNLI-VE, Image Captioning) and evaluates the would-be performance for previous state-of-the-art methods by replacing their visual backbones with CLIP-ResNet/ViT. The diversity of the tasks considered and the two scenarios explored in the paper (straightforward fine-tuning and V&L pre-training) illustrate the strengths of the CLIP-based encoders and highlight some of their limitations. 

Weaknesses: 
The motivation for all V&L experiments looks weak because it stems from the following single observation “if cast VQA 2.0 into a zero-shot image-to-text retrieval task, we only observe chance performance. Thus, we propose to integrate CLIP’s visual encoder with previous V&L models.” The prompt engineering considered for VQA is not persuasive and does not imply the same straightforward introduction of CLIP-based backbones to VLN / Image Captioning tasks. Also, I think it is necessary to explain the “chance performance” for VQA, at least for the binary yes/no question type. E.g., does 0.037 mean that one can simply flip the model’s answers?

“Unfreezing” the Visual Backbone helps if done correctly – it is a well-known fact supported by BUTD-Res101 “unfreezing” experiment in the paper. However, CLIP-Res50 benefits more from pre-training than BUTD-Res101. This observation requires further elaboration.

Most of the models considered in their respective tasks seem outdated as of 2021 (e.g., Pythia 2019, MCAN 2019). At the same time, the authors avoid direct comparison to the methods that require large-scale “supervised” or “self-supervised” pre-training, e.g., VLN-BERT, although CLIP itself can be considered as such.

The authors may want to include findings on localization ability of transformers from the following papers in the later versions of the manuscript:
-	Do Vision Transformers See Like Convolutional Neural Networks? arxiv 2021
-	Dynamic Head: Unifying Object Detection Heads with Attentions, CVPR 2021
-	Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions, ICCV 2021

summary_of_the_review: It’s not surprising that the CLIP-based vision backbone - after being trained on 400M image-text pairs - can outperform ImageNet-trained baselines in vision and language tasks, especially given its zero-shot performance illustrated in the original work.
The presented results can be helpful for the research community, although the answer to the question posed in the title remains largely open. The outcomes of experiments vary from task to task, with no appropriate discussion followed even for the difference between CLIP-ResNet / CLIP-ViT-B features. As it reads in the Conclusion section: “Analyses from different perspectives explain certain intriguing phenomena and offer new directions for future V&L research.”
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper examines how well CLIP's visual encoders are transferred on vision-and-language (VL) tasks.  
The paper conducted three without-VLP tasks: VQA, image captioning, and vision-and-language navigation, and three with-VLP tasks: VQA, SNLI-VE, and GQA to show CLIP's visual encoders' transferability,   
CLIP's resnet-based visual encoders consistently outperformed their Imagenet pre-trained counterpart.  
The authors found that CLIP's ViT-based visual encoder performed far worse than the resnets.  
From Grad-CAM and detection fine-tuning experiments, the authors speculated that ViT's features lack localization information.  
main_review: The tendency that more powerful visual encoders yield more performant VL models has been discussed and demonstrated in numerous papers (from the BUTD [^1] to VinVL [^2]), which emphasized the need for powerful visual backbones.  
As CLIP showed the great generalization power of their visual encoders compared to the encoders trained with imagenet classification, I feel no surprise they boosted the performance of VL tasks compared to their imagenet pre-trained counterpart.  
I must credit the paper for solidifying this tendency, but I am doubtful whether this paper opens any new directions to the community.
All experiments are replications of existing baselines, just switching the backbone to publicly shared CLIP weights.  
I think not many insights besides "switching the visual encoder to CLIP's bring the performance boost" are given.  
I was hoping the paper contains some analysis of CLIP's text encoder since the language side of VL models at least take something from what the CLIP's text encoder has learned, but sadly, I found only plug-and-plays of CLIP's visual encoders.  
To raise my recommendation, please clarify what directions the paper could suggest to the community other than the general tendency I've mentioned.  

[^1] Anderson, Peter, et al. "Bottom-up and top-down attention for image captioning and visual question answering." _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2018.  
[^2] Zhang, Pengchuan, et al. "Vinvl: Revisiting visual representations in vision-language models." _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2021.  
summary_of_the_review: Not many insights besides "switching the visual encoder to CLIP's bring the performance boost" are given.  
To raise my recommendation, please clarify what directions the paper could suggest to the community other than the general tendency I've mentioned.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper performs empirical analyses of applying the CLIP to various vision-and-language tasks. It demonstrates the potentials of the model in generalizing to different downstream applications, and provides some suggestions for model deployment. Experimental results show that CLIP pretraining leads to competitive performance and further combining it with V&L pretraining can outperform existing methods.
main_review: As a paper focusing on empirical studies, it has the following strengths:
+ The paper conducts extensive experiments to study the generalizability of CLIP pretraining across various tasks. The ablation studies on different types of visual encoders also provide insights into the application of pretraining models.

+ The proposed method is able to establish new state-of-the-art performance in multiple vision-and-language tasks, which can benefit future research in the related fields.

However, there are also some notable weaknesses:
- While I am convinced that CLIP could be beneficial for downstream tasks, it is still unclear to me what are the advantages of CLIP over existing V&L pretraining techniques and how does it achieve good results. For a purely empirical study on an existing technique,  I expect more in-depth analyses than simply showing a collection of results.

- After comparing CLIP with the other V&L pretraining techniques (e.g., Table 2 and Table 6), it seems that it is more advantageous only if used together with V&L pretraining. This is a bit tricky, because the V&L pretraining relies on 9.18M additional samples. It is unclear if the improvements achieved by the method truly result from the advantages of CLIP or are simply due to the use of more external data.

- Relating to the above comment, the analysis on unfreezing encoder is also problematic, as the comparison is between models without pre-training and with V&L pretraining. The higher accuracies of the latter ones could be attributed to the use of additional data, instead of the differences between frozen/fine-tuned visual encoders.

- It appears that CLIP only works well with grid-like features extracted from convolutional neural networks, i.e., no experiment on regional features and worse results when combined with a visual transformer. I am aware of the analyses on ViT-B, however, they only point out the defects of a single model. It does not answer the question of why CLIP can not be applied to different visual transformer models and features.

summary_of_the_review: While I have no doubt that CLIP could offer opportunities for developing new state-of-the-art models, this paper falls short of explaining the reasons behind the successes of CLIP. I also found several experiments and analyses problematic and do not quite support the claims. Overall, this is a borderline paper to me as it does demonstrate the potential of CLIP with good results. Therefore, I am slightly leaning towards accepting it.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper explores how features from CLIP (Contrastive Language-Image Pre-training, Radford et al., 2021) affect the performance of vision-and-language models across a series of tasks. The authors explore using CLIP as a visual encoder in two settings, plugging its features directly into task-specific fine-tuning; and combining CLIP with intermediary vision-and-language pre-training before fine-tuning on downstream tasks. The experiments suggest that the simple change to CLIP offers significant benefits over commonly used encoders such as BottomUp-TopDown, providing very strong results across a wide range of vision-and-language tasks.
main_review: **Strengths:**

1. This work presents experiments on straightforward, but significantly important baselines for the vision-and-language community. The vision features have been shown to be a bottleneck of vision-and-language systems, and a significant part of recent progress can be attributed to better visual encoders. CLIP has been greatly impactful in the field, and the experiments presented in this paper are therefore a must-know to the community, and would be of interest to many.

2. The experiments presented in this work are solid, informative and representative.

3. The paper is clear and well written.

**Weaknesses:**

1. While the the experiments presented in this paper are of great importance to practitioners and researchers interested in vision-and-language, a shortcoming of this paper is the lack of novelty.

2. The authors experiment only with a subset of the publicly released CLIP models, and do not provide numbers for the largest available models. While OpenAI only very recently released more models, it would be great to see their performance in a future version of this paper.

**Other comments:**

CLIP ViT-B could refer to both the ViT-B/32 and ViT-B/16 architectures, and it would be great if authors clarified this in their manuscript.
summary_of_the_review: This work introduces a must-know baseline for vision-and-language research: using CLIP as a visual encoder. While this paper does not introduce a novel method, it provides important experimental information to many in the community. The results here presented would be of great interest to many, and I recommend it's acceptance to the conference.
review_title: null

Meta-review: Reviewers are in agreement that this work is a useful, clear, documentary piece of work that shows the utility of CLIP on a number of popular V+L tasks.  There is a somewhat persistent concern that simply demonstrating that a stronger visual encoder leads to improvements downstream is not an insightful result on which the community can build.
Decision: Accept (Poster)