Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper shows the good efficiency and effect of parameter-efficient contrastive vision-language alignment through transfer learning. Specifically, contrastive vision-language models can be created by updating a small set of parameters of pretrained vision models(e.g. DeiT) and pretrained language models(e.g. SimCSE). 
strength_and_weaknesses: Strength

[+] experiment results demonstrate contrastive vision-language models can be created by updates to a relatively small (<7%) set of parameters in pretrained vision and language models, which is a very practical idea.

[+] empirical study illustrates contrastive vision-language models created with parameter-efficient transfer learning conserve useful existing knowledge from their initializations better than full model finetuning

Weaknesses

[-] As the authors claimed in the Limitation section, the conclusions of this work are limited to the datasets, model structure, and downstream tasks.

[-] It is known that batch size plays a key role in contrastive learning. I am curious about the influence of different batch size on your conclusion.

[-] parameter-efficient transfer learning is a hot but not a new topic. Except for Adapter and Unlocking, there are other methods such as Prompt-tuning. Not all parameter-efficient transfer learning methods work for contrastive alignment may be another limitation.

[-] In the Abstract, "vision-language models" and "vision and language model" may make readers confused
clarity,_quality,_novelty_and_reproducibility: clarity: this paper is mostly understandable to me

quality: this is a generally solid work

novelty: this work is creative, but some people could have put these ideas together

reproducibility: Most people can reproduce the experiments in this work
summary_of_the_review: This work illustrates contrastive vision-language models can be created by updating a small percentage of unimodals by gradient descent. Comprehensive experiment results show adapter and unlocking scheme can create CLIP-like models for natural-language classification or image-text retrieval with little effort. Although there is no explanation for this phenomenon.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of efficient fine tuning vision language models. Specifically, the authors seek to answer the following question: "given that strong, pre-trained vision and language models both exist, can we minimally update both of their parameters to align their representations?". The authors adopted the following options 1) initializing the vision and language encoders from pre-trained ones, 2) locking their params, 3) unlocking some of the params and 4) insert a trainable modules (adapters). They showed that by fine tuning just 7% of the parameters, they were able to match the performance of the full fine tuning strategy on retrieval (Flickr) and zero-shot classification (ImageNet)
strength_and_weaknesses: [Strength] The authors conducted many experiments to answer various research questions such as multilingual adaptation and alignment effects. They also did extensive zero-shot experiments comparing with corresponding baselines.[Weakness] This paper is not technically novel and the ideas are simple. It is an incremental step from LiT and from fine tuning approaches that use adapters before. In addition, as stated in the papers, it has a limited set up, which include 1) only testing with COCO dataset, and later 1.5M image-text pairs. It might not work for other datasets, 2) only focusing on zero-shot classification and retrieval tasks. Their conclusions might not hold for other downstream problems, 3) only testing on transformer architecture, which in my opinion, is not a main limitation, however. 
clarity,_quality,_novelty_and_reproducibility: The paper was written with clarity and it is easy to follow.  
The proposed idea and the experimental results are solid (with the understanding of the limitations stated above) 
The approach has limited novelty since it is similar to LiT and to various fine tuning approaches proposed before in the literature.  
It should be easy to reproduce the approach and the results reported in this paper. 
summary_of_the_review: This paper had experimental insights to various interesting research questions. However, it has limited technical novelty and the testbed  is narrow.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a parameter-efficient contrastive vision-language method by leveraging a pre-trained vision encoder and a text encoder. The key idea is to align a well-trained vision encoder and a text encoder by updating a small percentage of their parameters on a small-scale pre-training dataset. The proposed method can achieve comparable performance as full-model training.  
strength_and_weaknesses: Strength:
1. Vision-Language Pre-training (VLP) requires a huge amount of computational resources, which is not affordable by a lot of researchers. Therefore, it is important to design a parameter-efficient training strategy without sacrificing the model performance.
2. The proposed method is simple and can be easily applied to existing methods.

Weakness:
1. The proposed method is quite similar to [1]. Although [1] is designed for fine-tuning CLIP on downstream tasks, these two works share a lot of technical solutions (e.g., layerNorm tuning and adding adapters).
2. This paper is more like an empirical study (combination of existing techniques). The technical novelty and contribution is limited, and new empirical findings are not significant enough.
3. One contribution of this paper is “we show that contrastive vision-language models created with parameter-efficient transfer learning conserve useful existing knowledge from their initializations better than full model finetuning”. This conclusion is obvious. Since parameter-efficient training freezes most of the parameters of the well-trained vision/text encoder, this training obviously will keep existing knowledge.

[1] How to Adapt Your Large-Scale Vision-and-Language Model

clarity,_quality,_novelty_and_reproducibility: This paper is well-organized and easy to follow. The quality is good but the novelty and technical contribution is quite limited. Since the proposed method is simple, it shouldn’t be hard to reproduce the results.

summary_of_the_review: This paper proposes a simple parameter-efficient training method for contrative vision-language pre-training. Although the problem is important and some interesting finds are observed, the technical novelty and contribution are not significant enough.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper addresses an important question regarding a class of language/image models that embed both modalities into a shared vector space, with the embeddings (normally) trained jointly with a contrastive objective. The most famous recent example of this class is CLIP.  This paper investigates an alternative training scheme, in which pretrained models (trained separately for each modality) are adapted for the joint embedding task. 

The reviewers were lukewarm about this paper. The main concerns were limited technical novelty, questions about the scope of evaluation, and overall judgment of impact. The authors addressed all of these in the rebuttal, but unfortunately there was not much of a subsequent discussion. 

I read the paper, the reviews and the responses. Too go over the specific issues brought up:

A. "No technical novelty", "application of existing techniques". This doesn't seem like a real issue to me. A work that shows how to use known techniques better/differently could be much more important than work introducing a new technique that turns out to be a bust. 

B. Insufficient evaluation. I actually think COCO is in itself (barely) sufficient, but the authors report a lot more results in the rebuttal. 

C. "The work might not generalize beyond the specific datasets and tasks reported" - perhaps, but isn't it (not knowing whether it will generalize) true for almost every single empirical paper in the field?
I would consider more heavily a specific concern about a feature of this work that may make it less general, but none were brought up.

D. Similarity to [1] in review by osUw. I agree with the authors that the similarity is superficial.

In my opinion, while indeed the *technical* novelty (introduction of new technical material, like me math, new loss fuction or architecture, etc.) is limited, the contributions are above bar for acceptance. Joint image/language embedding models are at the moment the most promising advancement towards open world image understanding. The cost and complexity of training those models is a significant issue, and the paper provides an interesting component to our emerging understanding of how we may be able to reduce the cost and complexity. It's not perfect, but few papers are.
justification_for_why_not_higher_score: I don't see an obvious reason to recommend spotlight, bit wouldn't be upset if the paper is promoted.

justification_for_why_not_lower_score: It's interesting to a substantial number of people thinking about language and image models.
Decision: Accept: poster