Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel approach named context optimization (CoOp) for prompt engineering of vision-language pre-training models.   The main idea is to model context in prompts using continuous representations and perform end-to-end learning from data while keeping the pre-trained parameters fixed. Experiments on 11 datasets show that CoOp effectively turns pre-trained vision-language models into data-efficient visual learners, requiring as few as one or two shots to beat hand-crafted prompts with a decent margin and able to gain significant improvements when using more shots.
main_review: Strengths:
1. This paper provides a novel approach named CoOp for prompt engineering based on  CLIP. 
2. CLIP-based experiments are sufficient to prove the advantages of CoOp.

Weaknesses:
1. Only the results based on CLIP are compared, and there are no more experiments of other visual-language models.
2. It is an improvement of CLIP, and the idea is similar to many existing works such as [1].
[1] Prefix-Tuning: Optimizing Continuous Prompts for Generation

summary_of_the_review: This is a somewhat novel but solid work. The comparative experiment based on clip is very sufficient, but it also lacks other important experiments, such as the effect of CoOp on other vision-language models, just as the title of the paper is learning to prompt for vision language models
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes context optimization (CoOp) which learns task-aware continuous prompts to improve CLIP in terms of few-shot image classification. By fixing the pretrained backbone, CoOp performs end-to-end learning to update the learnable context vectors for target domain datasets. The simple yet effective approach substantially beats hand-crafted prompts with a large margin. Meanwhile, CoOp also exhibits better robustness to distribution shift than CLIP.
main_review: Strengths: 
1) The paper is generally well-written and easy to follow. The author conducts extensive experiments and ablation studies.
2) CoOp outperforms CLIP on all 11 diverse datasets, especially showing large improvements in specific domains like texture and satellite images.
3) CoOp is data-efficient and can boost the classification performance with only a few shots of training data.
4) CoOp demonstrates better robustness to distribution shift than zero-shot CLIP and linear probe CLIP. 

Weakness: 
1) Though effective, the technical novelty of this paper is quite limited. Since the soft prompt tuning approaches (e.g., Prompt Tuning [1] and P-Tuning [2]) have already been proposed in NLP domain. And CoOp adopts a very similar technique.
[1] Lester, Brian, Rami Al-Rfou, and Noah Constant. “The power of scale for parameter-efficient prompt tuning.” EMNLP 2021.
[2] Liu, Xiao, et al. “GPT Understands, Too.” *arXiv preprint arXiv:2103.10385* (2021).

2) According to Table 4 in the paper, the nearest neighbor words of the learned context vectors rarely have practical semantic meaning. This casts doubt on using the word “context”. From this perspective, these soft prompts are just more parameters to improve the model capacity.  Hence, can we say that CoOp is just a better version of fine-tuning? In other words, there might be a similar fine-tuning way to utilize additional parameters to get better performance.
summary_of_the_review: The paper achieves better few-shot image classification performance improvements but lacks enough technical novelty and explanations for learned prompts
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors demonstrate a more efficient form of few-shot learning
using CLIP compared to linear probing for image classification: CoOp.
Instead of fine-tuning a small linear classifier on the output of
CLIP, they propose fine-tuning a number of additional embeddings at
the input layer; this modification, in theory, allows CLIP to leverage
more computation when adapting to tasks, while still only optimizing a
small number of parameters. Experiments across several corpora
demonstrate the efficacy of the approach, which generally yields a
few accuracy points of gain versus a linear probe trained on the same
amount of data.
main_review: The paper's motivation is clear, experiments thorough, and results
convincing. In addition to the standard few-shot evaluations, I
appreciated the authors considering the distribution shift scenario:
in that case, they show that their model can more effectively leverage
labelled data from a different dataset versus the linear probes (even
though zero-shot CLIP remains somewhat competitive in that
regime). Additional experiments about the optimal context length,
placement of fine-tune-able token embeddings, and vision backbones
where appreciated.

While the CoOp method appears to work well, but I had some concerns
about novelty. In particular, this method is more-or-less identical to
Li and Liang (2021) --- that paper came out on arXiv Jan 1 of this
year, and was published at ACL earlier this year. In addition, this
idea has been "rediscovered" in the NLP context several times (which
are also cited, but perhaps cannot be considered contemporaneous,
given their arXiv dates...). While the authors cite the arXiv version,
I think that, given the timing, the authors should perhaps not pitch
their method as a "novel" approach (as is done in the abstract);
rather, this seems to be applying prefix-tuning to CLIP.

I had a few technical concerns:

1. there are a lack of details for the linear probe --- how was the
   regularization parameter chosen?
   
2. The baseline for comparison to CoOp was the linear probe, which
   makes sense. However, I would have also liked to have seen a
   comparison where all the parameters of CLIP are fine-tuned --- how
   well does that work for few shot learning?

I also had a few presentation concerns:

1. Figure 1 compares a supervised CoOp method to a zero-shot CLIP
   baseline. While the CoOp method is "few shot", in this figure,
   there are 16 examples per class provided, which, for some datasets
   may amount to hundreds or thousands of labelled examples. I would
   have appreciated Figure 1 comparing to the linear probe.

2. Figure 5b has a similar problem with being potentially misleading:
   I would recommend including not only zero-shot CLIP, but also
   linear probe CLIP.
summary_of_the_review: Overall: the work is generally clear with thorough and convincing
experiments. While there were a few presentation/technical concerns,
the main drawback of this work is novelty: the proposed idea is
identical to Li and Liang (2021) [arxiv in january], Zhong et
al. (2021) [arxiv in April], and perhaps Lester et al (2021) [arxiv in
Sep, this one is more recent and I haven't read it yet]. While these
papers consider only the NLP case and not the vision+language case,
it's still difficult to call this method "novel," as the authors do in
the intro.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposes a context optimisation (CoOp) approach that can automate prompt engineering and allow more efficient and task-specific transfer for pretrained vision-language models.
main_review: + Extensive experiments
+ Motivation for the automatic prompt is useful

o Investment in Fig 1 is super useful. It is helpful to see how fragile the prompt is affected by word twisting. However, the paper claimed prompt engineering is time-consuming, which I do not disagree with. Yes, human annotators have no way to know whether adding a "flower" or changing the sentence order will improve or harm the performance. But the key motivation of the prompt is to reduce the human working load and adding some general description is not super complicated. The task should be down to exploring more robust models and algorithms that can withstand these minor changes in my opinion.

- The biggest concern is about the lack of theoretical contribution. The main content of the methodology is broken down into two sections. The first section is just a review of CLIP which is already well-known. The key content of context optimisation in 2.2 is just half a page. The description is rough.

- It is not super clear why adding extra dimensions (claimed as the same number in that of word embedding) to the class token can help the prompt.

- On page 4 above Eq.3, the concept "unified context" and "class-specific context" are not well explained.

- paragraph under Eq.3, "Training is performed ..." is redundant and not informative.

- Figure 2 is not helpful in understanding the framework and looks very low-quality.
summary_of_the_review: Despite extensive evaluation, the paper presents in very low quality and lack of description of key methods and the theory behind it. The overall quality is clearly below the threshold of the ICLR standard.
review_title: null

Meta-review: Given the increasing scale of large models (e.g. CLIP), there's an argument that we need better automated techniques for properly utilizing (prompting) these models. Given the success of prompt learning within pure NLP models, the authors apply the same approach to the V+L domain and show that it also is applicable here.  Generally, reviewers felt that the results were clear and thorough, yet technically limited.  The approach is not novel and the result not surprising.  There is a documentary benefit to having this work out in the community for others to reference and extend.
Decision: Reject