Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, prompt learning is reexamined, and several unexpected findings that defy accepted notions of the prompt are presented. First , random prompts without learning or fine-grained design may likewise function effectively in zero-shot recognition. Second, direct linear classifier fine-tuning performs more effectively than prompt learning. Furthermore, prompt learning is essentially a subset of parameter-efficient learning and represents a trade-off between generalization and optimality.  Findings across 11 datasets show that the approach presented in this research can significantly influnce the use of trained vision-language models in subsequent challenges.
strength_and_weaknesses: Strength
1. The experiments are sufficient and exhaustive.
2. The results provide the valuable hints that what's the better way to deploy the pretrained vision-language model.
3. The paper inspire people to think about more effective prompt design.

clarity,_quality,_novelty_and_reproducibility: Weakness,

1. The results are sufficent and the conclusion are well established but the reasons behind the result need to be more explored. For example, why classifier fintuning is much better than learning prompt?

2. The novelty is limited but as a rethinking paper, it is fine.

3. Can other models except clip still support findings?

summary_of_the_review: Overall, this paper provides some valuable hints about prompts but the novelty is limited. Therefore, I give my initial rating as borderline
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper analyzes the behavior of CLIP towards various compositions of hand-crafted text prompts and puts out some observations. One of the key observations of the paper is the behavior of CLIP to negative prompts, in which the authors insert a “not” word to the prompt, e.g. “this is not a photo of a dog”. The paper shows that the zero-shot classification performance of CLIP is almost the same using negative prompts as using default prompts without the “not” word included. The paper also shows that the use of random tokens appended with the class name as prompts also give considerable performance. In the second part of the paper, the authors propose classifier fine-tuning in which they fine-tune the final layer of the text encoder instead of learning the prompts. The authors argue that this tackles the speed issues of prompt learning by not needing to backpropagate gradients through the whole of the CLIP model and gets better classification performance. Experiments include zero-shot classification using various hand crafted and random prompts, few-shot classification, robustness to distribution shifts and studies on optimality and generalization. Results are shown on 11 datasets.
strength_and_weaknesses: Strengths:

* The observations shown in the paper regarding the performance with various types of handcrafted prompts and their compositional variations is interesting.

Weaknesses/Questions/Suggestions:

* The experiments are only on the CLIP model, while the paper title says Vision and Language models.
* No parameter count included for classifier fine-tuning experiments. It is very much important to keep a track of the parameter count while proposing an alternative method for prompt tuning. An increase in parameters can increase the classification performance. How many parameters are tuned for classifier fine-tuning? By what number is it larger than CoOp? The parameter count needs to be the same for both the methods for an apples to apples comparison.
* What is the experimental setup for Figure-2? What are the epoch values?
* What is the justification for the experiment in Figure-2? Why is CoCoOp not run for multiple epochs?
* Experimental comparison with different prior works on prompt tuning is missing. Some examples include: Unsupervised Prompt Learning (UPL), Prompt Distribution Learning (ProDA), etc.

clarity,_quality,_novelty_and_reproducibility: I believe the paper is an analysis paper of existing prompt learning methods, but lacks clarity on the message it wants to convey, without proper justification of the claims, and comprehensive experimentation. No supplementary material and codes are provided for better reproducibility.
summary_of_the_review: All of my concerns are mentioned in the weaknesses section. I believe it is important to have clarity and proper experimental analysis which the paper lacks. I believe the paper needs to go through multiple iterations of revisions and rethinking on how to properly analyze and convey the findings for being fit for a publication at a venue like ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper focuses on analyzing the prompt learning paradigm in CLIP. The authors discuss some observations on the use of various types of hand-crafted prompts including class names, basic prompts (e.g., "a photo of a {CLASS}"), negative prompts (e.g., “this is not a photo of a {CLASS}”), and random prompts on 11 benchmark datasets. The highlighted observations include no major drop in performance using negative prompts. Additionally, the paper analyzes classifier fine-tuning as a faster alternative to prompt learning and shows superior performance compared to prompt learning. Experiments on few-shot classification, robustness to distribution shifts, and generalization to unseen classes are included using the CLIP model as the backbone.
strength_and_weaknesses: **Strengths**:

- The paper is well written and easy to follow.
- The paper has tried towards analyzing the effect of different compositional handcrafted prompts and studying the performance behavior on several downstream classification datasets.

**Weaknesses**:

- Although the paper states “Vision-Language Models” in its title, the experiments are only performed on CLIP models. It would be great to see similar findings for other vision-language models like DeCLIP, FILIP, CLOOB, CyCLIP, etc. Given that the paper is more of an analysis paper instead of a methodology paper, I would expect authors to verify their claims on other CLIP models like CLIP ViT variants beside CLIP ResNet. Does the size of models effect the conclusions presented in this paper? A more thorough comparison and anlysis should be included in the paper.
- It is interesting to observe that the four types of handcrafted templates used for zero-shot classification provide accuracies in a similar range, but this requires further extensive analysis to draw concrete conclusions. The {CLASS} token is very much important in classifying the images and is present in all the templates which provides the major information to the text encoder. The negative prompt has all the words the same as the revised prompt except “not”, obtaining accuracies in the similar range is expected as the text encoder is a language model and they lack compositionality which is well studied in NLP [a], and also the way CLIP has been pre-trained using the contrastive loss doesn’t guarantee compositionality. For the random prompts using random tokens and embeddings, the same argument of the presence of the {CLASS} token applies. Thus, a fair comparison between different prompts would be to remove the class token and then compare their performance on the downstream tasks. It might require some modifications at the output as done in "Learning to Decompose Visual Features with Latent Textual Prompts".
- The paper proposes classifier fine-tuning as a faster alternative to prompt learning and shows an average improvement of around 1.5%, but there is no parameter analysis (i.e., number of tunable parameters) provided for this experiment. It is known from the CoOp paper [b] that ensembling improves the performance, I assume that the authors fine-tuned the final projection layer of the text encoder which is of dimension 512x1024 for CLIP-ResNet-50, which has almost 64 times more parameters than learning 16 prompt tokens of 512 dimension. Therefore, it is very much essential to have a fair comparison with CoOp wrt the number of parameters each method uses. A fair comparison would be to use 64 learnable prompts and run CoOp for it. But, I believe from observing the results provided in CoOp [b], only 8 learnable prompts would be able to perform as good as the classifier fine-tuning. This argument applies to all the results in Figure-1 and Table-4.
- Additionally, the very motivation to learn continuous prompts for large vision and language models was to efficiently adapt the knowledge in the models to downstream tasks, which makes classifier fine-tuning contrasting to the motivation of prompt learning. I would like the authors to discuss on how classifier finetuning is a better alternative to prompt tuning for parameter efficient adaptation of large vision-language models for diverse downstream tasks.
- It would be good to add the average values across the datasets in table-2.
- No experimental comparison with other parameter efficient adaptation methods for CLIP, like CLIP-adapter [c], Tip-Adapter [d], UPL [e], PDL [f] etc. has been made in the paper. Authors should compare with these methods to verify the effectiveness of the proposed method over existing methods.
- Figure-2 is not clear. What are the epoch values corresponding to the points for CoOp?
- I feel that the section on optimality-generalization trade-off is very subjective and lacks proper experimental evaluation and theoretical support. In the beginning the authors ask the question - “why would the prompt learning methods or the improved conditional prompt learning methods have strong generalization ability?”, but get an answer - “Our assumption is that, due to the changed architecture, the improved prompt learning method maybe actually trying to find a better optimality-generalization trade-off.” I am confused on what it means. Additionally, I am not sure how the experiment of training CoOp for multiple epochs helps support the claims. Isn’t it known that training for a higher number of epochs makes networks susceptible to overfitting and is expected to lose generalizability? Also, why was only CoOp run for multiple epochs and not CoCoOp?

[a] Evaluating Compositionality of Sentence Representation Models: https://aclanthology.org/2020.repl4nlp-1.22.pdf

[b] Learning to Prompt for Vision-Language Models: https://arxiv.org/pdf/2109.01134.pdf

[c] CLIP-Adapter: Better Vision-Language Models with Feature Adapters: https://arxiv.org/pdf/2110.04544.pdf

[d] Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling: https://arxiv.org/pdf/2207.09519.pdf

[e] Unsupervised Prompt Learning for Vision-Language Models: https://arxiv.org/pdf/2204.03649.pdf

[f] Prompt Distribution Learning: https://arxiv.org/pdf/2205.03340.pdf




clarity,_quality,_novelty_and_reproducibility: Although the paper is well-written, the paper lacks clarity on what exactly it tries to study and mitigate. The experimental analysis is not extensive and lacks proper justification to them. 
summary_of_the_review: I think the identified problem is important but I’d like to rate the current submission as a clear rejection due to limited technical contributions and lack of convincing experiments. The paper needs significant changes including new experiments and possibly methodological improvements before being accepted to any major conference.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper offers a set of experiments around prompting methods for contrastive vision-language models.

In particular, the following experiments are presented:

1) Handcrafted & random prompts.
A number of simple templates are used to do image classification on 11 datasets on top of a CLIP pre-trained model. In addition, two types of random prompts are included in the comparison. The conclusion is that handcrafted prompts do well --some better than others--, even those that oppose our language logic (i.e. "this is not a photo of") seem to do a fairly decent job. Random prompts are inferior, but still offer non-trivial (sometimes even strong) zero-shot performance.

2) Prompt learning versus Classifier Fine-tuning.
Moving away from zero-shot evaluation into the realm of supervised tasks, the paper compares prompt learning (adding learnable parameters for the embeddings of the prompt tokens + finetuning only those) and classifier fine-tuning. The latter, if I understood it correctly, rather than learning the original embeddings for the prompting tokens, directly learns the final representation for each possible class. These will be compared against the final embedding for the input image, and pick the closest class as usual in contrastive learning. Basically, the pre-trained language tower is ignored here. The experiment focuses on a few-shot regime (between 1-shot and 16-shot). It concludes that classifier fine-tuning is almost always superior to prompt learning, and also computationally cheaper / faster.

3) Robustness to Distribution Shifts.
Following up on 2), prompt learning and classifier fine-tuning are evaluated in terms of robustness. Table 4 shows the results of finetuning on ImageNet and then evaluating on other related (out-of-distribution) ImageNet datasets, for a variety of vision towers. While it is not reasonable to expect Zero-shot CLIP to perform as well as finetuned methods, we see that the classifier fine-tuning overall works best (more robust, in this case).

4) Optimality-Generalization Trade-off.
This one I didn't fully get. It seems to me one model is trained with CoOp for varying amounts of epochs, and results show that it dominates (in terms of original finetuning and out of distribution evaluation) a new technique (CoCoOp) that uses an auxiliary network.
strength_and_weaknesses: Overall, the paper seems to be a set of interesting experiments, but with no clear takeaway. While I definitely think there's value in these experiments, I don't think the work is at the publication maturity level required by ICLR at the moment.

Questions.

- I don't think Linear Probe CLIP is formally defined or explained anywhere. What's the exact method here?
- In Table 4, what's the amount of few-shot examples per class used for finetuning?
- I think the classifier finetuning method isn't clearly explained, I was confused during a first reading. Maybe add a diagram or an example.
- I really struggled to understand Section 6. I think some major re-writing and experiment explanation is needed here.
- Figure 2 could benefit from a descriptive caption.

- Is the classifier finetuning method a new contribution? Or has this idea been tried elsewhere before? If it's a contribution, it should be made clear (and maybe even re-arrange the story and sections of the paper around it).
clarity,_quality,_novelty_and_reproducibility: The paper could benefit from some extra work on the writing and exposition side. Some of the methods are not formally and thoroughly presented, and some of the results (tables or plots) do not contain complete captions and lack key contextual information.

summary_of_the_review: The paper offers four interesting experiments around prompting (handcrafted, learned, finetuning "final class tokens", robustness, etc). While promising, I feel the paper still requires some work, including a more cohesive flow, and a bit more structured and unified takeaways.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a study on prompt learning, for example using negative or random prompts, and analyzes classifier fine-tuning as a faster and better alternative to prompt learning. While the analysis is interesting, and the paper is well written, all four reviewers raised major concerns regarding insufficient contributions, unfair comparisons with respect to number of parameters, and no clear takeaways. The author response was helpful, including for example new experiments on different vision-language models, but was not sufficient to eliminate the main reviewer's concerns.
justification_for_why_not_higher_score: All four reviewers recommend rejection. The AC agrees with this decision. In particular, some of the observations may not be so surprising as claimed by the authors (see the comments related to compositionality/negation by one of the reviewers). In addition, the paper would benefit from more analysis using the same number of parameters (extending the initial experiments provided in the author response)
Decision: Reject