Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Open vocabulary models require natural language prompts as an intermediate medium between category and visual content, thus it is important to choose a good prompt. In this paper, the authors study how to use a pretrained language model to help generate better language prompts for open vocabulary tasks. The authors feed the categories with language prompt templates to an LLM and use the output of the LLM as the image prompts. The authors show that the prompts generated by their method can achieve better performance on zero-shot image classification benchmarks. 
strength_and_weaknesses: Strength:
- The paper is well written. The method is intuitive and the performance gain is good.
- The authors did different ablation studies to show how different design choices would affect performance including the number of prompts, diversity of prompts, model size, etc. These ablations are helpful for the audience to understand the full picture of this method.

Weaknesses:
- Although the general idea is good, the authors only consider one task in this paper which is zero-shot image classification. It is more interesting for tasks like open vocabulary segmentation/detection etc. This method can even be used for image caption retrieval where LLM is used to paraphrase the caption. It is more convincing that the proposed method can universally work on different tasks.
Even for imagenet, it is more interesting to see the performance on the full imagenet21k instead of the 1k set. In addition, original CLIP is evaluated on different relatives of Imagenet like imagenet-v2, imagenet-R etc. which would also be good to have in this paper.
- It would be better if there can be a clear analysis or explanation of why the standard prompts and CuPL prompts have different preferences as shown in figure 7. It is a really interesting phenomenon to me, and it may also reveal some hints on how to improve CuPL.
clarity,_quality,_novelty_and_reproducibility: Clarity questions:
How are “hyperparameters” like the number of prompts and which specific prompts to choose decided? Is there a “validation set”?
Is there qualitative or anecdotal analysis of how “diverse” the generated image prompts are after increasing the temperature? For example, would higher temperatures also “hurt” by introducing more incorrect knowledge?

Minor comments:
Figure 6: CuPL outperforms the baseline even with just three hand-written sentence. Should be one hand-written sentence.
Also, how many prompts are there for Figure 6 left figure? If it is 10, it means that 10 CuPL image prompts can outperform 80 standard image prompts which is less than 25 in the caption.

The paper is novel and easy to reproduce.
summary_of_the_review: The authors propose a novel method that is simple and has the potential to apply to many different tasks. However, the authors did not show that this method is widely applicable which makes the paper weaker. My decision is borderline accept.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents combines CLIP with large language models like GPT3 to create customized prompts for zero-shot image classification. Specifically authors leverage the knowledge contained in LLMs in order to generate many descriptive sentences that are customized for each object category for prompting CLIP models. Experiments on 15 downstream classification datasets show the effectiveness of the proposed method over handcrafted prompts that were used in the original CLIP adaptation.
strength_and_weaknesses: **Strengths**: 

- The paper is well-written and easy to follow.
- The idea is very simple and performs reasonably well on standard classification datasets.

**Weaknesses**:

- The technical novelty of the paper is very limited. It simply combines GPT3 with CLIP without a clear motivation. It is not surprising that output of a large language model will have better descriptions/knowledge compared to vanilla prompt engineering like "a photo of a". So, what is the main contribution of the paper need to be clearly discussed in the paper. Why someone will adopt this approach given that it often requires higher computational resources for inferencing a GPT3 model (and sometime not accessible for all)?

- Although the proposed framework uses a strong LLM like GPT3, the zero-shot classification performance on downstream datasets are very minimal. Training a continuous prompt, e.g., CoOp in Learning to Prompt for Vision-Language Models using 1-2 labeled samples performs significantly better than hand engineered prompts. It is often very practical to have 1-2 labeled samples per class inn many applications.  It is not clear why learning a single prompt using few labeled samples is not practical but the use of GPT3 can be considered practical for zero-shot classification? A thorough comparison and discussion with other prompt learning methods should be included in the paper.

- How does the proposed framework with only one prompt compare to the baseline that uses prompt like "a photo of a" (no ensemble of prompts)? This is a very important experiment to verify the effectiveness of the proposed framework in fair manner.

- What other generative LLMs besides GPT3 can be used and how are they comparable to the current one?

- The experiments only show performance on CLIP, while there exists many other recent vision-langauge models, e.g., DeCLIP, FILIP, CLOOB, CyCLIP etc. Authors should perform experiments on a variety of V&L models to demonstrate the generalizability of the proposed method for generating better prompts.

- Like original CLIP, CuPL still requires handcrafted prompts for LLM which is suboptimal. Identifying the right hand-crafted prompt is a non-trivial task, which often requires significant amount of time and domain-specific heuristics.
clarity,_quality,_novelty_and_reproducibility: The paper is well written with clear presentation of the technical details, figures and tables. However, as discussed above, the novelty of the work is very limited. Also, how the proposed method can be applied for classification on new datasets without significant manual engineering and access to GPT3 is not clear.
summary_of_the_review: I’d like to rate the current submission as a clear rejection due to very limited technical novelty and lack of convincing experiments. Despite all the changes, I still feel the paper will not have enough contributions to be accepted for ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to generate prompts for zero-shot learning inference with CLIP by querying the large language model like GPT. The results show that the class prompts generated from GPT improves the zero-shot classification performance across many datasets. 
strength_and_weaknesses: Strength

+It is interesting to generate prompts from LLM. Querying the description of an object class from LLM makes a lot sense because LLM is learned from large text corpus, covering knowledge about most of vocabularies. 

Weakness

-I found this paper has little technical novelty. The proposed method generates prompts for each class from the outputs of a LLM i.e., GPT-3 and then use those prompts for zero-shot inference with the pretrained CLIP, which is straightforward and expected to achieve some improvement. 

-The author's main contribution is querying GPT but the design decision of why a particular prompt is chosen to query GPT3 for CuPL base and full are not discussed or ablated.

-To my knowledge, using GPT-3 is not entirely free, which makes the method less appealing. Does this approach work well with other free LLM?

-In Fig. 5, increasing the temperature of LLM also increases the randomness of the outputs. What is the standard deviation at each temperature?

-The performance improvement are very minor and comes with a decrease in performance on other similar classes as shown in figure 7. This insight indicates that such a simple query setup is suboptimal for finegrained setting. The authors have not tried to mitigate this basic drawback. 

-Detailed analysis on what makes the GPT3 prompts work better than human designed ones is missing when it is the main contribution of the work. For example in figure 5 we see that GPT3 prompts only surpass generic hand designed prompts at ~0.56 temperature. Why is this so? How do the outputs of the language model change with this temperature setting and what contributes to this change. If the method is this sensitive to temperature value, it adds to additional labelling cost of running the GPT3 model to ablate over the temperature value. How sensitive is this temperature value on different datasets? This needs to be analysed and discussed in light of labelling cost of generic human prompts.


clarity,_quality,_novelty_and_reproducibility: This paper is written well and easy to understand because the method is rather simple. It might not be easy to reproduce the results because this requires users to query GPT-3 many times, which is not free.
summary_of_the_review: Due to the weakness mentioned above, I think this paper does not have sufficient contributions for ICLR. I would recommend a reject and suggest the authors submit it to a workshop.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes to replace hand-written prompts used in CLIP style models with GPT-generated prompts for zero-shot image classification. The authors show that their approach (CuPL) of using LLMs to generate diverse text prompts, can improve classification accuracy while removing the need for hand-crafted prompts.
strength_and_weaknesses: Strengths
1. The approach is simple and effective. They show that by using the knowledge baked into LLMs like GPT can be used to improve zero-shot classification accuracy. I think this contribution is neat! 

2. While some analysis is still missing in the paper (See weaknesses), I think the authors did a thorough empirical evaluation of their approach. I appreciated the author's effort on showing how different model sizes, number of LLM prompts, number of image prompts, LLM temperature affect performance. Per class accuracy difference (Figure 7) was also quite informative to look at! 

Weaknesses: 
1. Higher accuracy yes, but it doesn't remove the need for hand-designed prompts. As the authors mention in Figure 7, CuPL leads to significant accuracy boosts for certain classes (like mushrooms), but doesn't do well in other classes (canoe). This kinda means that hand-designed prompts can't be completely replaced because they bake in domain knowledge about the dataset. 

2. I am curious how the GPT prompts are helping improve classification accuracy. It would be nice to sort of look at which words in the image prompt (maybe by looking at attention distribution) are important. Doing this both for standard as well as LLM based image prompts might help understand why there is such a drastic jump ( +/- ~40%) in accuracy for different classes.  
    - I found the 40% accuracy jump for `mushroom' quite surprising. My first intuition was that the proposed approach will work better for classes that have distinct parts (cars, scooters, chair, etc). I wonder why LLM based prompt helped for a category like a mushroom?

3. Diversity Analysis (Figure 5) done on single LLM-prompt. It's quite possible that different LLM prompts, produces the same Image prompt. So an analysis of how different all the image prompts generated from different LLM prompts for a single image are would be nice. 

4. In figure 6, 
    - how did they choose which prompt to select when using fewer prompts. For instance, when only using 1 LLM prompt, which prompt did they use? 
    - I also didn't understand the caption for Figure 6 (left). The caption says that CuPL outperforms the baseline with just 3 hand-written sentences. But in the figure, CuPL is outperforming the baseline even with 1 LLM prompt. Can the authors please clarify? 

5. May I suggest adding a baseline that only uses 'a photo of a {}' prompt? The proposed approach lies somewhere in the spectrum of no-effort (using one standard prompt for everything), and high-effort (using standard hand-written prompts). We see where it lies with respect to high-effort. It'd be great to also see how the proposed approach compares to a _no-effort_ baseline.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. I think using GPT for generating prompts for CLIP style models is unique. The experiments are quite exhaustive demonstrating the effectiveness of the approach. The authors do a good job discussing each detail, and the work seems easy to reproduce.
summary_of_the_review: Overall, I like the paper. The paper presents a simple contribution and shows its effectiveness through thorough empirical evaluation. However, some analysis is still missing from the paper to build a better understanding of why the proposed approach is working. After reading the paper, apart from empirical evidence, there is little else to see the benefits of using the proposed approach. I think providing some of the analysis mentioned in my review will make the paper stronger and more insightful.

Update after rebuttal: I thank the authors to provide answers to my questions and incorporating all feedback! I think the paper presents a simple to understand/implement/use contribution that is very effective! I quite like the paper and I am bumping my rating to 8 to reflect that.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes the use of GPT-3 to generate a prompt for a zero-shot object classification method. Upon the request of the reviewers, the authors add further results with other language models. While the proposed method is simple and effective in some settings, the authors question whether the technical contribution is sufficient, whether the use of GPT-3 (which is not free) is warranted (and results with other LMs are not as good), and practicality (given that prompts can be learned with a small amount of data).
justification_for_why_not_higher_score: Two reviewers recommend rejection; one recommends acceptance but raises several concerns.
Decision: Reject