Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper explores using a list of category descriptions as a better representation for image categorization using VLMs such as CLIP. Decomposing the category word into a set of related descriptions could bring a better textural representation when doing matching. In addition, these descriptions could provide interpretability.  The paper also demonstrates more benefits such as improvements in accuracy on ImageNet across distribution shifts; recognizing unseen concepts, mitigating bias.
strength_and_weaknesses: Strength:
1. This idea is quite novel and intuitive. 
2. The experiments have clearly demonstrated the benefits of the proposed methods. 

Weakness:
1. the aesthetics of the figures could be further improved. 
some potential questions were not answered
1. generalization: how this generalizes to other tools other than CLIP and GPT?
2. how do you weight the prob. of a set of features linked with "or" logic? for example, in the appendix, the lemur could be "black, grey, white, brown, or red-brown", do you just sum these prob up? A better way might be using max?

clarity,_quality,_novelty_and_reproducibility: Novelty: to my best knowledge, this is an interesting first exploration of using off-the-shelf models without any training to improve classification tasks. Even though there are a few questions unanswered, this paper overall suggests and interesting and important direction.
Quality: it shows clear improvements over the baseline, but I think this is method's performance could be further improved. 
Reproducibility: (this is from the paper): 
We use CLIP as our vision-language model and GPT-3 as our large language model, both of which can be queried by anyone – CLIP having weights available, and GPT-3 having a public API. Sufficient details to reproduce the method can be found in Section 2 (for inference) as well as Appendices A (for querying language models) and B (for editability). We will release the data for the editability and adaptability experiments where the appropriate licenses permit and sufficient de-identification is possible. We will also release code upon publication.

summary_of_the_review: This paper proposes a simple and yet effective method to improve current clip image categorization quality.  The results are relatively preliminary but it proposes a novel direction and clearly demonstrated the benefits (interpretability, removing biases, etc.)

This paper might be stronger if it:
1. demonstrates if this method is generalizable to other big models
2. explore in more systematic treatment on how to combine descriptors in addition to simply adding them. (Eq.1)
3. provide more in-depth analysis on the potential different results on categories at different levels in a hierarchy  (e.g., dog vs husky)
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper propose an image recognition paradigm named "classification by description". To be specifically, instead of matching an image with its categorical description, the method first queries a large language model to obtain some textual descriptions for each category and base the recognition decision on these descriptors through consensus. In addition to interpretability, the method boosts recognition accuracy on the ImageNet and ImageNetV2. The paradigm is also able to recognize novel concepts and mitigate bias by descriptor editing. 
strength_and_weaknesses: ### Pros:

+ It is a conceptually novel to obtain text descriptors for image recognition by querying large language models.

+ The classification via text description naturally provides explanations for decision of objection recognition (Sec 3.1) and can potentially reduce existing biases in existing methods (Sec 3.3).

### Cons: 

- The performance from the original CLIP is a weak baseline which does not use prompt ensembles. To obtain the text descriptors from GPT-3, we still needs to prompt and it is also mentioned in Appendix A that "the list formatting becomes more reliable when one
or two examples of desired output are provided" which needs additional manual efforts. That's why we need to allow the CLIP baseline to use better-designed prompts as well.

- What's the unit of values in the bar plots (Fig 1, 4)? From Eq (1), if $\phi(d, x)$ refers to a log probability, then the score would be in the range of $(-\infty, 0]$, no?

- The quantitative experiment is weak. I would classify Sec 3.2-3.4 as qualitative analysis since they are not evaluated on well-recognized benchmarks. I think it would strengthen the paper if there is more quantitative experiment other than image classification. Maybe long-tail object detection on LVIS or so?

- I didn't fully get the statement "Our approach does not fit on the ImageNet training dataset"? Do you mean that your method is not trained on ImageNet train set (equivalent to a zero-shot setting)?
clarity,_quality,_novelty_and_reproducibility: + Clarity: The paper is in general well written and the given examples are illustrative. I am unsure how to interpret the values of predicted score though (See weaknesses).

+ Quality: The analysis is great. The experiments need more quantitative results (See weaknesses).

+ Novelty: The paper is novel. Besides improving the CLIP baseline on public benchmarks (ImageNet, ImageNet-V2 and CUB), it also presents new applications such as retrieving novel classes and mitigating biases by editing text description.

+ Reproducibility: The method is conceptually simple and the CLIP model is publicly available. But it would be great if the authors can share the list of text descriptions (since GPT-3 is not fully publicly available).
summary_of_the_review: This is generally a well-written paper with a novel approach to achieve both interpretability and performance boost. It would be better if there is more quantitative experiment to support.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a simple but effective approach to zero-shot learning and explainability for image classification. The title of the paper says it all, but for completeness here is a summary. The main idea is to query a large language model, in particular GPT-3, to obtain multiple short descriptions about a class. These short descriptions are then projected into a joint embedding space which was learned to encourage visual grounding via multi-modal training pairs (ie CLIP). Then when an image from this class is encountered it is projected to the joint embedding space and its similarity to each class is computed by aggregating the similarity of the image's encoding to each encoding of the class's short descriptions. The image is then classified as the class that gives the highest overall similarity score. A by-product of this system is that the short descriptions with the highest scores then give a ready made interpretable explanation of why the system made its decision.

The contributions of the paper:

1) The paper demonstrates, and adds to the evidence, that the publicly available large language and multi-modal models, trained on large scale data such as CLIP, can be leveraged via descriptions (and prompt engineering) to perform explanability without a drop in classification performance and zero-shot learning. The interesting part is that the amount of manual prompt engineering can be minimized and can be performed by GPT-3 instead of by a human.

2) The paper raises interesting questions to think about how to best leverage the large scale foundational models in vision and language to perform zero-shot learning especially in a world with ambiguities and that is evolving constantly and how best to transfer knowledge from large scale language models to visual models.
strength_and_weaknesses: Strengths:

- The paper is generally well written and a fun read. It has a interesting experiments and shows that interesting results in explainability can be achieved almost for free from manual intervention by harnessing GPT-3 descriptions and models in the mould of CLIP. 

- A nice set of interesting experiments showing that the CLIP results on ImageNet can be improved upon from text descriptions automatically obtained from GPT-3 as opposed to the manually specified in the original CLIP paper. There is just one reservation here in that it is not clear which text prompts were used for the CLIP results in table 1. 

Weaknesses:

- The original CLIP paper has already demonstrated that prompt engineering can have a significant impact on the zero-shot recognition results. So this finding in the paper is not new.


- The experimental evidence presented is nice. But some of these experiments are a little sparse and a little artificial. This is especially true for the "acquiring and utilizing novel information" learning experiments. It is fun to see that "Wordle" and "Ever Given" images can be recognised from their GPT-3 descriptions but it is obviously far from a complete study of the model's capabilities on this front and it is a bit unclear what it demonstrates. "Ever Given" is a proper noun and the name of a specific container ship. It is not surprising that CLIP cannot do image retrieval based on a proper noun which was generally unknown at the time of training CLIP. 

   * The paper mentions that very little in the way of prompt engineering is done. The engineering that is performed is mainly done to ensure a list of descriptions is created by GPT-3. How sensitive are the results to the temperature and max token length used in the language model? Were the default values used or were they set using some form of hyper-parameter sweep.
   

- There are some technical details missing or brushed over. So for example:

   * In equation (1) how exactly is \phi(d, x) computed? Is a cos similarity used, does it use a softmax to compute the probabilities, etc...?
   
   * For table 1 for the CLIP column what is the text prompt used to acquire the classification? Is it just a classname or does it use the prompt "A photo of a {label}" as suggested in the CLIP paper as the basic prompt that boosts performance over just using the classname.

   * In figure 1 and figure 4 there are bar charts showing the compatibility between a text description and an image. What exactly are the numbers being displayed? Also the horizontal scaling seems to be different between the different examples and thus gives an inconsistent visual cue via the length of the bars. 

- Figure 4 gives some qualitative results for the explainability for particular images. But it would be nice to have some more quantitative measures of how well the explainability is doing or thoughts on how to do this or show which explanations are most useful for each class etc... Also would some calibration be required to decide which descriptions are actually "valid/present" for a given image.
clarity,_quality,_novelty_and_reproducibility: The paper is overall an easy and enjoyable read. As highlighted above there a few missing details which might make exact reproducilibilty hard, but hopefully the idea is resistant to the exact method of similarity etc.


The main novelty of the work is in using GPT-3 descriptions of a class for both defining it and then subsequently using them to explain the content of the image. 
summary_of_the_review: My recommendation is based on the fact that I think the results achieved are interesting and stimulating for a reader with an interest in zero-shot learning. There are some weaknesses in the presentation of the technical and some of the experimental evidence seem more like tidbits as opposed to fully fleshed out ideas. However, I think overall I would be in favour of publication with some cleaning up of the presentation.  

(
Typos spotted along the way

* Page 5 last line: valiation --> validation
* In section 4 on the 2nd line of the 2nd paragraph there is an empty reference. 
)

Response to the authors' feedback:
I would like to thank the authors for their responses and the extra information and results provided. They have addresses my concerns I have now upgraded my rating accordingly. Well done on an interesting and fun paper!
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a simple idea to improve zero-shot image classification using CLIP-like models. It uses a pre-trained LLM (GPT-3) to automatically generate text descriptors for visual categories. The text descriptors are then used by CLIP to compute image-text similarities as prediction scores for each category. The method shows improved zero-shot classification performance on ImageNet and CUB datasets over the naive CLIP prompt.
strength_and_weaknesses: Strengths:
- The proposed idea is simple and well-executed. The paper is also well-written.
- Visual-prompt-engineering can be a heuristic and tedious process. It is refreshing to see that LLMs can help to automate this process. The prompt for GPT-3 is also well-designed to guide the model in generating visual descriptors. 
- The method provides interesting additional capabilities such as explainability and novel category classification.

Weakness:
- Could the proposed prompt-generation method generalize to other domains other than common objects and animals? It is necessary to evaluate on more datasets as in the CLIP paper, such as food/scene/satellite/medical images. In order for the method to be practical, it should at least not be harmful on the more difficult domains.
- It should be made clearer (rather than in the footnote) that the CLIP results in Table 1 uses a single base prompt, which I assume is "a photo of {class_name}"? Does the proposed method include the same base prompt with class_name?
- It is suggested to include the result for the hand-crafted CLIP prompts in Table 1.
- There exist other methods to construct visual descriptors that can offer similar advantages such as explainability. For example, using WordNet (K-LITE) or wikipedia articles [a]. It would be good to see some comparison between descriptors generated by GPT-3 and descriptors retrieved from knowledge base.

[a] Large-Scale Zero-Shot Image Classification from Rich and Diverse Textual Descriptions. Bujwid and Sullivan. ACL 2021.

Minor suggestions:
- One of the entry in Table 1 has an inconsistent number of digits. 


clarity,_quality,_novelty_and_reproducibility: - The proposed method is described clearly. Due to its simplicity, it should be easy to reproduce the reported result. The authors also promise to release code upon acceptance.
- The novelty of the propose method is limited. The proposed method is more like a "clever trick" rather than a fundamental technical innovation.
summary_of_the_review: This paper opens up an interesting new direction that uses LLMs to generate visual descriptors for CLIP-like models. I have two major concerns:
- The ability of the proposed method to generalize to other domains.
- Comparison with CLIP's prompt is not entirely clear. Comparison with knowledge-based method could be better.

I will happily raise more score if my concerns can be cleared.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents Classification by Description, where a vision and language model for classification is asked to verify typical descriptions for object categories along with an overall category name. Using these descriptors results in a more robust model with improved ImageNet results and also a highly interpretable system.

Four reviewers provided reviews for this paper. Overall they were positive about this paper at the initial review. They enjoyed reading the paper, found the idea to be intuitive, simple and effective, found the idea of complementing image classification with description verification as novel and the presented experiments to be clear. Their main concerns revolved around the presentation of some of the material and requested additional experiments. The rebuttal addressed these points very well, which caused multiple reviewers to respond positively and raise their ratings. They are now in strong favor of this paper. Given the reviews, rebuttal, discussion, score and my own reading of the work to be interesting an novel, I am recommending an accept.
justification_for_why_not_lower_score: I found the work to be very interesting. The idea is so simple, but intuitive and effective. It presents a new way to interact and understand systems, even for simple tasks like classification. An oral presentation at the conference would enable this idea and work to reach a broad audience. This paper can inspire future work in vision that can lead to similarly interpretable and effective models for more tasks.
The reviewers and scores were also very positive.

I am recommending an Oral, but I would also be in favor of a spotlight. I do feel that the idea and paper deserves more than a poster.
Decision: Accept: notable-top-5%