Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors propose to utilize the label hierarchies to boost the performance of CLIP for zero shot classification. The main steps include the generation of the subclasses for each class by either using the GT label hierarchies or by querying GPT-3, then conduct the CLIP  via these sub-classes, and finally map the sub-classes back to their parents. The performance gain has been observed on the benchmark datasets.
strength_and_weaknesses: Strength:
The paper is generally well-written with a clear motivation and decent performance gain.

Weakness:
The main concern would be the proposed model is essentially an ensemble of sub-class based CLIP models, though authors address this concern via comparisons against linear average approach, the novelty is still somewhat limited.
clarity,_quality,_novelty_and_reproducibility: The organization of the paper and the overall writing is clear. And one should be able to reproduce with the firm understanding of the CLIP model.
summary_of_the_review: Overall, the paper is well-written, and easy to follow, yet the main concern as expressed in the weakness part is the novelty, as it resembles a lot with an ensemble model.

Moreover, if the hierarchical structure does not explicitly exist in the label vocabulary, how would the proposed model handle it, namely, you cannot rely on the ready-to-go GT structure or query it via the GPT model. It is somewhat unfair if the compared models can also utilize these structures to boost their performance. Authors are suggested to clarify their proposed model in a more general sense.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method to improve predictions of the CLIP model by utilizing a potential structure/hierarchy among classes. Instead of generating predictions on the target classes only, the authors propose to generate predictions on a set of all target labels’ subclasses (which are more granular) and use predictions in subclass space to decide which target class (superclass) to output.  
For finding subclasses the authors use either use an existing hierarchy of labels or generate them by prompting a GPT-3 model.
strength_and_weaknesses: Strengths:

- (S1) A practical and easy-to-implement/utilize method
- (S2) In the case of existing class hierarchy, the experiments indicate a very significant improvement over not relying on that class hierarchy (although see W2)
- (S2) The experiments cover w/ and w/o hierarchical explicit information (although see W1)
    

Weaknesses:
- (W1) The paper doesn’t clearly show whether the proposed approach is valid/useful only on datasets where some hierarchical structure between classes exist (regardless whether explicitly present or not) or is something that would work on any generic problem/datasets. The authors do address in their experiments two scenarios: one with hierarchical information provided explicitly and another with a hierarchy generated from a GPT model. However, a good class structure/hierarchy may or may not exist even if it is not explicitly provided or utilized. Taking ImageNet as an example - there are many animal/plant species with a very deep hierarchy, many closely related classes on one hand, but on the other hand, some classes have a very shallow hierarchy and few only loosely related classes (I would suppose maybe classes like “cliff” or “traffic sign”?). The authors however seem to choose datasets where one could expect some hierarchy to exist, even if not explicitly present. How would the method behave on datasets like e.g. StanfordCars where all classes are somewhat similar and it’s unclear if some meaningful hierarchy exists, or even the whole ImageNet where maybe there are groups of classes with a nice hierarchy/structure and groups of classes where such hierarchy might not exist?
- (W2) The authors present results for their method but no comparisons to any alternative approaches. One would expect at least some simple baselines that utilize the hierarchical structure of labels.
- (W3) On many datasets the improvement in accuracy is very significant, but on some other, like “living17” or “fruits-360” there’s a relatively much smaller improvement. This is not explained/discussed by the authors - is it something related to some properties of the hierarchies?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and proposes a simple, yet conceptually novel approach to utilize label hierarchy.
The authors provide code and most implementation details seem to be available.
summary_of_the_review: Not so clear how general or practically useful the method is: how specific the results are to datasets/problems with some class structure (whether explicit or implicit). The most significant improvements are when utilizing an existing hierarchical structure of labels but it’s unclear how much of that improvement is specific to the proposed method as no alternative approaches or simple baselines utilizing such information are considered.

**EDIT: Updated the score to lean towards acceptance. The authors' comments are generally convincing and the updated version of the paper makes the scope of the contributions clearer.
Also, the additional results provide more insight into the model's performance and in which scenarios it is expected to work well.**
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies zero-shot image classification with class hierarchy.
A Classification with Hierarchical Label sets (CHiLS) model is proposed to improved classical CLIP model.
This model leverages predefined subclasses, and perform CLIP on them first to obtain a set of class embeddings.
These subclass embeddings are aggregated together to form the embedding of class of interests that facilitate zero-shot image classification.


strength_and_weaknesses: 
- The outstanding problem of this paper is that it is of not self-contained. It lacks of proper definition and review of referred models, terminologies, and problem setup. For example, "CLIP", "prompt", "open vocabulary models".

- The idea of dividing a super class into a set of subclass is not new, and therefore the contribution is not significant enough.

- The sensitivity of the proposed model on the different levels of granularity of the class hierarchy is unclear.

- Lacks of proper comparisons with related works in line of zero-shot image classification.
clarity,_quality,_novelty_and_reproducibility: In terms of clarity, this paper can be improved further by providing proper introduction of the problem formulation, related works. 

The proposed method looks pretty straightforward. 
While it is interesting to see the improvement of performance, it is only a modest incremental work over the arts.
summary_of_the_review: The paper improves the previous arts CLIP by leveraging class hierarchy, which can be seen as a modest incremental work.
Unfortunately, the reviewer does not find it is novel enough.
Besides, the writing and empirical study can be further improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper provides an algorithm to improve zero-shot open-vocabulary classifiers by better prompting them. The authors propose to (1) create sub-classes for each parent classification category using existing (human-created) or inferred (GPT-3 generated) label hierarchies, (2) perform standard zero-shot classification on the sub-classes, and (3) aggregate the results to associate probabilities with the parent category. The paper provides empirical evaluation of the method on various datasets and additionally ablates key design decisions related to their use of labels, GPT-3, and aggregation strategy.
strength_and_weaknesses: **Strengths**

*S1.* The work is well motivated.
 
*S2.* The work seems novel. I have only seen GPT-3 used in zero-shot open-vocab classification in concurrent work and consider this to be novel.

*S3.* The method is general for classification problems and does not require additional training.

*S4.* The ablations test major design decisions and shed light on when a practitioner may or may not want to make similar decisions. 

*S5.* Method is simple and easy to implement.

*S6.* The introduction and method section are very well written.

**Weaknesses**

*W1.* Related work could be more comprehensive and the paper could be better situated in the literature. More specifically, here are some suggestions for references. I suggest doing a more comprehensive literature review of recent work.

Transfer learning:
* LiT: tuning text towers while keeping language tower fixed (https://arxiv.org/abs/2111.07991)
* Visual prompt tuning: tuning learnable visual prompt (https://arxiv.org/abs/2203.12119)
* Model Soups: ensembling many fine-tuned CLIP models in weight space (https://arxiv.org/abs/2203.05482)
* Patching CLIP models on downstream tasks while maintaining zero-shot performance on tasks where CLIP is already successful (https://arxiv.org/abs/2208.05592)
* CLIP-CL: similar to Model Patching above (https://arxiv.org/abs/2207.09248)
* ...

Zero-shot prediction:
* CLIP-ViL: showing the limitation of ZS CLIP in VQA-like settings (https://arxiv.org/abs/2107.06383)
* CoW: showing how CLIP can be adapted to do object navigation without additional training (https://arxiv.org/abs/2203.10421)
* ...

Additionally, I suggest adding a section on hierarchical classification.

*W2.* The paper states: "Here, we reweight each set of subclass probabilities by its superclass probability. In this way, we can attempt to avoid the behavior in which CLIP makes an incorrect subclass prediction despite a confident and correct superclass prediction, and thus bias our model to never do worse than the raw superclass predictions."

While this is maybe meant to provide intuition for the method, it reads like a claim that is not supported. I suggest making it clear that this is intuition and not some mathematical/provable property of the algorithm related to "bias".

*W3.* How does the proposed method perform on ImageNet? Without ImageNet evaluation it becomes hard to compare the proposed method to other methods.

*W4.* It seems many of the datasets are chosen because it is easy to construct class hierarchies. However, the method is presented as a general purpose zero-shot image classification improvement over CLIP. To be convinced of this, I would be interested to see results on more datasets. Please consider Cars, DTD, EuroSAT, GTSRB, KITTI, MNIST, RESISC45, SUN397, which Ilharco et al. (https://arxiv.org/abs/2208.05592) use because they are tasks ZS CLIP is known to struggle on. Note: dataloaders for these datasets can be found [here](https://github.com/mlfoundations/patching/tree/main/src/datasets).

*W5.* The main experimental setup has baselines that may not be fair points of comparison. Specifically, not all datasets in the original CLIP paper use the ImageNet prompt templates. For example, CIFAR-10/Food101 use different prompts (see [here](https://github.com/openai/CLIP/blob/e184f608c5d5e58165682f7c332c3a8b4c1545f2/data/prompts.md)). What are the deltas between the 75 prompt set and the set that OpenAI provides? For Food101 specifically, there seems to be a discrepancy of ~2% between the ZS number reported in the manuscript and the L/14-336px number reported in the CLIP paper in Tab 11. For datasets that OpenAI did not evaluate in their original paper, some prompt engineering on a val set or some validation that ImageNet prompts are reasonable seems needed.
 
*W6.* It seems that the experimental setup is sufficiently different in cases where the hierarchy exists and when it does not for me to compare the performance between col 2 and 3 in Table 1. For example, in the *GPT-3 map* the authors include the superclass label in the label set, while in *Existing Map* they do not. It would be good to standardize the algorithm being compared in the two settings. Another idea is to provide the full results in the appendix (i.e., *GPT-3 map* w/ superclass labels, *GPT-3 map* w/o superclass labels, *Existing map* w/ superclass labels, *Existing map* w/o superclass labels).

*W7.* I am not able to completely understand the experimental setup for “Noisy Available Hierarchies” from the text provided. Consider adding an appendix for that section to give more details on how the subclasses are constructed from the ImageNet hierarchy.

*W8.* How do things look if the label set size is 1 or 100 (i.e., m=1 or m=100)? Doe these more extreme values of m affect the performance? At least m=1 should be worse than zero-shot, which seems like a valuable bases of comparison.

*W9.* Some visualizations of cases where the vanilla zero-shot and CHiLS models disagree could be helpful to provide intuition.

*W10.* I am not clear on the linear class ensemble experiment from reading the text. Is the following correct? For a single class, loop over all subclasses and ImageNet prompt templates computing the CLIP text features. Average all of these features to represent the single class as a feature. Repeat for all classes to get a zero-shot classification head.

*W11.* The method involves expanding the zero-shot head at test time. There is a compute and memory overhead associated with this that may limit the scalability of the method when many classes or subclasses are targeted. For example, for the 1000 ImageNet classes, with m=10, the instantiated head would have 10k classes.

**Minor**

*M1.* The Radford et al. 2021 caption example is “a photo of a {}.” not “a photo of a {}” as presented in the manuscript. The difference is the period at the end of the prompt, which may make a difference in downstream performance.

*M2.* This is a relevant reference when discussing CLIP confidence in the method section: https://arxiv.org/abs/2106.07998

*M3.* For Sec. 4.4 is there some theory as to why weighting is necessary in one case and not in the other?

clarity,_quality,_novelty_and_reproducibility: * The use of GPT-3 in zero-shot open-vocabulary image classification is novel (also appearing in concurrent work as the authors recognize).

* While I have not seen hierarchical classification methods applied in zero-shot image classification, these kinds of techniques are well studied in the literature in more classical ML settings. The authors can improve the manuscript by positioning their hierarchical classification algorithm relative to others. However, the authors do not claim to present hierarchical classification in-and-of-itself as novel.

* I have a minor concern about reproducibility given that the GPT-3 component of the work uses a temperature parameter of 0.7 and hence introduces some randomness into the results. However, the authors release all generated labels and hence the numbers in the paper should be reproducible for the datasets presented.

* Some of the experiments require better explanation and could benefit from appendices giving more experimental details (see W7, W10).

* The introduction and method section are very well written.
summary_of_the_review: The paper provides a method to prompt open-vocabulary models for zero-shot image classification, which leverages class hierarchies. It is straightforward to implement and the authors provide some empirical evidence that their method performs significantly better than the baseline, especially when hierarchies are known for a dataset.

However, I currently recommend weak rejection of the manuscript. I am most concerned about the lack of benchmarking on ImageNet (*W3.*), the performance of the method on at least a few more standard classification datasets (*W4.*), the fairness of the zero-shot baselines (*W5.*), and the clarity of the presentation associated with some of the ablations studies (*W7.*, *W10.*).

I am willing to revisit my evaluation during the rebuttal/discussion period.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an approach to improve the performance of CLIP for zero-shot image classification by utilizing class hierarchies (or using GPT-3 to access such hierarchies) to make predictions for all subclasses of a class, and then mapping the subclass predictions back to the parent class to get a final prediction.

The key advantage of the approach is that it is a simple adaptation of existing methods that can lead to performance improvements on some image-recognition problems. The main problem with the paper is that it remains unclear which image-recognition problems: in what scenarios does the approach help and in which scenarios does it not help (or even hurt)? 

The initial version of the paper somewhat overstated how generally applicable the method is. While that issue appears to have been addressed in the revised paper, the underlying question still remains. Answering this question likely requires more in-depth analysis, for example, on the ImageNet dataset. (One could imagine doing experiments on different versions of the ImageNet dataset in which target classes are chosen at different levels from the WordNet hierarchy as a way to facilitate more systematic analysis.) Doing such analyses is important for an ICLR audience to understand the applicability of the proposed method.
justification_for_why_not_higher_score: The paper is missing analyses that are important for readers to understand when they should use the proposed method.
Decision: Reject