CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets
Authors: Zachary Novack
Saurabh Garg
Zachary Chase Lipton
Venue: Submitted to ICLR 2023 (2023)
Keywords: open vocabulary models
CLIP
zero-shot learning
zero-shot image classification

Abstract: Open vocabulary models (e.g. CLIP) have shown strong performance on zeroshot classification through their ability generate embeddings for each class based on their (natural language) names. Prior work has focused on improving the accuracy of these models through prompt engineering or by incorporating a small amount of labeled downstream data (via finetuning). In this paper, we propose Classification with Hierarchical Label Sets (or CHiLS), an alternative strategy that proceeds in three steps: (i) for each class, produce a set of subclasses, using either existing label hierarchies or by querying GPT-3; (ii) perform the standard zero-shot CLIP procedure as though these subclasses were the labels of interest; (iii) map the predicted subclass back to its parent to produce the final prediction. Across numerous datasets, CHiLS leads to improved accuracy yielding gains of over 30% in situations where known hierarchies are available and more modest gains when they are not. CHiLS is simple to implement within existing CLIP pipelines and requires no additional training cost.

Under review as a conference paper at ICLR 2023
CHILS: ZERO-SHOT IMAGE CLASSIFICATION WITH HIERARCHICAL LABEL SETS
Anonymous authors Paper under double-blind review
ABSTRACT
Open vocabulary models (e.g. CLIP) have shown strong performance on zero-shot classification through their ability generate embeddings for each class based on their (natural language) names. Prior work has focused on improving the accuracy of these models through prompt engineering or by incorporating small amount of labeled downstream data (via finetuning). In this paper, aim to tackle clas- class labels. We propose Classification with Hierarchical Label Sets (or CHiLS), alternative strategy that proceeds in three steps: (i) for each class, produce a set of subclasses, using either existing label hierarchies by querying GPT-3; (ii) perform the standard zero-shot CLIP procedure as though these subclasses were the labels of interest; (iii) map the pre- dicted subclass back to its parent to produce the final prediction. Across numerous datasets (with implicit semantic hicrarchies), CHiLS leads to improved accuracy yielding gains of over 30% in situations where known hierarchies are available and more modest gains when they are not. CHiLS is simple to implement within existing CLIP pipelines and requires no additional training cost.
sification problems with coarsely-define
INTRODUCTION
Recently, machine learning researchers have become captivated by the remarkable capabilities of
pretrained open vocabulary models (Radford et al. 2021 Wortsman et al. 2021: Via et al. 202T
Gao et al. 2021 Pham et al.) 222TT Cho al.] [022] Pratt et al.] (022). These models, like CLIP (Radford et al., )021) and ALIGN (Jia et al. 2021 learn to map images and captions into shared embedding spaces such that images are close in embedding space to their corresponding captions but far from randomly sampled captions. The resulting models can then used to assess the relative compatibility of a given image with an arbitrary set textual "prompts". Notably, RRdford et al. (22TT) observed that by inserting each class name directly within natural language prompt, one can then use CLIP embeddings to assess the compatibility of an images with each among the possible classes. Thus, open vocabulary models are able to perform zero-shot image classification, and do so with high rates of success (Radford et al. 2021 Zhang et al. 2021b).
Despite the documented successes, the current interest in open vocabulary models poses a new ques- tion: How should we represent our classes for a given problem in natural language? As class names are now part of the inferential pipeline (as opposed to mostly an afterthought in traditional scenarios) for models like CLIP in the zero-shot setting, CLIP's s performance is now directly tied the descriptiveness of the class "prompts" (Santurkar al. 2022). While many researchers have focused on improving the quality of the prompts into which class names are embedded (Radford al.) 202T Pratt al., 2022] Zhou al. 22022b al Huang et al.) (022). surprisingly little attention has been paid to improving the richness of the class names themselves. This can be particularly crucial in cases where class names are not very informative or are too coarsely-defined to match the sort of descriptions that might arise in natural captions. Consider, for example, the classes "large man-made outdoor things" and "reptiles" in the CIFAR20 dataset (Krizhevsky 2009).
In this paper, we introduce new method to tackle zero-shot classification with CLIP models for problems with coarsely-define class labels. We refer to our method as Classification with Hierarchical Label Sets (CHiLS for short). Our method utilizes a hierarchical map to convert each class into a list of subclasses, performs normal CLIP zero-shot prediction across the union set of all subclasses, and finally uses the inverse mapping convert the subclass prediction to the requi-

---

Under review as a conference paper at ICLR 2023
Standard Method
Our Method (CHiL5)
Dog Superclass Mapping
Dog
Dog +
Cat
Shiba Pug
Text Eneader
Subclass Mapping
Text Fncoder
Sphynx Tabby
CLIP
Cat
CLIP
Shiba
Image Encoddr
Image
Fncader
Figure 1: (Left) Standard CLIP Pipeline for Zero-Shot Classification. For inference, a standard CLIP takes in input set of classes and image where we want make a prediction and makes prediction from that set of classes, (Right) Our proposed method CHiLS for leveraging hierarchical class information into the zero-shot pipeline. We map each individual class set of subclasses, perform inferences in the subclass space (i.e., union set of all subclasses), and map the predicted subclass back to original superclass.
site superclass. We additionally include reweighting step wherein we leverage the raw superclass probabilities in order to make our method robust less-confident predictions at the superclass and subclass level.
We evaluate CHiLS wide array of image classification benchmarks with and without available hierarchical information. In the former case, leveraging preexisting hierarchies leads to strong ac- curacy gains across all datasets. In the latter, show that rather than enumerating the hierarchy by hand, using GPT-3 to query list of possible subclasses for each class (whether or not they are actu- aly present in the dataset) still leads to consistent improved accuracy over raw superclass prediction. We summarize our main contributions below:
We propose CHiLS, a method for improving zero-shot CLIP performance in scenarios with ill-defined and/or overly general class structures, which requires no labeled data or training time and is flexible both existing and synthetically generated hierarchies.
We show that CHiLS consistently performs as well or better than standard practices in situations with only synthetic hierarchies, and that CHiL- can achieve up to 30% accuracy gains when ground truth hierarchies are available.
RELATED WORK
2.1 TRANSFER LEARNING
While the focus of this paper is to improve CLIP models in the zero-shot regime, there a large body of work exploring improvements CLIP's few-shot capabilities. In the standard fine-tuning paradigm for CLIP models, practitioners discard the text encoder and only use the image embeddings as inputs for some additional training layers. This however, leads certain problems.
One particular line of work on improving the fine-tuned capabilities of CLIP models leverages model weight interpolation. WWrtsman al. (202T) proposes linear interpolate the weights of fine- tuned and a zero-shot CLIP model to improve the fine-tuned model under distribution shifts. This idea extended by Wortsman et al. (2022) into general purpose paradigm for ensembling models' weights in order to improve robustness. IIharco et al. (022) then builds on both these works and puts forth a method to "patch" fine-tuned and zero-shot CLIP weights together
order to avoid the issue of catastrophic forgetting. Among all the works in this section, our paper is perhaps most similar to this vein work (albeit in spirit), as CHiLS too seeks to combine two different predictive methods. Ding et al. (022) also tackles catastrophic forgetting, though they propose orthogonal direction and fine-tune both the image encoder and the text encoder, where the latter draws from a replay vocabulary of text concepts from the original CLIP database.
2

---

Under review as a conference paper at ICLR 2023
There is another line of work that seeks to improve CLIP models by injecting small amount of learnable parameters into the frozen CLIP backbone. This has been commonly achieved through the adapter framework (Houlsby et al.) 2019) from parameter-cfficice learning; specifically, in Gao et al. (2021) they fine-tune a small number of additional weights on top of the encoder blocks, which then connected with the original embeddings through residual connections. [Zhang al. (2021a) builds on this method by removing the need for additional training and simply uses a cached model. In contrast to these works, Jia et al. (2022) forgoes the adapter framework when using a Vision Transformer backbone for inserting learnable "prompt" vectors into the transformer's input layers, which shows superior performance over the aforementioned methods.
Additionally, some have looked at circumventing the entire process of prompt engineering. Zhou jtt al. (022a) and/Zhou et al. (022b) tackle this by treating the tokens within each prompt as learn- able vectors, which are then optimized within only a few images per class. [Huan et al. (2022) echoes these works, but instead does not utilize any labeled data and learns the prompt represen- tations in an unsupervised manner. Zhai et al. (2022) completely forgoes the notion of fine-tuning in the first place, instead proposing to reframe the pre-training process as only training a language model to match pre-trained and frozen image model. In all the above situations, some amount of data, whether labeled or not, is used in order to improve the predictive accuracy of the CLIP model.
2.2 ZERO- SHOT PREDICTION
The field of Zero-Shot Learning (ZSL) has existed well before the emergence of open vocabularly models, with its inception traced to Larochelle al. (008). With regards to non-CLIP related methods, the 7SL. paradigm has shown success in improving multilingual question answering (Kuo & Chen. (022) with large language models (LLMs), and also in image classification tasks where wikipedia-like context is used in order to perform the classification without access to the training
labels (Bujwid Sullivan 202T Shen et al. 2022).
With CLIP models, ZSL success has been found in a variety of tasks. Namely, Zhang et al. (02 expands the CLIP 2D paradigm for 3D point clouds. Tewel al. (202I) shows that CLIP models can be retrofitted to perform the reverse task of image-to-tex. generation, and Shen et al. 202T likewise display's CLIP's ability to improve performance on an array of Vision& I tasks. Both Yu et al. (022) Cho et al. (2022) expand CLIP's zero-shot abilities through techniques drawn from reinforcement learning (RL), with the former using CLIP for the task of audio captioning. Gadre al. (2022) similarly works with the RL literature and retrofits CLIP to improve the embodied AI lask of object navigation without any additional training. Zeng et al. (0022) shows the capabilities of composing CLIP-like models and LLMs together to extend the zero-shot capabilities to tasks like
assitive dialogue and open-ended reasoning. Unlike our work here, these prior directions mostly
focus on generative problems or, in the case of (Bujwid & Sullivan 2021 and Shen et al. 2022
require rich external knowledge databases to employ their methods.
In the realm of improving CLIP's zero-shot capabilities for image classification, we particularly note the contemporary work of Pratt et al. (2022). Here, authors explore using GPT-3 to generate rich textual prompts for each class rather than using preexisting prompt templates, and show improve- ments in shot accuracy across a variety of image classification baselines. In another work, Ren al. (2022) proposes leveraging preexisting captions in order improve performance, though this is restricted to querying the pre-training set of captions. In contrast, our work explores a comple- mentary direction of leveraging hierarchy class names to improve zero-shot performance of CLIIP. with fixed set of preexisting prompt templates.
2.3 HIERARCHICAL CLASSIFICATION
Our present work is related to the domain of Hierarchical Classification (Silla & Freitas 2010 i.e. classification tasks when the set of labels can be arranged in a DAG-like class hierarchy. Methodolo- gies from this domain have been extensively used for multi-label classification (Dimitrovski et al.)
2011
Liu et al. 2021 Chalkidis et al. 2020). and recent works have shown that this paradigm can aid in zero-shot learning by attempting uncover hierarchical relations between classes (Chen et al.) Mensink et al. 2014) and/or leveraging existing hierarchical information during training (Y)
202T
al.
2022 Cao al. 2020). Our line of inquiry is orthogonal to these approaches, as CHil.S is
3

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Classification with Hierarchical Label Sets (CHiLS)
input data point class labels C, prompt function T, label set mapping G, CLIP model f Union of subclasses for subclass prediction b Subclass probabilities Superclass probabilities
1: Set Cubb < U ((()
2: Juu =(f T(Cw))))
3: Shu (a T(C)))
4: for to |c] do
5: Sr, = G((s)
6: for #e,,) E Sc, do
7:
Joub (5 Juub Seps [upla]
Combining subclass ass and superclass predi in probabilig
8: end for
9:
end for
output (arg max Fon)
easily composable into standard CLIP pipelines, only models unseen class hyponyms (rather than relationships between known classes), and requires training of hierarchical embeddings.
3 PROPOSED METHOD
In this paper, we are primarily concerned with the problem of zero-shot image classification in CLIP models (see App. for an introduction to CL.PP and relavent terminology). For CLIP models, zero- shot classification involves using both pretrained image encoder and pretrained text encoder (see the left part of Figure [ To perform a zero-shot classification, we need predefined set of classes written in natural language. Let C = {ci, C2, C2, ................................ C& be such a set. Given an image and set of classes, each class is embedded within natural language prompt (through some function T(-)) to produce "caption" for each class (e.g. one standard prompt mentioned in Radford et al. (2021) is "A photo of a ()""). These prompts are then fed into the text encoder and after passing the image through the image encoder, we calculate the cosine similarity between the image embedding and each class class-promjj prompt embedding. These similarity scores form the output "logits" " of the CLIP model, which can be passed through softmax to generate the class probabilities.
As noted in Section previous work has focused on improving the T(-) for each class label o;- With CHiLS, we instead focus on the complementary task of directly modifying the set of classes C when C is ill-formed or overly general, while keeping T(-) fixed. Our method involves into two main steps: (I) using hierarchical information perform inference across subclasses, and (2) leveraging raw superclass probabilities to combine the best of subclass and superclass prediction probabilities.
3.1 ZERO-So -HOO PREDICTION WITH HIERARCHICAL LABEL SETS
Our method CHiLS slightly modifies the standard approach for zero-shot CLIP prediction. As each class label represents some concept in natural language (e.g. the label "dog"), we acquire subclass set S., {Sc,.1 ( 2... Se.,m. through some mapping function G, where each se,, is a linguistic hyponym, or subclass, of e; (e.g. corgi for dogs) and m, is the size of the set S..-
Given a label set Se, for each class, we proceed with the standard process for zero-shot prediction, but now using the union of all label sets as the set of classes. Through this, CHiLS will now produce its guess for the most likely subclass. We then leverage the inverse mapping function G-1 to coarse- grain our prediction back into the corresponding superclass. Our method is detailed more formally in Algorithm
In our work, we experiment with two scenarios: (i) when hierarchy information is available and can be readily queried; and (ii) when hierarchy information is not available and the label set for each class must be generated, which we do so by prompting GPT-3.
3.2 REWEIGHTING PROBABILITIES WITH SUPERCLASS CONFIDENCE
While the above method is able to effectively utilize CLIP's ability to identify relatively fine- ane-grained concepts, by predicting on only subclass labels lose any positive benefits of the superclass label,
4

---

Under review as a conference paper at ICLR 2023
Superclass Accuracy
CHiLS Accuracy (Existing Map)
CHiLS Accuracy (GPT-3 Map)
Dataset
Nonliving26
79.82
90.67 (+10.85)
81.51 (+1.69)
Living17
91.08
93.80 (+2.72)
91.43 (40.355
Entity
77.46
92.59 (+15.13)
78.11 (+0,65)
Entity30
70.32
88.87 (4-18.55)
71.75 (41,43)
CIFAR20
59.54
85.30
(4-25.76)
65.90 (4-636)
Food-101
91.82
N/A
Fruits-360
60.47
60.87 (4-040)
62.17 (41.70)
FashionIM
45.78
N/A
47.44 (+1.66)
Fashion-MNIST
68.50
N/A
70.85 (+2.35)
LSUN -cene
88.20
N/A
88.97 (+0.77)
Office31
89.13
N/A
89.37 (40.24)
OfficeHome
88.85
N/A
88.76 (-0.09)
ObjectNet
53.10
85.34
(+-32.24)
53.52 (+0.42)
EuroSAT
62.10
N/A
62.40 (40.30)
RESISC45
72.13
N/A
72.52 (400410)
Table 1: Zero-shot accuracy performance across image benchmarks with superclass labels (base- line), CHiLS with existing hierarchy (whenever available), and CHiLS with GPT-3 generated hier- archy. CHiLS improves classification accuracy in all situations with given label sets and all but datasets with GPT-3 generated label sets.
and performance may vary widely based the quality of the subclass labels. Given recent evidence (Minderer et al. |022 Kadavath et al. 2022) that large language models (like the text encoder in CLIP) are well-calibrated and generally predict correct labels with high probability, we modify our initial algorithm to leverage this behavior and utilize both superclass and subclass information. We provide empirical evidence of this property in Appendix A
Specifically, we include an additional reweighting step within our main algorithm (see lines 4-9 in Algorithm M Here, we reweight each set of subclass probabilities by superclass probability. Heuristically, as the prediction is now taken as the argmax over products of probabilities, large disagreements between subclass and superclass probabilities will be down-weighted (especially one particular superclass is confident) and subclass probabilities will be more important in cases where the superclass probabilities are roughly uniform. We show ablations on the choice of the reweighting algorithm in Section 4.4
4 EXPERIMENTS
In this section, we first lay out the experimental set-up. We then discuss, in order, the efficacy of our proposed method in situations with available class hierarchy information and in situations without any preexisting hierarchy. After these main results, we present series of ablations over various design choices showing where method is robust and what might be crucial for its performance.
4.1 SETUP
Datasets. As we are primarily concerned with improving zero-shot CLIP performance in situa- tions with ill-formed and/or semantically coarse class labels, test our method on the follow- ing image benchmarks: the four BREEDS imagenet subsets (living17, nonliving26, entity 13, and
30) (Santurkar et al. 2021). CIFAR20 (the coarse-label version of CIFAR100) (Krizhevsky)
2009). Food-101 (Bossard et al. 2014), Fruits-360 (Mureşan & Oltean 2018), FashionIM (Xiao
et al. 2015), Fashion-MNIST (Xiao et al. 2017). LSUN-Scene (Yu et al. 2015). Office31 (Saenko
et al. 2010). OfficeHome (Venkateswara et al. 2017). ObjectNet (Barbu et al.) 2019). EuroSAT
(Helber et al.) 2019 2018), and RESISC45 (Cheng et al. 2017). These datasets constitute wide range of different image domains and include datasets with and without available hierarchy informa- tion. Additionally, the chosen datasets vary widely the semantic granularity of their classes, from overly general cases (CIFAR20) to settings with mixture of general and specific classes (Food- 101, OfficeHome). We also examine CHiLS's robustness to distribution shift within a dataset by aver- aging all results for the BREEDS datasets, Office31, and OfficeHome across different shifts (see
5

---

Under review as a conference paper at ICLR 2023
Appendix H for more information). We additionally modify the Fruits-366 and ObjectNet datasets to create existing taxonomies. More details for dataset preparation are detailed in Appendix
Model Architecture. Unless otherwise specified, we use the ViTL/14 Vipp backbone RRafoord et al. 2021) for our CLIP model, and used Da Vinci-002 (with temperature fixed at 0.7) for all abla- tions involving GPT-3. For the choice of the prompt embedding function T(-), for each dataset we experiment (where applicable) with two different functions: (1) Using the average text embeddings of the 75 different prompts for each label used for ImageNet in Radford et al.] (202T). where the prompts cover wide arry of captions and (2) Following the procedure that Radford et al. (2021) puts forth for more specialized datasets, we modify the standard prompt to be of the form "A photo of (. a type of [context].", ", where [context] is dataset-dependent (e.g. "food" in the case of food- 101). In the case that a custom prompt set exists for a dataset, as is the case with multiple datasets that the present work shares with Radford al. (2021), we use the given prompt set for the latter option rather than building it from scratch. For each dataset, we use the prompt set that gives us the best baseline (i.e. superclass) zero-shot performance. More details are in Appendix C
Choice of Mapping Function G. In our experiments, we primarily look how the choice of the mapping function G influences the performance of CHiLS. In Section 4.2 we focus on the datasets with available hierarchy information. Here, G and G are simply table lookups to find the list of subclasses and corresponding superclass respectively. In Section[4.3 we explore situations in which the true set of subclasses in each superclass is unknown. In these scenarios, we use GPT-3 to generate our mapping function G. Specifically, given some label set size m, superclass name class name,, and optional context (which we use whenever using the context- based prompt embedding), we query GPT-3 with the prompt:
Generate a list of types of the following [context] : class-name
The resulting output list from GPT-3 thus defines our mapping from superclass to subclass. Unless otherwise specified, we fix m = 10 for all datasets. Additionally, in 4.4 we explore situations in which hierarchical information is present but noisy, i.c. the label set for each superclass contains the true subclasses and erroneous subclasses that not present in the dataset.
4.2 LEVERAGING AVAILABLE HIERARCHY INFORMATION.
We first concern ourselves with the scenario in which there is hierarchy information already avail- able (or readily accessible) for a given dataset. In this situation, the set of subclasses for each superclass exactly specified and correct (i.e. every image within each superclass falls into one of the subclasses). We emphasize that here we do not need information about which example belongs to which subclass, we just need mapping of superclass to subclass, For example, each class in the BREEDS dataset living!7 is made up of 4-8 ImageNet subclasses at finer granularity (e.g. "parrot" includes "african grey" and "macaw").
Results In Table T we can see that our method performs better than using the baseline superclass labels alone across all 7 of the datasets with available hierarchy information, in some cases leading to +15% improvement in predictive accuracy.
4.3 CHILS IN UNKNOWN HIERARCHY SETTINGS
Though we have seen considerable success in situations with access to the true hicrarchical struc- ture, in some real-world settings our dataset may not include any available information about the subclasses within each class. In this scenario, we turn to using GPT-3 to approximate the hierarchi- cal map G' (as specified in Section 4.1). It is important to note that GPT-3 may sometimes output suboptimal label sets, most notably in situations where GPT-3 chooses the wrong wordsense or when GPT-3 only lists modifiers on the original superclass (e.g. producing the list [red, rewlow, green] for types of apples). In order to account for these issues in an out-of-the-bbo fashion, we automatically append the superclass name (if not already present) each generated subclass label, and also include the superclass itself within the label set. For controlled analysis about the effect of including the superclass itself in the label set, see Appendix D

---

Under review as a conference paper at ICLR 2023
Figure 2: Selected examples of behavior differences between the superclass performance (which is our baseline) and CHiLS performance across three different datasets. (Upper left): CHiLS is correct, superclass prediction is not. (Lower left): Both correct. (Upper right): Both wrong. (Lower Right): superclass prediction correct, CHiLS is not.
Sperddaa
72.28
Supercaa
lewe shiing
$1.00
CIILS
imwelghting
rewe ghanggt
25 25
72.57
mean chiingt
11.52
celss mevegghingg wi sspartiaa))
14.49
mese chiintt
Aeeragg Auuaseyy
Figure 3: (Left) Average accuracy across datasets for superclass prediction, CHiLS (ours), and CHiLS without the reweighting step. While when given the hierarchy omitting the reweight- ing step can slightly boost performance beyond CHiLS, in situations without the true hierarchy the reweighting step crucial improving on the bascline accuracy. (Right) Average accuracy across datasets with GPT-generated label sets for different reweighting algorithms. Using aggre- gate subclass probabilities for reweighting performs noticeably worse than our initial method and reweighting in superclass space. CHiLS too only performs slightly worse than the contrived best possible union of subclass and superclass predictions.
Results In this setting, our method is still able to beat the baseline performance in most datasets, albeit with lower accuracy gains (see Table T). Thus, while knowing the true subclass hierarchy can lead to large accuracy gains, it is enough to simply enumerate a list of possible subclasses for each class with prior information about the dataset in order to improve the predictive accuracy. In Figure we show selected examples to highlight CHiLS's behavior across three datasets.
4.4 ABLATIONS
Reweighting Necessary? Though the reweighting step CHiLS motivated by the evidence that CLIP generally assigns higher probability to correct predictions rather than incorrect ones (see Appendix A for empirical verification), is not immediately clear whether
truely necessary. Averaged across all documented datasets, in Figure [ (left) we show that in the true hierarchy set- ting, not reweighting the subelass probabilities can actually slightly boost performance (as the label sets are adequately tuned to the distribution of images). However, in situations where the true hier- archy is not present, omitting the reweighting step puts accuracy below the baseline performance. We attribute this difference in behavior to the fact that reweighting multiplicatively combines the superclass and subclass predictions, and thus if subclass performance sufficient on its own (as the case when the true hierarchy is available) then combining it with superclass predictions can

---

Under review as a conference paper at ICLR 2023
Superclass Accuracy
CHiLS Accuracy (Existing Map)
CHiLS Accuracy
Dataset
(Existing Map + Noise)
nonliving26
79.82
90.67 (-10.85)
89.48 (+9.66)
living17
91.08
93.80 (+2.72)
92.47 (+1.39)
entity13
77.46
92.59 (- 1-1..33)
90,34 (+12.88)
entity30 70.32
88.87 (-18.55)
87.56 (+17.221)
Table 2: CHiLS zero-shot accuracy when G includes all subclasses the ImageNet hierarchy de- scended from the respective root node. Even in the presence of noise added to the true label sets, CHiLS is able to make large accuracy gains.
cause the model to more closely follow the behavior of the underperforming superclass predictor. Thus, as not having the true hierarchy is considerably more likely in the wild, the reweighting step is imperative to utilizing CHiLS to its fullest.
Different Reweighting Strategies We too experimented on whether the initial reweighting al- gorithm is the optimal method for combining superclass and subclass predictions. Namely, investigated whether superclass probabilities could be replaced by the sum over the matching sub- class probabilities, and whether we can aggregate subclass probabilities and reweight them with the matching superclass probabilities (i.e. performing the normal reweighting step but in the space of su- perclasses). In Figure (right) we show that replacing the superclass probabilities in the reweighting step with aggregate subclass probabilities removes any accuracy gains from CHiLS, but that doing the reweighting step in superclass space does maintain CHiLS accuracy performance. This suggests that the beneficial behavior of CHiLS may be due to successfully combining two different sets of class labels. We also display the upper bound for combining superclass and subclass prediction (i.e. the accuracy when datum is correctly labeled if the superclass or subclass predictions are correct) in purple, which we note is impossible in practice, and observe that even the best possible performance is not much higher than the performance of CHiLS.
Noisy Available Hierarchies While the situation described in Section 4.3 is the most probable in practice, we additionally investigate the situation in which the hierarchical information is present but overestimates the set of subclasses. For example, the scenario in which a dataset with the class "dog" includes huskies and corgis, but CHiLS is provided with huskies, corgis, and Labradors as possible subclasses, with the last being out-of-distribution datasets presented in [anttrkar et al. (202T). As the BREEDS datasets were created so that each class contains the same number of subclasses (which are ImageNet classes), we modify G' such that the label set for each superclass corresponds to all the ImageNet classes descended from that node in the hierarchy (see Appendix for more information). As we can see in Table CHiLS is able to improve upon the baseline performance even in the presence of added noise in each label set.
To do this, we return to the BREEDS
Label Set Size In previous works investigating importance of prompts in CLIP's performance, it has been documented that the number of prompts used can have decent effect on the overall performance (Pratt et al.) 2022] Santurkar et al., 2022). Along this line, we investigate how the size of the subclass set generated for each class effects the overall accuracy by re-running our main experiments with varying values of m (namely, 1, 5, 10, 15, and 50). In Figure 4 (right), there is little variation across label set sizes that is consistent over all datasets, though m has few very low performing outliers due to the extremely small label set size. We observe that the optimal label set size is context-specific, and depends upon the total number of classes present and the semantic granularity of the classes themselves. Individual dataset results are available in Appendix E
Model Size In order to examine whether the performance of CHiLS only exists within the best performing CLIP backbone (e.g. ViT-L/14@ 336), we measure the average relative change in ac- curacy performance between CHiLS and the baseline superclass predictions across all datasets for an array of different CLIP models. Namely, we investigate the RN50, RN101, RN50x4, ViT-B/16, ViT-B/32, and ViT-L/140 336 CLIP backbones (see Radford et al. 2021 for more information on the model specifications). In Figure 4 (left), we show that across the 6 specified CLIP backbones, CHiLS performance leads to relatively consistent relative accuracy gains, with slight (but not con- fidently significant) trend showing improved performance for the ResNet backbones over the ViT
8

---

Under review as a conference paper at ICLR 2023
Map GPT
23
RN50
AM:301
RA5064
VIBB16
VI7822 VIT.340336
10 Label Set Size
CLIP Model Backboone
Figure 4: (Left) Average relative change between CHiLS and baseline for true mapping and GPT- 3 generated mapping. Across changes in CLIP backbone size and structure, the effectiveness of CHiLS at improving performance only varies slightly. (Right) Average relative accuracy change from the baseline to CHiLS (across all datasets), for varying label set sizes. In all, there not much difference in performance across label set sizes.
backbones, which is be expected given their worse base capabilities. This shows that CHiLS's benefits are not an artifact of large model size.
Alternative Aggregating Methods While CHiLS is based a set-based mapping approach for subclasses and linear averaging for prompt templates (based on Raaford et al.] (202T)'s proce- dure), we experimented with two alternative ensembling methods for different parts of the CHiLS pipeline: (1) Using linear average of subclass embeddings rather than the set-based mapping (that is, every superclass's 's text embedding the average across all subclass embeddings, each themselves averaged across every prompt template) and (2) Using set-based mapping for prompt templates rather than a linear average (i.e. instead of averaging across prompt templates, predict across each prompt template separately at inference time and then use embedded class to map back to the set of superclasses). Note the latter case we only experiment with how this effects superclass prediction (where each class maps to set of the dataset's chosen prompt embeddings), as using set-based ensembling for both prompts and subclasses within CHiLS quickly becomes computationally ex- pensive. In Figure (in Appendix F we see that using initial aggregation methods (i.e. linear averaging for prompts and set mappings for subclasses) achieves greater accuracy.
CONCLUSION
In this work, demonstrated that the zero-shot image classification capabilities of CLIP models can be improved by leveraging hierarchical information for given set of classes. When hierarchical structure available given dataset, our method shows large improvements in zero-shot accuracy, and even when subclass information isn't I explicitly present, we showed that we can leverage GPT-3 generate subclasses for each class and still improve upon the baseline (superclass) accuracy.
We remark that CHiLS may be quite beneficial to practitioners using CLIP as an out-of-the-boo im- age classifier. Namely, we show that in scenarios where the class labels may be ill-formed or overly coarse, even without existing hierarchical data accuracy can be improved with a fully automated pipeline (via querying GPT-3), yet CHiLS is flexible enough that any degree of hand-crafting label sets can be worked into the zero-shot pipeline. Our method has the added benefit of being both completely zero-shot (i.e. no training or fine-tuning necessary) and is resource efficient.
Limitations and Future Work As with usual zero-shot learning, we don't 't have way to validate the performance of our method. Additionally, we recognize that CHiLS suited for scenarios in which a semantic hierarchy most likely exists, and thus may not be particularly useful in classifica- tion tasks where the classes are already decently fine-grained. We believe that this limitation will not hinder the applicability of our method, as practitioners would know if their task contains any la- tent semantic hierarchy and thus choose to use our method or not a priori. Given CHiLS's empirical successes, we hope to perform more investigation to develop understanding of why CHiLS is able to improve zero-shot accuracy and whether there is a more principled way of reconciling superclass and subclass predictions.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBLITY STATEMENT
The source code for reproducing the work presented here is available bhttpss///githii com/ ambnnopenReeieewwclllp-----
We implement our method in Py Torch (Paszke al.)
2017) and provide an infrastructure to run all the experiments to generate corresponding results. We have stored all models and logged all hyperparameters and seeds to facilitate reproducibility. Additionally, all necessary data preprocessing details are present in Appendix H
REFERENCES
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlle dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative com- ponents with random forests. In European Conference on Computer Vision (ECCV), 2014.
Sebastian Bujwid and Josephine Sullivan. Large-scale zero-shot image classification from rich and diverse textual descriptions. In Proceedings of the Third Workshop Beyond Vision and LAN- guage: inggaainng Real-world kNowledge (LANTERN), 2021.
Zhong Cao, Jiang Lu, Sen Cui, and Changshui Zhang. Zero-shot handwritten chinese character recognition with hierarchical decomposition embedding. Pailern Recognition, 2020.
Ilias Chalkidis, Manos Fergadiotis, Sotiris Kotitsas, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. An empirical study on large-scale multi-label text classification including few and zero-shot labels. arXiv preprint arXiv:2010.01655 2020.
Shiming Chen, GuoSen Xie, Yang Liu, Qinmu Peng, Baigui Sun, Hao Li, Xinge You, and Ling Shao. Hsva: Hierarchical semantic-visual adaptation for zero-shot learning. 2021.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, 2017.
Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal. Fine-grained image captioning with clip reward. In Findings of NAACL, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Computer Vision and Pattern Recognition (CVPR), 2009.
Ivica Dimitrovski, Dragi Kocev, Suzana Loskovska, and Sašo Džeroski. Hierarchical annotation of medical images. Pattern Recognition, 2011.
Yuxuan Ding, Lingqiao Liu, Chunna Tian, Jingyuan Yang, and Haoxuan Ding. Don't stop learning: Towards continual learning for the clip model, 2022.
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Clip on wheels: Zero-shot object navigation as object localization and exploration. arXiv preprint arXiv:2203. 10421, 2022.
Peng Gao, Shijie Geng, Renrui Zhang. Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter Better lasion-language models with feature adapters. arXiv preprint arXiv:2 10.04444, 2021.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723, 2020.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. 2018.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
10

---

Under review as a conference paper at ICLR 2023
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficiee transfer learning for nlp. In International Conference on Machine Learning (ICML), 2019.
Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language mod- els, 2022.
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Si- mon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpo- lating weights. arXiv preprint TrXiv:2208.05592, 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203. 12/ 19, 2022.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know, 2022.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Chia-Chih Kuo and Kuan-Yu Yu Chen. Toward zero-shot and zero-resource multilingual question an- swering. IEEE Access, 2022.
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In Associ- ation for the Advancement of Artificial Intelligence (AAAI), 2008.
Hui Liu, Danqing Zhang, Bing Yin, and Xiaodan Zhu. Improving pretrained models for zero- shot multi-label text classification through reinforced label hierarchy reasoning. arXiv preprint arXiv:2104. 0066666666,6... 2021.
Thomas Mensink, Efstratios Gavves, and Cees GM Snoek. Costa: Co-occurrence statistics for zero-shot classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2014.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Horea Mureşan and Mihai Oltean. Fruit recognition from images using deep learning, Acta Univer- sitatis Sapientiae, Informatica, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, al. Combined scaling for open-vocabulary image classification. arXiv preprint arXiv: 2111.10050, 2021.
Sarah Pratt, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning (ICML), 2021.
11

---

Under review as a conference paper at ICLR 2023
Shuhuai Ren, Lei Li, Xuancheng Ren, Guangxiang Zhao, and Xu Sun. Rethinking the openness of clip. 2022.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference Computer Vision (ECCV), 2010.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. In International Conference on Learning Representations (ICLR), 2021.
Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. caption worth thousand images? a controlled study for representation learning, 2022.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai- Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and-language tasks? arXiv preprint arXiv:2107.06383, 2021.
Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang. Pengchuan Zhang, Anna Rohrbach, Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, Kurt Keutzer, Trevor Darrell, and Jianfeng Gao. K-lite: Learning transferable visual models with external knowledge, 2022.
Carlos Nascimento Silla and Alex Alves Freitas. A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 2010.
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-text genera- tion for visual-semantic arithmetic, 2021.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. arXiv preprint wwXiv:2109001903 2021.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Loper Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Leurning, pp. 23965-23998. PMLR, 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist a novel image dataset for benchmark- ing machine learning algorithms, 2017.
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recog- nition from abbey to zoo. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Kai Yi, Xiaoqian Shen, Yunhao Gou, and Mohamed Elhoseiny. Exploring hierarchical graph repre- sentation for large-scale zero-shot image classification. arXiv preprint arXiv:2203.01388. 2022.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop, 2015.
Youngjae Yu, Jiwan Chung, Hecseung Yun, Jack Hessel, JaeSung Park, Ximing Lu, Prithviraj Am- manabrolu, Rowan Zellers, Ronan Le Bras, Gunhee Kim, and Yejin Choi. Multimodal knowledge alignment with reinforcement learning, 2022.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveck Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Van- houcke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv, 2022.
12

---

Under review as a conference paper at ICLR 2023
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03931) 2021a.
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip, arXiv preprint 11X.v:21 2021b.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision (ICV), 2022b.
13

---

Under review as a conference paper at ICLR 2023
APPENDIX
A EMPIRICAL EVIDENCE OF CLIP CONFIDENCE
1.0
0.8
Probability
0.6
0.4
0.2
Correct Predictions
Incorrect Predictions
bag
ball
body armor
boat
bottle
bus
car
digital computer
chair
coat
dwelling
halt keyboard Instrument
mercantile estab sttabishmeet
fence
percussion Instrument
outhuilding
pot
roof
stringed instrument
ship skirt
timeplece
wind Instrument
truck
squash
(a) nonliving26
1.0
0.9
0.8
Probability
0.7
0.6
0.5
0.4
Correct Predictions
Incorrect Predictions
0.3
sala valamander
turtle
lizard
snake
spider
grouse
parrot
crab
dog
wolf
domestic cat
fox
bear
beetle
butterfly
ape
monkey
(b) living
Figure 5: Distribution of argmax probabilities across ImageNet BREEDS datasets for correctly and incorrectly classified data points, with the diamonds representing average probability for each class. Conrectly classified probabilities are average higher than the misclassified probabilities.
The motivation behind the reweighting step of CHiLS primarily comes from the heuristic that LLMs make correct predictions with high estimated probabilities assigned to them (Kadavath et al.] 2022). and that CL.P models themselves are well-calibrated (Minderer et al. 2021). However, also
14

---

Under review as a conference paper at ICLR 2023
1.0
09
0.8
Probability
400 0100 9019
0.6
0.4
0.2
Correct Predictions
Incorrect Predictions
garment
bird
reptile
arthropod
marnmal
accessory
craft
equipment
furniture
Instrument
man-made structure
wheeled vehicle
produce
(c) entity13
1.0
0.8
Probability
0.6
0.4
0.2
Correct Predictions
Incorrect Predictions
serpentess
passerine
saurian
arachnid
aquatic bird
crustacean
carnivore
insect ungulate
primate
bony fish
electronic equipment
barrier building
footwear
garment
home appllance
headdress
measuring instr insrrmment
kitchen utensil
musical Instrument
motor vehicle
sports equipment
neckwear
tablewars
tool
vessel
dish vegetable fruit
(d) entity30
verify whether there is some evidence of this behavior in CLIP models. Given that the output of CLIP model is probability distribution over the provided classes, we care specifically about the probability of the argmax class (i.e. the predicted class) when the model is correct and when it is incorrect. Across the BREEDS datasets for the standard ImageNet domain, in Figure 3 we show the distribution of the correct and incorrect argmax probabilities for each class (i.e. for each class e,, we show the output probabilties for & when it was correctly classified and the output probabilities of the predicted classes when the true class c;). Whenever CLIP is correct, the associated probability is on average much higher than the probabilities associated with misclassification.
15

---

Under review as a conference paper at ICLR 2023
B CLIP PRIMER
Open Vocabulary models (as termed in Pham et al. images by associating them with natural language descriptions of each class. These models are "open" in the sense that they are to predict on an arbitary vocabulary of descriptions (as opposed to fix set), thus allowing for arbitrary-way image classification. Popular open vocabulary models include the model of focus CLIP (Radford et al. 2021) and ALIGN (Jia et al. 2021 as examples.
(021 refer to models that are able to classify
Contrastive Language Image Pretraining (CLIP) is family of open vocabulary models, and the focus of the present work. CLIP, which is comprised of a text encoder and an image encoder that project into the same latent space, is trained in the following way: Given a set of image-caption pairs (e.g. a photo of dog with the caption "a photo of a dog.") "), CLIP is trained to predict which caption goes with which image as a contrastive learning objective by comparing the similarity between each image embedding and each caption embedding.
At inference time (in the zero-shot setting). a naive method for image classification (which is the initial baseline tried in Radford et al. (202T)) involves simply passing in the list of class names for given dataset, and calculating the similarity between a particular image embedding and each one of these class embeddings. However, [Radford et al.] 2 found that by taking cue from the recent literature on prompt engincering for large language models (Gao et al. 2020 CLIP can perform significantly better prompt that resembles some sort of image caption (as that is what CLIP was trained on). As an example, the standard baseline prompt mentioned is "A photo of ().". In our work, we define prompt (or prompt template, which we use interchangeably) as any caption-like phrase in natural language that class name can be injected into.
zero-shot predictor if each class name included in natural language
C ADDING CONTEXT TO PROMPTS AND GPT-3 QUERIES
Dataset
[context]
Prompt Set Used
Nonliving26
N/A
ImageNet
Living 17
N/A
ImageNet
Entity13 13
N/A
ImageNet
E
N/A
ImageNet
CIFAR20
N/A
ImageNet
Food-101
"food"
Dataset-Spectee Speecffcc
Fruits-360
"fruit"
Pataset-Speciff
Fashion MM
"article of clothing"
Speecfc
Fashion-MNIST
"article of clothing"
ImageNet
LSUN-Scene
N/A
ImageNet
Office31
"office supply" "office supply" N/A
Dataset-Specifit
OfficeHome
ImageNet
ObjectNet
ImageNet
FuroSAT
N/A
Dataset-Speche
RESISC45
N/A
Dataset-Specific
Table 3: Context tokens and prompt sets used for each dataset.
In order to disentangle the effect that well-formed prompt templates have on the success of CHiLS, for each dataset (besides the BREEDS datasets and ObjectNet as they are already semantically similar to ImageNet) we compare the ImageNet 75 classes against dataset-specific set of prompt templates. In the case of AUToSAT, RESISC45, CIFAR20 and Food-101, we directly use the prompt template set from Radford et al.) (021). For ASUN-Scene, we use the prompt template set for SUN397 (Xiao et al.] 2010), as the two datasets are semantically similar. For the rest of the datasets not yet mentioned (namely Fruits360. Fashion 1M, Fashion-MNEL
Office31, and OfficeHome) we add the [context] marker into the standard prompt template as mentioned in Section 4.] The prompt sets themselves can be directly found in the code implementation for this project.
16

---

Under review as a conference paper at ICLR 2023
For the GPT-3 Query with additional context, add the respective [context] token to the query if the -ataset-specific prompt template used. Note that we did not create [context] tokens for EuroSAT, L.UU--Scene not seem to be a concise semantic label to describe the classes in these datasets. In Table B we list the dataset, the [context] token (if applicable), and the final prompt used for all the experiments. Here, we found that while dataset-specific prompts often improved baseline performance, they were gauranteed to improve performance, as in both Fasion-MNIST and OfficeHome the general ImageNet prompt set performed better.
RESISC45 despite testing dataset-specific prompt templates, there did
D INCLUDING SUPERCLASS LABELS IN LABEL SETS
CHiLS Accuracy (Existing Map)
CHiLS Accuracy (Existing Map+)
CHiLS Accuracy (GPT-3 Map)
CHiLS Accuracy (GPT-3 Map+)
Dataset
Nonliving26
90.67 (+10.85)
89.80 (49.98)
81.46 (+1.63)
81,51 (+1.69)
Living17
93.80 (+2.72)
93.62 (+2.54)
91.30 (+0.22)
91.43 (+0.35)
Entity13
92.59 (+15.13)
92.06 (+14.60)
76.97 (-0.48)
78.11 (+0.65)
Entity30
88.87 (+18.55)
87.29 (+16.97)
71.80 (+1.48)
71.75 (+1.43)
CIFAR20
85.30
(+25.76)
81.40 (+21.86)
65.70 (+6.16)
65.90 (46.36)
Food-101
N/A
N/A
91.63 (-0.19)
91.73 (-0.09)
Fruits-360
60.87
(40.40)
60.63 (40.16)
62.48 (+2.01)
62.17 (+1.70)
Ia
N/A
N/A
47.51 (+1.73)
47.44 (+1.66)
Fashion-
N/A
N/A
67.82 (-0.98)
70.85 (+2.35)
MNIST
LSUN-Scene
N/A
N/A
88.80 (+0.60)
88.97 (+0.77)
Office31
N/A
N/A
86.58 (-2.71)
89.37 (+0.24)
OfficeHome
N/A
N/A
87.88 (-0.97)
88.76 (-0.09)
ObjectNet
85.34
(+32.24)
81.30
(+28.20)
51.23 (-2.07)
53.52 (+0.42)
EuroSAT
N/A
N/A
62.21 (+0.11)
62.40 (+0.30)
RESISC45
N/A
N/A
71.64 (-0.48)
72.52 (+0.40)
Table 4: Zero-Shot Accuracy Performance across benchmarks, controlling for the presence of the superclass label within each respective label set. In the existing map case, adding the superclass labels removes some of the performance gains of the raw existing map. In the GPT-3 Map case, adding the superclass is crucial to maintaining performance in most datasets
With CHiLS when the existing map is not available, we append the superclass name to each label set to account for possible noise the GPT-generated label set. In Table 4 we show the effect that this inclusion has in both the existing map and GPT-map cases. Note that in the main paper, columns and 4 correspond to the main results (i.c. no superclass labels in existing maps and su- perclass labels in GPT-3 maps). In both cases, the presence of the superclass label more effectively strikes a balance between subclass and superclass predictions. In the existing map case, this actually hurts performance, the subelass labels optimal in the given dataset. In the GPT-3 map case, while there are some datasets where removing the superclass label improves performance (namely F60 and Entity30), in ever other case removing the superclass label hurts performance, some- times by multiple percentage points.
17

---

Under review as a conference paper at ICLR 2023
E LABEL SET ABLATION ACCURACY
CHiLS
CHiLS
CHiLS
CHiLS
CHiLS
Dataset
(m = 1)
(m = 5)
(m = 10)
(m 15)
(m = 50)
Nonliving26
80.05 (++0.23)
81.12(+1.30)
81.51 (+1.69)
81.79 (+1197)
79.60 (-0.20)
Living17
91.47 (+-0.99)
92.69 (+1161)
91.43 (+0.355
91.55 (+0.448)
91.52 (+0.45)
Entity13 13
76.75 (-0.70)
78.15 (+00770)
78.11 (+0.65)
78.42 (+0.96)
75.82(11.8)
Entity30
72.25 (+1.993)
71.47 (41.15)
71.75 (+1.43)
73.38 (43.006)
70.18 (-0.44))
CIFAR20
63.61 (+44.02)
64.95 (45.41)
65.90 (46.36)
62.80 (4-3226)
63.72 (+4.13)
Food-101
91.57 (-0.25)
91.75 (-0.07)
91.73 (-0.09)
91.51 (-0.31)
91.58 (-0.24)
Fruits-360
61.18 (+-0.47)
12.33(44.888
62.17 (4-1.70)
62.51 (42..04)
61.19 (+0.48)
M
38.57 (-7.21)
45.77 (-0.01)
47.55 (4-1.66)
46.93 (41.15)
41.98 (-3.80)
Fashion-
67.16 (-1.34)
70.93 (+2.44)
70.84 (+2.35)
69.09 (+0.60)
69.98 (+1.49)
MNIST
LSUN-Scoo
87.20 (-1.00)
86.33 (-1.87)
88.97 (+0.77)
86.80 (-1.4) (-1.00)
85.60 (-2.60)
Office31
89.46 (+0.36)
88.08 (-1.05)
89.37 (+0.24)
89.03 (-0.10)
90.23 (+1.10)
OfficeHome
88.06 (-0.79)
89.12 (+0.27)
88.76 (-0.99)
89.06 (+0.211)
88.39
ObjectNet
50.12 (-2.98)
53.29 (40.18)
58.19 (+0.42)
57.66 (+11.56)
58.19 (45.09)
EuroSAT
62.59 (+-0.49)
62.21 (40.10)
62.40 (40.30)
62.89 (4-0.79)
61.39 0.71
RESISC45
73.19 (+-1.06)
72.05 (-0.08)
72.52 (40044)
72.50 (40.38)
70.61 (-1.22)
Table Accuracy across different label set sizes generated by GPT-3, with best performing label set size in each row bolded. In general, there is no consistent trend related to label set size and zero-shot performance across datasets.
Table 5 displays the raw accuracy scores for CHiLS across different label set sizes.
F
ALTERNATIVE AGGREGATION RESULTS
poople avataging!
72.28
Suparian baaed pronpt mapeing
thiss frus Mun on bated
frear averagingl
91.82
chiss Map. Clds mapping!
CHISS Iom Sap,
45
Anaaggu Accuroty
Figure 6: Average accuracy across datasets for varying aggregative methods on both the prompt and subclass steps of the zero-shot pipeline. In general, linear averaging for subclasses performs worse than our proposed set-based method, while linear averaging for prompts (for raw superclass prediction) performs better thant using set-based mapping.
F displays the results of our ablation on alternative ways to aggregate subclasses and prompts.
18

---

Under review as a conference paper at ICLR 2023
G NOISY AVAILABLE HIERARCHY DETAILS
The ImageNet (Deng et al. 2009) dataset itself includes a rich hierarchical taxonomy, where every class is leaf node of the hierarchy. In the original BREEDS (Santurkar et al. [021 work, the authors modify the structure slightly in order to place concepts at semantically-simie -imillr levels of granularity at the same depth, and additional restrict the number of subelasses within each of the BREEDS datasets in order to balance the data. Thus, it is possible for each BREEDS dataset to use the dataset with its superclasses and restricted set of subclasses but provide CHiLS with all the subclass labels present in the ImageNet hierarchy for each superclass (i.e. all leaf nodes descended from each superclass node). In Table i we display subset of the living17 BREEDS dataset class structure with the original subclasses and the ImageNet subclasses. Observe that in some cases, there are many subclass labels provided to CHiLS than is present in the data.
H DATASET DETAILS
Dataset
Domains
ImageNet, ImageNet-Sketch, ImageNetv2, ImageNet-c
BREEDS
[Fog-1, Contrast-2, Snow-3, Gaussian Blur-4, Saturate-5]
Office31 OfficeHome
Amazon, DSLR, webcam
Clipart, Art, Real World, Product
Table 6: Domains used for BREEDS, Office31, and OfficeHome.
CHiLS Across Domain Shifts (Saenko et al. 2010), and OfficeHome (Venkateswara al., (017). all results presented are the average over different domains. The specific domains used are show Table
For each of the BREEDS datasets (Santurkar et al. 2021). Office31
Fruits-360 For zero-shot classification with CLIP models, Fruits-360 (Mureşan Oltean 2018) in its raw form is somewhat ill-formed from a class name perspective, as there are classes only differentiated by a numeric index (e.g. "Apple Golden I" and "Apple Golden 2") and classes at mixed granularity (e.g. "forest nut" and "hazelnut" are separate classes even though hazelnuts are type of forest nut). We thus manually rename classes using the structure laid out in Table 9. which results in 59- superclass classification problem, with 102 ground-truth subclasses.
ObjectNet The ObjectNet dataset (Barbu et al. 2019) has partial overlap (113 classes) with the the
ImageNet (Deng et al. (2009) hierarchical class structure. From this subset of ObjectNet, BREEDS hierarchy (Santurkar et al. 2021 to generate a coarse-grained version of ObjectNet that shown in Table g In this 11 11-way classification task, the subclasses are the original ObjectNet classes
19

---

Under review as a conference paper at ICLR 2023
Superclass
Original BREEDS subclasses
All ImageNet subclasses
salamander
European fire salamander, common newt, eft, spotted salamander
European fire salamander, common newt, eft, spotted salamander, olotl
turtle
loggerhead, leatherback turtle, mud turtle, terrapin common
loggerhead, leatherback turtle, mud turtle, terrapin, box turtle banded gecko, common iguana, American chameleon, whiptail, agama, frilled lizard, alligator lizard, Gila monster, green lizard, African chameleon, Komodo dragon
lizard
iguana. American chameleon, agama, frilled lizard
snake
thunder snake, ringneck snake, dia- mondback, sidewinder
thunder snake, ringneck snake, hog- nose snake, green snake, king snake, garter snake, water snake, vine snake, night snake, boa con- strictor, rock python, Indian cobra, green mamba, sea snake, horned viper, diamondback, sidewinder black and gold garden spider, barn spider, garden spider, black widow, tarantula, wolf spider
spider
black and gold garden spider, barn spider, garden spider, black widow
grouse
black grouse, ptarmigan, ruffed grouse, prairie chicken African grey, macaw, sulphur- crested cockatoo, lorikeet Dungeness crab, rock crab, fiddler crab, king crab
black grouse, ptarmigan, ruffed grouse, prairie chicken African grey, macaw, sulphur- crested cockatoo, lorikeet Dungeness crab, rock crab, fiddler crab, king crab
parrot
crab
Table 7: Subset of living17 class hierarchy, showing the difference between the original BREEDS subclasses and the ImageNet subclasses used for the ablation in Section 4.4 Noisy Available Hier- archies.
Table 9: Mapping from original class names to new subclass and superclasses for Fruits-360.
Original Class
Cleaned Subclass
Cleaned Superclass
Apple Braeburn
braeburn apple
apple
Apple Crimson Snow
crimson snow apple
apple
Apple Golden
golden apple
apple
Apple Golden
golden apple
apple
Apple Golden 3
golden apple
apple
Apple Granny Smith
granny smith apple
apple
Apple Pink Lady
pink lady apple
apple
Apple Red
red apple
apple
Apple Red
red apple
apple
Apple Red
red apple
apple
Apple Red Delicious
red delicious apple
apple
Apple Red Yellow
red yellow apple
apple
Apple Red Yellow
red yellow apple
apple
Apricot
apricot
apricot
Avocado
avocado
avocado
Avocado ripe
avocado
avocado
Banana
banana
banana
Banana Lady Finger
lady finger banana
banana
Banana Red
red banana
banana
Beetroot
bectroot
beetroot
Blueberry
blueberry
blueberry
Cactus fruit
cactus fruit
cactus fruit
20

---

Under review as a conference paper at ICLR 2023
Cantaloupe 1
melon
melon
Cantaloupe 2
melon
melon
Carambula
star fruit cauliflower
star fruit
Cauliflower
cauliflower
Cherry 1
cherry
cherry
Cherry 2
cherry rainier cherry
cherry
Cherry Rainier
cherry
Cherry Wax Black
black cherry red cherry
cherry
Cherry Wax Red
cherry
Cherry Wax Yellow
yellow cherry
cherry
Chestnut
nut orange cocos com com husk cucumber
nut
Clementine
orange
Cocos
cocos
Corn
corn
Corn Husk
corn husk cucumber cucumber date eggplant
Cueumber Ripe
Cucumber Ripe 2
cucumber
Dates
date eggplant
Eggplant
Fig
fig ginger root granadilla blue grape pink grape white grape white grape white grape white grape pink grapefruit white grapefruit gauva nut huckleberry
fig ginger root
Ginger Root
Granadilla
passion fruit
Grape Blue
grape
Grape Pink
grape
Grape White
grape
Grape White 2
grape
Grape White 3
grape
Grape White 4
grape
Grapefruit Pink
grapefruit
Grapefruit White
grapefruit
Guava
gauva
Hazelnut
nut
Huckleberry
huckleberry
Kaki
kaki
persimmon
Kiwi
kiwi
kiwi
Kohlrabi
kohlrabi
kohlrabi
Kumquats
kumquat
kumquat
Lemon
lemon
lemon
Lemon Meyer
meyer lemon
lemon
Limes
lime
lime
Lychee
Iychee
lychee
Mandarine
orange
orange
Mango
mango
mango
Mango Red
red mango
mango
Mangostan
mangostan
mangostan
Maracuja
maracuja
passion fruit
Melon Piel de Sapo
melon mulberry
melon mulberry
Mulberry
Nectarine
nectarine
nectarine
Nectarine Flat
flat nectarine forest nut pecan nut
nectarine
Nut Forest
nut
Nut Pecan
nut onion onion onion orange
Onion Red
red onion red onion
Onion Red Peeled
Onion White
white onion
Orange
orange
Papaya
papaya passion fruit
papaya
Passion Fruit
passion fruit
Peach
peach
peach
21

---

Under review as a conference paper at ICLR 2023
Peach 2
peach flat peach pear
peach
Peach Flat
peach
Pear
pear
Pear
pear abate pear forelle pear
pear
Pear Abate
pear
Pear Forelle
pear
Pear Kaiser
kaiser pear
pear
Pear Monster
monster pear
pear
Pear Red
red pear stone pear williams pear pepino green pepper orange pepper red pepper yellow pepper groundcherry groundcherry pineapple
pear
Pear Stone
pear
Pear Williams
pear
Pepino
pepino
Pepper Green
pepper
Pepper Orange
pepper
Pepper Red
pepper
Pepper Yellow
pepper
Physalis
groundcherry
Physalis with Husk
groundcherry
Pineapple
pineapple
Pineapple Mini
mini pineapple
pineapple
Pitahaya Red
dragon fruit
dragon fruit
Plum
plum
plum
Plum 2
plum plum pomegranate
plum
Plum 3
plum
Pomegranate
pomegranate
Pomelo Sweetie
pomelo red potato red potato
pomelo
Potato Red
potato
Potato Red Washed
potato
Potato Sweet
sweet potato
potato
Potato White
white potato
potato
Quince
quince
quince
Rambutan
rambutan
rambutan
Raspberry
raspberry
raspberry
Redcurrant
redcurrant
redcurrant
Salak
salak
snake fruit
Strawberry
strawberry
strawberry
Strawberry Wedge
strawberry
strawberry
Tamarillo
tamarillo
tamarillo
Tangelo
tangelo
tangelo
Tomato
tomato
tomato
Tomato
tomato
tomato
Tomato
tomato
tomato
Tomato
tomato
tomato
Tomato Cherry Red
cherry tomato
tomato
Tomato Heart
heart tomato
tomato
Tomato Maroon
maroon tomato yellow tomato
tomato
Tomato Yellow
tomato
Tomato not Ripened
unripe tomato
tomato
Walnut
nut melon
nut
Watermelon
melon
22

---

Under review as a conference paper at ICLR 2023
Table 8: Class Structure for ObjectNet experiments.
Superclass
Subelasses (Original ObjectNet)
(Dress, Jeans, Skirt, jacket, Sweater, Swimming trunks, T-shirt]
garment
[Bath towel, Desk lamp, Dishrag or hand towel, Doormat, Lampshade, Paper towel, Pillow]
soft furnishings
(Backpack, Dress shoe (men), Helmet, Necklace, Plastic hag, Running shoe, Sandal, Sock, Sunglasses, Tie, Umbrella, Winter glove}
accessory
{Coffee/French press, Fan, Hair dryer. Iron (for clothes), Microwave, Portable heater, Toaster, Vacuum cleaner) [Cellphone, Computer mouse, Keyboard, Laptop (open), Monitor, Printer, Remote control, Speaker,
appliance
equipment
Still Camera, TV, Tennis racket, Weight (exercise))
furniture toiletry
(Bench, Chair]
(Band Aid, Lipstick}
wheeled vehicle cooked food produce beverage
[Basket, Bicycle}
(Bread loaf)
[Banana, Lemon, Orange}
[Drinking Cup)
23


Reviews:
[{"review_id": "ug7ETT1YYH", "reviewer": "Reviewer_ohpn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to improve predictions of the CLIP model by utilizing a potential structure/hierarchy among classes. Instead of generating predictions on the target classes only, the authors propose to generate predictions on a set of all target labels’ subclasses (which are more granular) and use predictions in subclass space to decide which target class (superclass) to output.  \nFor finding subclasses the authors use either use an existing hierarchy of labels or generate them by prompting a GPT-3 model.", "summary_of_the_review": "Not so clear how general or practically useful the method is: how specific the results are to datasets/problems with some class structure (whether explicit or implicit). The most significant improvements are when utilizing an existing hierarchical structure of labels but it’s unclear how much of that improvement is specific to the proposed method as no alternative approaches or simple baselines utilizing such information are considered.\n\n**EDIT: Updated the score to lean towards acceptance. The authors' comments are generally convincing and the updated version of the paper makes the scope of the contributions clearer.\nAlso, the additional results provide more insight into the model's performance and in which scenarios it is expected to work well.**", "strength_and_weaknesses": "Strengths:\n\n- (S1) A practical and easy-to-implement/utilize method\n- (S2) In the case of existing class hierarchy, the experiments indicate a very significant improvement over not relying on that class hierarchy (although see W2)\n- (S2) The experiments cover w/ and w/o hierarchical explicit information (although see W1)\n    \n\nWeaknesses:\n- (W1) The paper doesn’t clearly show whether the proposed approach is valid/useful only on datasets where some hierarchical structure between classes exist (regardless whether explicitly present or not) or is something that would work on any generic problem/datasets. The authors do address in their experiments two scenarios: one with hierarchical information provided explicitly and another with a hierarchy generated from a GPT model. However, a good class structure/hierarchy may or may not exist even if it is not explicitly provided or utilized. Taking ImageNet as an example - there are many animal/plant species with a very deep hierarchy, many closely related classes on one hand, but on the other hand, some classes have a very shallow hierarchy and few only loosely related classes (I would suppose maybe classes like “cliff” or “traffic sign”?). The authors however seem to choose datasets where one could expect some hierarchy to exist, even if not explicitly present. How would the method behave on datasets like e.g. StanfordCars where all classes are somewhat similar and it’s unclear if some meaningful hierarchy exists, or even the whole ImageNet where maybe there are groups of classes with a nice hierarchy/structure and groups of classes where such hierarchy might not exist?\n- (W2) The authors present results for their method but no comparisons to any alternative approaches. One would expect at least some simple baselines that utilize the hierarchical structure of labels.\n- (W3) On many datasets the improvement in accuracy is very significant, but on some other, like “living17” or “fruits-360” there’s a relatively much smaller improvement. This is not explained/discussed by the authors - is it something related to some properties of the hierarchies?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qVw9aujdhEq", "reviewer": "Reviewer_oLxZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies zero-shot image classification with class hierarchy.\nA Classification with Hierarchical Label sets (CHiLS) model is proposed to improved classical CLIP model.\nThis model leverages predefined subclasses, and perform CLIP on them first to obtain a set of class embeddings.\nThese subclass embeddings are aggregated together to form the embedding of class of interests that facilitate zero-shot image classification.\n\n", "summary_of_the_review": "The paper improves the previous arts CLIP by leveraging class hierarchy, which can be seen as a modest incremental work.\nUnfortunately, the reviewer does not find it is novel enough.\nBesides, the writing and empirical study can be further improved. \n", "strength_and_weaknesses": "\n- The outstanding problem of this paper is that it is of not self-contained. It lacks of proper definition and review of referred models, terminologies, and problem setup. For example, \"CLIP\", \"prompt\", \"open vocabulary models\".\n\n- The idea of dividing a super class into a set of subclass is not new, and therefore the contribution is not significant enough.\n\n- The sensitivity of the proposed model on the different levels of granularity of the class hierarchy is unclear.\n\n- Lacks of proper comparisons with related works in line of zero-shot image classification.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "p9ls4nw52c", "reviewer": "Reviewer_VtPj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors propose to utilize the label hierarchies to boost the performance of CLIP for zero shot classification. The main steps include the generation of the subclasses for each class by either using the GT label hierarchies or by querying GPT-3, then conduct the CLIP  via these sub-classes, and finally map the sub-classes back to their parents. The performance gain has been observed on the benchmark datasets.", "summary_of_the_review": "Overall, the paper is well-written, and easy to follow, yet the main concern as expressed in the weakness part is the novelty, as it resembles a lot with an ensemble model.\n\nMoreover, if the hierarchical structure does not explicitly exist in the label vocabulary, how would the proposed model handle it, namely, you cannot rely on the ready-to-go GT structure or query it via the GPT model. It is somewhat unfair if the compared models can also utilize these structures to boost their performance. Authors are suggested to clarify their proposed model in a more general sense.", "strength_and_weaknesses": "Strength:\nThe paper is generally well-written with a clear motivation and decent performance gain.\n\nWeakness:\nThe main concern would be the proposed model is essentially an ensemble of sub-class based CLIP models, though authors address this concern via comparisons against linear average approach, the novelty is still somewhat limited.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CSnO-XhaFf", "reviewer": "Reviewer_msk3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides an algorithm to improve zero-shot open-vocabulary classifiers by better prompting them. The authors propose to (1) create sub-classes for each parent classification category using existing (human-created) or inferred (GPT-3 generated) label hierarchies, (2) perform standard zero-shot classification on the sub-classes, and (3) aggregate the results to associate probabilities with the parent category. The paper provides empirical evaluation of the method on various datasets and additionally ablates key design decisions related to their use of labels, GPT-3, and aggregation strategy.", "summary_of_the_review": "The paper provides a method to prompt open-vocabulary models for zero-shot image classification, which leverages class hierarchies. It is straightforward to implement and the authors provide some empirical evidence that their method performs significantly better than the baseline, especially when hierarchies are known for a dataset.\n\nHowever, I currently recommend weak rejection of the manuscript. I am most concerned about the lack of benchmarking on ImageNet (*W3.*), the performance of the method on at least a few more standard classification datasets (*W4.*), the fairness of the zero-shot baselines (*W5.*), and the clarity of the presentation associated with some of the ablations studies (*W7.*, *W10.*).\n\nI am willing to revisit my evaluation during the rebuttal/discussion period.", "strength_and_weaknesses": "**Strengths**\n\n*S1.* The work is well motivated.\n \n*S2.* The work seems novel. I have only seen GPT-3 used in zero-shot open-vocab classification in concurrent work and consider this to be novel.\n\n*S3.* The method is general for classification problems and does not require additional training.\n\n*S4.* The ablations test major design decisions and shed light on when a practitioner may or may not want to make similar decisions. \n\n*S5.* Method is simple and easy to implement.\n\n*S6.* The introduction and method section are very well written.\n\n**Weaknesses**\n\n*W1.* Related work could be more comprehensive and the paper could be better situated in the literature. More specifically, here are some suggestions for references. I suggest doing a more comprehensive literature review of recent work.\n\nTransfer learning:\n* LiT: tuning text towers while keeping language tower fixed (https://arxiv.org/abs/2111.07991)\n* Visual prompt tuning: tuning learnable visual prompt (https://arxiv.org/abs/2203.12119)\n* Model Soups: ensembling many fine-tuned CLIP models in weight space (https://arxiv.org/abs/2203.05482)\n* Patching CLIP models on downstream tasks while maintaining zero-shot performance on tasks where CLIP is already successful (https://arxiv.org/abs/2208.05592)\n* CLIP-CL: similar to Model Patching above (https://arxiv.org/abs/2207.09248)\n* ...\n\nZero-shot prediction:\n* CLIP-ViL: showing the limitation of ZS CLIP in VQA-like settings (https://arxiv.org/abs/2107.06383)\n* CoW: showing how CLIP can be adapted to do object navigation without additional training (https://arxiv.org/abs/2203.10421)\n* ...\n\nAdditionally, I suggest adding a section on hierarchical classification.\n\n*W2.* The paper states: \"Here, we reweight each set of subclass probabilities by its superclass probability. In this way, we can attempt to avoid the behavior in which CLIP makes an incorrect subclass prediction despite a confident and correct superclass prediction, and thus bias our model to never do worse than the raw superclass predictions.\"\n\nWhile this is maybe meant to provide intuition for the method, it reads like a claim that is not supported. I suggest making it clear that this is intuition and not some mathematical/provable property of the algorithm related to \"bias\".\n\n*W3.* How does the proposed method perform on ImageNet? Without ImageNet evaluation it becomes hard to compare the proposed method to other methods.\n\n*W4.* It seems many of the datasets are chosen because it is easy to construct class hierarchies. However, the method is presented as a general purpose zero-shot image classification improvement over CLIP. To be convinced of this, I would be interested to see results on more datasets. Please consider Cars, DTD, EuroSAT, GTSRB, KITTI, MNIST, RESISC45, SUN397, which Ilharco et al. (https://arxiv.org/abs/2208.05592) use because they are tasks ZS CLIP is known to struggle on. Note: dataloaders for these datasets can be found [here](https://github.com/mlfoundations/patching/tree/main/src/datasets).\n\n*W5.* The main experimental setup has baselines that may not be fair points of comparison. Specifically, not all datasets in the original CLIP paper use the ImageNet prompt templates. For example, CIFAR-10/Food101 use different prompts (see [here](https://github.com/openai/CLIP/blob/e184f608c5d5e58165682f7c332c3a8b4c1545f2/data/prompts.md)). What are the deltas between the 75 prompt set and the set that OpenAI provides? For Food101 specifically, there seems to be a discrepancy of ~2% between the ZS number reported in the manuscript and the L/14-336px number reported in the CLIP paper in Tab 11. For datasets that OpenAI did not evaluate in their original paper, some prompt engineering on a val set or some validation that ImageNet prompts are reasonable seems needed.\n \n*W6.* It seems that the experimental setup is sufficiently different in cases where the hierarchy exists and when it does not for me to compare the performance between col 2 and 3 in Table 1. For example, in the *GPT-3 map* the authors include the superclass label in the label set, while in *Existing Map* they do not. It would be good to standardize the algorithm being compared in the two settings. Another idea is to provide the full results in the appendix (i.e., *GPT-3 map* w/ superclass labels, *GPT-3 map* w/o superclass labels, *Existing map* w/ superclass labels, *Existing map* w/o superclass labels).\n\n*W7.* I am not able to completely understand the experimental setup for “Noisy Available Hierarchies” from the text provided. Consider adding an appendix for that section to give more details on how the subclasses are constructed from the ImageNet hierarchy.\n\n*W8.* How do things look if the label set size is 1 or 100 (i.e., m=1 or m=100)? Doe these more extreme values of m affect the performance? At least m=1 should be worse than zero-shot, which seems like a valuable bases of comparison.\n\n*W9.* Some visualizations of cases where the vanilla zero-shot and CHiLS models disagree could be helpful to provide intuition.\n\n*W10.* I am not clear on the linear class ensemble experiment from reading the text. Is the following correct? For a single class, loop over all subclasses and ImageNet prompt templates computing the CLIP text features. Average all of these features to represent the single class as a feature. Repeat for all classes to get a zero-shot classification head.\n\n*W11.* The method involves expanding the zero-shot head at test time. There is a compute and memory overhead associated with this that may limit the scalability of the method when many classes or subclasses are targeted. For example, for the 1000 ImageNet classes, with m=10, the instantiated head would have 10k classes.\n\n**Minor**\n\n*M1.* The Radford et al. 2021 caption example is “a photo of a {}.” not “a photo of a {}” as presented in the manuscript. The difference is the period at the end of the prompt, which may make a difference in downstream performance.\n\n*M2.* This is a relevant reference when discussing CLIP confidence in the method section: https://arxiv.org/abs/2106.07998\n\n*M3.* For Sec. 4.4 is there some theory as to why weighting is necessary in one case and not in the other?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject