ChatGPT-Powered Hierarchical Comparisons for Image Classification
Authors: Zhiyuan Ren
Yiyang Su
Xiaoming Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: ChatGPT
Hierarchical Comparisons
Image Classification
Zero shot
TL;DR: A training-free, explainable, and effective zero-shot image classification method with enriched hierarchical descriptions powered by LLMs.

Abstract: The zero-shot open-vocabulary setting poses challenges for image classification.
Fortunately, utilizing a vision-language model like CLIP, pre-trained on image-text
pairs, allows for classifying images by comparing embeddings. Leveraging large
language models (LLMs) such as ChatGPT can further enhance CLIP’s accuracy
by incorporating class-specific knowledge in descriptions. However, CLIP still
exhibits a bias towards certain classes and generates similar descriptions for similar
classes, disregarding their differences. To address this problem, we present a
novel image classification framework via hierarchical comparisons. By recursively
comparing and grouping classes with LLMs, we construct a class hierarchy. With
such a hierarchy, we can classify an image by descending from the top to the bottom
of the hierarchy, comparing image and text embeddings at each level. Through
extensive experiments and analyses, we demonstrate that our proposed approach is
intuitive, effective, and explainable. Code will be released upon publication.

ChatGPT-Powered Hierarchical Comparisons for Image Classification
Zhiyuan Ren, Yiyang Su, and Xiaoming Liu
Department of Computer Science and Engineering, Michigan State University East Lansing, MI 48824
(renzhiy1, suyiyan1, liuxm] @msu.
Abstract
The zero-shot open-vocabulary setting poses challenges for conventional image classification. Vision-language models pretrained image- pairs like CLIP offer a solution based on comparing image and class label embeddings. Incorporat- ing class-specific knowledge provided by large language models (LLMMs) such ChatGPT in descriptions can further enhance CLIIP's accuracy. However, CLIP still exhibits bias towards certain classes and generates similar descriptions for closely related but different classes. To address these problems, we present novel image classification framework via hierarchical comparisons. By recursively comparing and grouping classes with LLMs, we construct a class hierarchy. With such a hierarchy, we can classify an image by descending from the top the bottom of the hierarchy, comparing image and text embeddings at each level. Through extensive experiments and analyses, we demonstrate that our proposed approach is intuitive, effective, and explainable. Code is available here
1 Introduction
Conventional image classification approaches typically evaluate their performance on the same set of categories their training data. However, this evaluation paradigm fails capture the challenges in real-world scenarios, where classes in the test set are not overlapped with the training set, i.e. zero- shot, and the test set may comprise an unknown number of classes, i.e. open-vocabulary. Zero-shot open-vocabulary image classification, on the other hand, accommodates the open-ended nature of class diversity in practical applications, thereby serving as a more accurate proxy for real-world generalizability. and is the focus of this paper.
Through training with aligned image-text pairs, CLIP 37 generate embeddings for both seen and unseen classes. To apply it in zero-shot open-vocabulary image classification,, one simply obtains an image embedding from CLIP's image encoder and compares it with the text embeddings of all label candidates. Recently, Menon and Vondrick (29] propose simple yet effective method, classification by description. They use LLMs generate text descriptions for unseen class labels and compare the image embedding with the embeddings of the descriptions. Their method exploits the semantic information offered by LLMs bolster the efficacy of CLIP image classification.
Nevertheless, their classification accuracy is still limited by a few factors. Firstly, in image classi- fication, CLIP is biased towards some classes in classification [45). In practice, observe that CLIP fails to distinguish among semantically similar categories, and often misclassifies a few classes specific class. For instance, in the CUB [43) dataset, the great-crested flycatcher, the least flycatcher, and the yellow-bel belliedlied flycatcher are misclassified by CLIP as Arcadian flycatcher most of the time. Classification by description [29] does not address this issue because the descriptions for these classes are similar. Secondly, descriptions generated by LLMs just from the labels are generic
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

"tiger"A
"vertical stripes"
"cat"
"slender"
"lion"
"tiger"
"agile"
"cat"
"solitary"
"lion"
"short tails"
"graceful" "muscular"
"bush tail"
"tawny"
Figure 1: A toy example of classifying felid. Top left: CLIP 37 only uses the class labels. Bottom left: Menon and Vondrick (29) expand the space of each class by using text descriptions. But their text descriptions are not dependent on other class so their descriptions are suboptimal for classification. Right: We generate text prompts by comparing different classes to reduce inter-class similarity and make the decision boundary of each class more compact.
and unsuitable for discriminating class from its semantic neighbors. As shown in Fig. the LLM generates "agile" the description for the classes cat and tiger, "graceful" for cat and lion, and 'muscular' for lion and tiger. These descriptions are not helpful for classifying images of cat-like animal because all of them are shared by two classes.
In light of these problems, we propose a novel framework for zero-shot open-vocabulan vocabularray image classification. Inspired by how human beings hierarchically organize visual and linguistic concepts in the open world, our method organizes the classes in the test set into hierarchy with the help of LLMs.. A hierarchy is preferable for classification because different levels of classification require different features descriptions. Telling tiger from a car nowhere near telling a tiger from a cat. Therefore, when we build the hierarchy and perform inference, we only compare classes or groups of classes within the level or nodes where they belong.
Building the hierarchy without human interaction is possible because the state-of-the-art LLMs have common sense and some expert knowledge in, egg., ornithology. In our approach, under the scope of wrens. LLMs describe that house wren has "a coiddish-boww coloration with fine streaking on their back and wings" and winter wren has "a dark brown coloration with a mottled appearance on their back and wings". : With such discriminative text descriptions for each class, when differentiating between house wrens and winter wrens, the CLIP classifier will specifically examine the color tone of the bird and the texture on the birds' backs and wings, which is both intuitive and logical. Additionally, we will be able to explain the grounds on which the classifier makes decisions by looking at the descriptions with the highest score at each level of the hierarchy.
Through comprehensive evaluations, show that our performance is better than the existing method, Menon and Vondrick (29). and the vanilla CLIP (37) image classifier across various backbones and datasets, which showcases the effectiveness of the proposed hierarchical comparison approach.
In summary, the contributions of this paper include
We leverage prior knowledge the LLMs to construct knowledge trees by iterative grouping and comparing, Combined with CLIP 37 we can obtain a hierarchical and comparative text representation.
V We achieve state-of-the-ee performance in zero-shot image classification different datasets and backbone settings. Without additional training, we can gain appreciable increases of up to 10%.
Moreover, we show the strong explainability of our model brought by the hierarchical descriptions. It can tell why the model chooses category the prediction and which layer of description, the model makes the choice.

---

2 Related Work
have witnessed significant advances, revolutionizing natural language processing tasks with hundreds of billions of parameters and special tricks such as Reinforcement Learning from Human Feedback. Researchers have started to harness the generation powers of LLMs. Frozen [4T] and PICa [47] apply LLMs in visual question answering. Similar to our approach, Menon and Vondrick (29) directly generate textual descriptions from labels assist language-vision models in image classification. However, discussed above, their approach suffers from a lack of discriminability among semantically similar classes.
LLMs.
Recently, LLMs 1
20 25 34
40 46
49
Contrarily, embed ChatGPT into hierarchical framework for image classification, starting at a high or coarse level (e-g., animals vs. vehicles) and going down into the realm of low-level or fine-grained classification (e.g., a house wren vs. winter wren). ChatGPT will be prompted to compare object classes at different levels so that it can focus on different class-defining aspects at the respective level.
Language- Language-Vision Vision Pretraining. CLIP 6 to learn single embedding space for both image and text using contrastive learning. bridging the gap between the two modalities. CLIP has been widely adopted to text to guide image {8 32 35
14 37 and other vision-language models (VLMs) manage
and
3D generation
[17 30 38). In particular, with an image and a textual description of the target class as input, CLIP can associate visual and textual representations make predictions without explicitly training on that specific class. This enables classifying images into unseen classes based on semantic
[1]
relationships between images and text. In our approach, we take advantage of CLIP map both the input image and GPT-generated distinguishing texts to their embedding space.
Text Prompting. Text prompting can be generally divided into soft prompting and hard prompting. Compared soft prompting 5 21 [2 36). which refers to using learnable token as the prompt, hard prompting (5) gets prompts by filling phrases into preset templates. Soft prompts have demonstrated the ability to achieve commendable performance in image classification [3] 5
However, hard prompts could be preferable because they do not require any training and offer better explainability. Our approach belongs to the group of hard prompts, thus inheriting the advantages associated with such prompts while not sacrificing performance compared to soft prompts.
refer to the organization of visual information in a hierarchical manner, where higher-level concepts or features are built upon lower-level ones. It captures the inherent hierarchical structure and relationships present in real- world tasks, allowing for more efficient and discriminative processing. This concept draws inspiration from the human brain's hierarchical organization (28), where lower-level neurons form simple patterns, which are then combined to form higher-level representations. HGR-Net (48 organizes the classes with the help of WordNet (31) while we exploit the prior knowledge of L.Mss to produce descriptions based on other classes. Plus, their network needs to trained but we only require pretrained VLMs. Like approach, CHiLS (33] also uses LLMs and does not require training, but relies solely hierarchical labels and does not leverage class-specific descriptions or comparisons.
Hierarchical Representations. Hierarchical structures 15 23
3 Methodology
3.1 Problem Definition
Given an input image I, we attempt to assign to the correct category 4 among an arbitrary set of categories (ci, 62 Ca, on which the model has not been trained for classification. One of the most common methods (37) is to use a pretrained VLM to map the image and each class to the aligned latent space and select the pair that has the highest cosine similarity with the image embedding.
E.(I,
(1)
t; E,(c),
(2)
pred = arg max(x
(3)
3

---

Text CLIP Encoder
Compare
6
E-Lage
200
Image CUP Enooder
Ccean
Laptop
Final Score
Image Embeddings
Multi-lere Score Fusion
Text Embeddingge
Tiger
Parson
Dog Arrunde doil amredd steped
<riqu cou
wall he
con ofty wit amber color sywa
coal smooth Kiii varing ofe
kee
Figure Illustration of architecture. Given of labels, alternately employ grouping and LLM's generated descriptions build knowledge trees. Grouping provides more detailed scope for generating descriptions, and generating descriptions provides more distinguished embeddings for grouping. Upon building the knowledge tree, we compute the similarity score between the hierarchical description and the image embedding to obtain the final classification score.
where F.g and F, are the vision encoder outputting the image feature x and the text encoder outputting the text feature t, respectively. In Eq. 2 the text encoder can take either simple template- -aapete--based prompt like "a photo of [category name I" [37] or a learnable prompt (13).
In spite of the promising results of the existing prompt generation methods, we aim to leverage the abundant prior knowledge in LLMs build class hierarchy. The overall approach is inspired by how people perceive and recognize objects in the real world where classes are hierarchically grouped and compared each other.
3.2 Building Knowledge Tree by Grouping and Comparing
To recognize objects in the real world. humans often follow coarse-to-fine and hierarchical approach. For example, to recognize a husky dog, one can follow a classification process by grouping it into the animal kingdom. Subsequently, based specific characteristics such as the shape of its nose and the texture of its fur, it can be further classified into the dog family. Finally, within the dog family, a feature comparison can be conducted with other dogs to identify their specific breed category.
Within this cognitive process, two fundamental operations play a pivotal role: grouping and comparing. In the proposed methodology, we initially consolidate similar categories into one unified group. Subsequently, we leverage the knowledge derived from the LLM to acquire comparative descriptions pertaining to distinct categories within the same group. Building upon the improved features, we can then perform a successive round of grouping, iteratively exccuting the above steps. The architecture is shown in Fig.
Specifically, following 29). utilize ChatGPT to generate initial descriptions for each object under consideration and employ the pretrained CLIP text encoder to map these descriptions into the latent space. Using these features, we invoke clustering algorithm, e.g., k-means [27). for grouping. Even with extensive prior knowledge of LLMs, comparing a large number of categories (> 10) presents a formidable challenge, as it needs to generate comprehensive description matrix that grows quadratically with the number of categories. To address this issue, we employ tailored strategies for generating descriptions based on the group size, allowing for more efficient and effective comparisons. We adopt summary-base comparison for larger groups and direct comparison for smaller ones.

---

D...... Dv}. where N is the number of test classes, D, IR M,C is the hierarchical text embeddings for category i, the dimension the text embedding. Summary-based Comparison. Given the substantial number of categories involved, we can leverage the capabilities of the LLM summarize the overarching characteristics of the group (see example below). By incorporating these summarized characteristics into new query, we can generate a comparative description that captures the distinctive features and relationships among the categories. This approach enhances our ability to effectively compare and contrast the elements within group.
Q: Summarize the fall Tollowing categories with one sentence: (category listy?
A: {subset description>
Q: what are useful features for distinguishing (category nane) in dataset: {subset description)?
A:
Direct Comparison. When the number of elements within group is relatively small, a direct query to the LLM enables us to obtain an exceptionally comparative description (see example below). This approach capitalizes the LLM's capabilities provide comprehensive and insightful analysis, enhancing the comparative understanding of the elements within the group.
q: what are useful features for distinguishing <target category> from {other categories> in a photo?
Utilizing these detailed and comparative descriptions, we
T_initin, di
minibatch of initial text embeddings CLIP Text Eacoder
Textt_nnodde LLM (LMpproop)
can achieve more nuanced text
ChatGPT model
embedding. Subsequently, we
e Targer: collect tree-level deseription
continue to perform clus-
def build tree Jaop emb)::
tering, followed by additional comparisons to derive new de-
new_groups K- X-meaar(T__mb))
group groups: (group)
scriptors.
a Skip processing single-item group
As shown in Fig. this pro- cess recursively executed un- til the number of leaf nodes he- comes fewer than predefined threshold. At this point, successfully generate the en- tire knowledge tree. Within narrower group range, we uti- lize queries to the LLM com- pare classes within the group, generating more comparative descriptions. Unlike [29). our method ensures that the de- scription is adaptive to differ- class sets.
elif len (group) thres:
Generate more fine ggrained descriptions summary LIM 'summarize the following categories (...)))
descriptions
group. summary) collect( (dsecriptinns)
A Keep Grouping into smaller ones Text_eneeder [exx_eeeeder(ddserippe (deserrpit))))) [ili__tre__n_ooopCC
clif 1 < Ten [gnogr) <- thres:
comparative descriptions
descriptions 1IM ccmparr the following categories G.))")
collect ( (descriptions)
Starting tree_ loop (oopoooooo- (T_emb-T_inni)
Figure 3: Psuedo-code for building the knowledge trees.
3.3 Knowledge Tree Traversal and Multi-level Score Fusion
After building the knowledge tree, we can have text features at multiple levels of granularity, which represented D (D1,
and M, is the number of comparative descriptions for category i depending the tree structure. Given the image feature x, the fused score s can be expressed as:
q()) = x (D.)'
(4)
r(x, i) = g(l) Σ Mr
j= OH la(k+1) ((k) +7]
(5)
1+ II 1g(k +1) > (k)) +7]
s(x, = (1 (,) xx((xx)))
(6)
Here, 4(j) calculates the cosine similarity between the image embedding x and the j-th text embed- ding of category r(x,3) calculates the running average of the longest sequence of monotonically
5

---

increasing values. The final similarity score s(x, i) is weighted average of the cosine similarity between the image embedding x and the i-th class label embedding t; and r(x, i).
In this fusion method, the new score is merged only when encountering a higher score in the subsequent node compared the current node. This addresses the ambiguity issue that arises from excessively detailed descriptions. For instance, when comparing a yellow sparrow and a red sparrow, the most comparative description might include a specific detail such "yellow coloration across the full body". However, such detailed description could inadvertently increase the scores of unrelated objects like banana or wood. To mitigate this issue, our fusion method ensures that the subsequent scores will be dropped if the current description has low score, such as "a short and wide bill" for a banana or wood. Besides, our fusion method takes into account the score of the initial description as an offset, and we introduce a constant hyperparameter À [0, 1] to balance the two terms according to Eq. [ r is the tolerance for score reduction, which is close to
4 Experiments
4.1 Implementation Details
There are four hyperparameters in our method, including the number of groups N in the k-means algorithm [27]. the threshold for leaf nodes, the weight À assigned to score offset, and the tolerance for score reduction. N exhibits an inverse proportionality relationship with the number of labels and X is roughly determined by the inter-class similarity of the dataset. We generally set to 2 or 3 and 7 to O. our approach and baseline setting, we employ CLIP ViT-L/14 for encoding the initial text embedding and leverage ChatGPT to handle all text completion tasks. On single Nvidia RTX A6000 GPU, it feasible replicate all the results of our paper within approximately two hours.
4.2 Quantitative Results
Unlike many prior works that require fine-tuning for downstream tasks, our approach eliminates the need for any training in the zero-shot setting. Instead, we simply enrich the text information by incorporating hicrarchical description for each category. In line with the methodology employed by [29]. we expand the ImageNet validation by introducing two new categories, each containing five additional images.
We conduct experiments on six different image classification benchmarks, i.e. ImageNet (10), CUB [33), Food101 [2] Place365 [26), Oxford Pets (50), and Describable Textures (7). using two baseline models: CLIP (37) and OpenCLIP (6. For CLIP, we evaluate its performance with four backbones (Res-50 [16), ViT-B/32, ViT-B/16, and ViT-L/14 (II), while for OpenCLIP,
use three backbones (ViT-B/16, ViT-L/14, and ViT-G/14). When generating the class hierarchies, we fix the ratios of the number of clusters the class number of each dataset throughout all experiments.
As shown in Tab. D our proposed hicrarchical comparison framework can improve upon the baseline methods of image classification across various datasets and backbones. This demonstrates that our prompts generated from comparing classes hierarchically are more suitable for zero-shot image classification. Moreover, it is evident that our approach is robust and generalizable to different zero-shot image classification settings.
There is an isolated exception- the proposed approach does not improve over OpenCLIP 6 on the Food101 dataset. Although, we note that our approach outperforms OpenCLIP as baseline on all other datasets and CLIP on Food101. We speculate that since OpenCLIP uses different training data from CLIP, its image and text encoders may generate indistinguishable embeddings for foods. In fact, following the reported results in (39) Food101 is the one of few datasets where CLIP has higher accuracy than OpenCLIP under the same backbone, which justifies our speculation to some extent.
On the CUB dataset, the margin between our approach and the baseline is relatively smaller. We attribute this to the fact that both CLIP and OpenCLIP lack relative perception capability. Their encoders, while being sensitive to features such as colors and textures, cannot comprehend the comparatives. The semantic difference between large and larger, long and longer, and yellow and darker yellow is not reflected in their embeddings. This phenomenon in turn stems from the text- image pairs used for training CLIP and OpenCI.IP. As the CUB dataset contains exclusively birds, the differences between classes are more fine- -raaiidd.dd Consequently, the comparisons would more

---

Table 1: Zero-shot classification accuracy gains CLIP and OpenCLIP category embedding baseline.
see significant increases across all the settings except Food101 with OpenCLIP.
Dataset
ImageNet Ours
CUB
Food101
Architecture
CLIP
CLIP
Ours
A
CLIP
Ours
Res-50
54.85
60.63
+5.78
49.02
49.86
+0.74
71.61
75.44
+3.83
ViT-B/32
58.47
63.88
+5,41
52.23
54.18
+1.95
78.83
83.02
+4.19
VIT-B/16
63.53
68.95
+5.42
56,66
59.25
+2.59
86.17
88.13
+1.96
VIT-L/14
76.60
79.64
+3.04
63.46
65.45
+1.99
91.86
93.17
+1.31
Dataset
Places365
Oxford Pets
Describable Textures Ours
Architecture
CLIP
Ours
CLIP
Ours
CLIP
Res-50
32.48
36.83
+4.35
74.60
79.99
+5.39
37.63
45.67
+8.04
VIT-B/32
37.23
40.73
+3.50
79.80
82.69
+2.89
40.82
49.19
+7.37
VYT-B/16
38.21
41.52
+3.31
81.68
87.19
+5.51
43.56
51.26
+7.70
VIT-L/14
38.65
41.13
+2,48
87.92
92.84
+4.92
51.01
58.04
+7.03
Dataset
ImageNet Ours
CUB
Food101
Architecture
OpenCLIP
OpenCLIP
Ours
OpenCLIP
Ours
A
VIT-B/16
62.96
67.08
+4.12
66.81
68,20
+1.39
84.85
84.59
-0.26
VIT-L/14
68.34
72.69
+4.35
75.32
75.80
+0.48
89.40
89.35
-0.05
ViT-G/14
70.94
75.90
+4.96
84.19
85.19
+1.00
93.41
92.68
-0.73
Dataset
Places365
Oxford Pets Ours
Describable Textures Ours
Architecture
OpenCLIP
Ours
OpenCLIP
A
OpenCLIP
A
VIT-B/16
38.80
42.52
+3.72
84.03
87.47
+3.44
46.72
54.01
+7.29
VÍT-L/14
38.46
44.12
+5.66
87.13
89.62
+2.49
51.69
61.79
+10.10
VIT-G/14
41.20
45.61
+4.41
92.37
95.07
+2.70
63.70
69.69
+5.99
Table 2: Comparsion zero-shot SoTA method under fair settings. Overall, we gain ~1% improvement across different backbones and datasets.
Backbone
Method
ImageNet
CUB
Food101
Place365
Pots
Texture
Mcan
29
60.20
49.10
74.94
35.79
78.58
44,63
57.20
50
Ours
60.63
49.86
75.44
36.83
79.99
45.67
58.07
29
63.48
53.80
82.79
40.15
81.60
45.69
61.25
ViT-B/32
Ours
63.88
54.18
83.02
40.73
82.69
48.19
62.12
29
68.43
58.85
87.68
40.87
86.56
49.15 15
65.25
VIT-B/16
Ours
68.95
59.25
88.13
41.52
87.19
51.26
66.05
29
75.36
65.02
93.00
40.13
92.10
55.95
70.26
ViT-L/14
Ours
79.64
65.45
93.17
41.13
92.84
58.04
71.71
often rely on comparing the same attributes, which is not one of the strengths of CLIP or OpenCLIP. For more experiments and analyses please refer to the supplementary materials.
Additionally, we compare our method to the oftte-of-the-ssd arr zero-shot approach. To ensure fair comparison, we employ the same model weights for each distinct backbone setting. Moreover, we utilize same LLM, ChatGPT (3). to generate descriptions, where the different factor lies in our hierarchical description structure. As shown in Tab. [ our approach can consistently outperform the current state-of-the-an method in zero-shot classification.
Notably, we surpass the state of the art (29) on Describable Textures by considerable margin. This because, without the information other classes as the context, LLMs generate class descriptions that lack discriminability. ChatGPT (3] characterized the gauzy class "a thin, translucent fabric" with "a light, airy feel" and "a delicate or lacy appearance" and the lacelike class as an "intricate, detailed pattern" "often used for decoration". Even though correct, these descriptions do not provide any discriminative information, which renders them ineffective in classifying the two textures. the other hand, in our hierarchy, gauzy and lacel ike are two leaf nodes with a common parent. When comparing them, our approach describes the gauzy class as "a more transparent or sheer appearance" and "a more delicate texture with softer hand-feel" and the lacel ike as "a lightweight

---

25.50
21.44
FARNAC
27.08
25.33
2000
27.32
21.03
24.11
20.37
27.32
desciption
37.2%
$7.99
COTO INTh Seary
2114
26.22
184
2732
26.02
26.33
37.33
27.05
2637
26.96
$7.00
Coro
1735
3332
26.33
71
75.91
26.99
26.4
27.13
19.0
20.12
28.00
34.49
Dhe
21.10
AMEAAE
34.64
27.50
26,95
27.23
27.56
18.00
26.11
18.2
29.77
27.06
33.56
25.22
Figure 4: We show two inference examples and their score distribution by bicrarchically querying the description with the image embedding, The description corresponds to the left input image and the right image is the sample from the confusing categories relative to the left onc (zoom in for small fonts).
and almost transparent appearance" which has "visible holes or gaps throughout the material". Such descriptions accurately capture the subtle differences between these two materials and facilitate zero-shot classification between these two categories, which manifests in the superiority of our proposed hierarchical comparison approach for zero-shot image classification.
4.3 Interpretability and Analysis
Fig. illustrates two groups of examples that demonstrate the inference and score calculation in method. Specially, we input two images and compute their similarity the hierarchical descriptions associated with the left image. Due to the weak comparative information in the early descriptions, similar images tend to yield similar scores at the beginning. As progress towards more comparative description, the disparity between the scores of the two images gradually increases. Besides, this also highlights the interpretability of method, regarding the attributes that elicit stronger responses to the input. Additionally, it indicates the specific layer of description at which the scores of different images diverge. widening the gap.
it provides the user insights
In order to further investigate the factors contributing to the improved performance of our model, we conduct a comparison of the confuusio matrices between our model and CLIP, using the same backbone. As shown in Fig. 5) the number of cases where Ragdolls is misclassified as Birman is reduced from 54 to 37, as contrasting descriptions bring out distinguished text features.
Fig. 6 illustrates the step-by-step progression of our comparative hierarchical description, pushing the image embedding closer to its corresponding ground-truth text embedding. Simultaneously, this process gradually distances the confusing label from the image embedding. We visualize this process using t-SNE [42 reduction to two dimensions, while also regularizing them into unit vectors.
8

---

Birman
Ragdoli
Pephili, riuifyy
(opell mut to lerg, sirkking bow on
Thes
Bormin wiich by oraaggtier alond
Eirnan, bas azogetly tilted shappd oyex
Predicted Labels
Mimu
Confusion Natrix
Our Confusion Natrix
Figure 5: We highlight the difference between the confusion matrices at the position where Ragdoll samples are misclassified into Birman. At the right part, show the comparative deseriptions which explain where the gain comes from.
Figure T-SNE visualization similarity between the image emhedding and the text emhedding. Row shows how our hierarchical descriptions help pull the image emhedding closer to the groudtruth text embedding: Row 2 shows how our hierarchical descriptions push the confusing text embedding away from the current image embedding. / represents the image embedding: / represents the text embedding (groundtruth in Row confusing category Row 2): Other points represent remaining text embedkings the latent space.
4.4 Ablation Study
We ablate on four distinct hyperparameters: N, depth, and T utilizing a ViT-B/32 architectural
foundation. Metrics derived from mean accuracy employed for À and r across datasets including
ImageNet 10). CUB (43), Food101 (2) Place365 (26) Oxford Pets [50). and Describable Textures 7
Alternatively, depth and N are exclusively evaluated based ImageNet (T0) performance.
Score Momentum X. As illustrated Fig. 7a there exists a direct relationship between increasing À values and performance enhancement. However, solely relying scores from the proposed methodology without offset utilization results decline in performance. Optimal performance is achieved when À is set to 0.9.
Group Number N. Larger group numbers, such as 8, yield rudimentary hierarchies and generalized descriptions, whereas smaller values extend the hierarchy's 's development phase. Represented in Fig. 7b it reveals that moderate increases in accuracy corresponded to fourfold evaluations. Inversely, sixfold acceleration observed, albeit between 4 to and directly correlates with the class count.
of 1. 1.64% dip in accuracy. Typically, N oscillates
The Depth the Prediction. Incorporating average aggregation advantageous for score com- putation across variable depths. Results presented in Fig. 7cc when evaluated on ImageNet with ViT-B/32, underscore the superior performance the next layer over its predecessor, attributed to the incorporation of divergent descriptions. A successive rise in performance is evident with depth augmentation.
Tolerance for Score Reduction r. As delineated in Fig. 7 the evolution of 7 from to maintains nearly consistent performance, peaking at 1.0. Although optimal performance is observed at 7=1, marginal variations across datasets necessitate the retention of this hyperparameter.

---

Lambda Abalation
Impact of Group Number N
64.5
Mean Accuracy
64.0
N=6
Nwd
Accuracy
63.5
63.0
62.5
61.4
62.0
N-8
0.8
1.0
20
80 100
Lambda
Hlerarchy Construction Time (Hours)
(a) Ablation study of A. We fix as O, use ViT-B/32 as backhone, and compute the mean accuracy across six datasets.
(h) Ablation study of N. An ablation the ImageNet with ViT-B/32 for varying N val-
ucs follows.
Depth Ablation
Ablation
64
63
Accuracy
62.0
Accuracy
62
61
61.6
60
59
61.2
2
0.0
0.2
0.4
0.6
0.0
1.0
Depth
7
(c) Ablation study Depth. Average ag- gregation is used in computing scores across various depths on ImageNet.
(d) Ablation study r. We fix as 0.9, use ViT-B/32 as the backhone, and compute the mean accuracy across six datasets.
Figure 7: Ablation study across different hyperparameters.
4.5 Failure Cases
By comparing confusion matrices, we find that there are indeed some classes whose classification accuracy decreases when using descriptions greater depths. We pick representative classes and their descriptions of where they have lower classification accuracy than only using the previous depth.
We observe that when ChatGPT differentiates between closely related classes in its deeper layers, often references attributes such length and size, which be challenging to represent visually. Consider the statement: "Newfoundland dogs are typically larger than Chows. Male Newfoundlands can weigh up to 150 pounds and reach heights of 28 inches, whereas Chows generally weigh between 50-70 pounds and stand 20-25 inches tall." The CLIP model's 's encoder, however, struggles interpret this length information accurately due to variations in camera angles and positions. This observation suggests that detailed text descriptions might pose comprehension challenges for CLIP. Thus, enhancing the synergy between LLMs and CLIP presents itself as promising avenue for future research.
5 Conclusion
In this work, we propose a training-free. explainable, and effective zero-shot image classification method with enriched hierarchical descriptions powered by LLMs, such as ChatGPT. The know mowledge tree constructed by alternately grouping descriptions and generating descriptions allows to achieve SoTA in different settings. A central theme our work is to inspire the field by emphasizing that LLMs possess the potential to enrich semantics and make them useful for contemporary machine learning tasks. How to extract semantics worth expanding in the current machine learning tasks is the key to exploring the potential of L.MMs.
Limitations. Our approach is unfortunately still limited by some limitations of CLIP. Although we have good performance overall, as discussed in Sec. 4..2 our approach is hindered by the inability of CLIP to reason about relative relations in extremely fine- ime-grained classification tasks. In addition, the entire approach based on the premise that natural language descriptions can helpful for image classification given enough context. Our experiments demonstrate that this is the case in general, but exceedingly nuanced scenarios, the differences between different classes might be too intricate to be described by natural languages.
10

---

References
[1] Stella Biderman, Hailey Schoelkopf. Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan. Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth. Edward Raff, et al. Pythia: A suite for
analyzing large language models across training and scaling. arXiv preprint 1xXiv2334.00337. 2023.
[2] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 1ad--10--ma 101-minnni discriminative components with random forests. In ECCV, 2014.
[] Tom Brown, Benjamin Mann. Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamerr.
In NeurIPS, 2020.
[4] Ming-Yes Yen Chen, Mahdin Rohmatillah, Ching-Hsien Lee, and Jen- len-Tzung Chien. Meta learning for domain agnostic soft prompt. In ICASSP, 2023.
[5] Xiaoqing Chen, Chengyu Wang, Junwei Dong, Minghui Qiu, Liang Feng, and Jun Huang. Boosting prompt-based few-shot learners through out-of-domain knowledge distillation. In ICASSP, 2023.
[6] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-inage learning. arXiv preprint arXiv:2212.07143 2022.
[7] Mircea Cimpoi. Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
[8] Katherine Crowson, Stella Biderman, Daniel Korii, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vggan-clip: Open domain image generation and editing with natural language guidance.
In ECCV, 2022.
[9] Dehayan Deb, Xiaoming Liu, and Anil Jain. Unified detection of digital and physical face attacks. In FG, 2023.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datahase. In CVPR, 2009.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and words: Transformers for image recognition scale. In ICLR,
Neil Houlsby. An image is worth 16x16
2021.
[12] Danny Driess, Fei Xia, Mehdi S. M. Sajaadi, Corey Lynch, Aakanksha Chowdhery, Brian Iehter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang, Yevgen Chehotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine. Vincent Vanhoucke, Karol Hausman. Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.0 03378, 2023.
[13] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprini arXiv:2110.04544, 2021.
[14] Gabriel Goh, Nick Cammarata t. Chelsea Voss t. Shan Carter. Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021.
[15] Xiao Guo, Xiaohong Liu, Zhiyuan Ren, Steven Grosz, Iacopo Masi, and Xiaoming Liu. Hierarchical fine-grained image forgery detection and localization. In CVPR, 2023.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, 2016.
[17] Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip:
Zero-shot text-driven generation and animation of 3d avatars. ACM TOG, 2022.
[18] Boyu Hou, Chengyu Wang, Xiaoqing Chen, Minghui Qiu, Liang Feng, and Jun Huang. Prompt-distiller: Few-shot knowledge distillation for prompt-hased language learners with dual contrastive learning. In ICASSP, 2023.
[19] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprin! arXiv:2302. 1404443,0..1444,, 2023.
[20] Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasumuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12001 12017, 2022.
[21] Feihu Jin, Jinliang Lu, and Jiajun Zhang. Unified prompt learning makes pre-trained language models better few-shot learners. In /CASSP, 2023.
[22] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionelip: Text-guided diffusion models for robust image manipulation. In CVPR, 2022.
[23] Qinbin Li, Zhaomin Wu, Yanzheng Cai, Yuxuan Han, Ching Man Yung. Tianyuan Fu, and Bingsheng He. Fedtree: A federated learning system for trees. In ICMI,, 2023.
[24] Yaojie Liu, Joel Stehouwer, Amin Jourabloo, and Xiaoming Liu. Deep tree learning for zero-shot face anti-spoofing. In CVPR, 2019.
11

---

[25] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou. Quoe V Le. Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.
2023.
[26] Alejandro López-Cifuenter Marcos Escudero- Vinolo, Jesús Bescós, and Álvaro Carcia-Martia. Semantic-
aware scene recognition. Puttern Recognition, 2020.
[27]
MacQueen. Classification and analysis of multivariate observations. In 5th Berkeley Symp. Math. Statist. Probability, 1967.
[28] Damiel S Margulies, Satrajit Ghosh, Alexandros Goulas, Marcel Falkiewicz, Julia M Huntenburg, Georg Langs, Gleb Bezgin, Simon B Eickhoff, F Xavier Castellanos, Michael Petrides, et al. Situating the default-mode network along a principal gradient of macroscale cortical organization. PNAS, 2016.
[29] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2023.
[30] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mes:: Text-driven neural stylization for meshes. In CVPR, 2022.
[31] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 1995.
[32]
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever. and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
[33] Zachary Novack. Saurabh Garg. Julian McAuley, and Zachary Lipton. Chils: Zero-shot image classification with hicrarchical label sets. In ICML, 2023.
[34] OpenAI. GPT-4 technical report. arXiv preprint w7Xiv:2303.08774 2023.
[35] Or Patashnik, Zongze Wu, Eli Shechtmam, Daniel Cohen-Or, and Dami Lischinski. Styleclip: Text-driven
manipulation of stylegan imagery. In ICCV, 2021.
[36] Guanghui Qin and Jason Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. In NAACL-HLT, 2021.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In ICML, 2021.
[38] Zhiyuan Ren, Zhihong Pan, Xin Zhou, and Le Kang. Diffusion motion: Generate text-guided 3d human motion by diffusion model. In ICASSP, 2023.
[39] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski. Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-tex text models. In NeurlPS, 2022.
[40] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 1 13971, 2023.
[41] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. NeurlPS, 2021.
[42] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of machine learning research, 2008
[43] Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The caltech-uese birds-200-201 1 dataset. 201
[44] Wenhao Wang. Yifan Sun, Wei Li, and Yi Yang. Transhp: Image classification with hierarchical prompting. In NeurlPS, 2023.
[45] Xudong Wang. Zhirong Wu, Long Lian, and Stella X. Yu. Debiased learning from naturally imbalanced labels. In CVPR, 2022.
[46] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgcaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
[47] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In AAAI, 2022.
[48] Kai Yi, Xiaogian Shen, Yunhao Gou, and Mohamed Elhoseiny. Exploring hierarchical graph representation for large-scale zero-shot image classification. In ECCV, 2022.
[49] Aohan Zeng, Xiao Liu, Zhengxiao Du Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLMM-130b: An open bilingual pre-trained model. In ICMI., 2023.
[50] Hui Zhang, Shenglong Zhou Geoffrey Ye Li. and Naihua Xiu. 0/1 deep neural networks via block coordinate descent. arXiv preprint Atd:22206009379, 2022.
[51] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng
Li. Tip-adapter: Training- free alip-adapter for better vision- language modeling. In ECCV, 2022.
12

---

[52] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
[53] He Zhu, Ce Li. Haitian Yang. Yan Wang, and Weiqing Huang. Prompt makes mask language models better adversarial attackers. In ICASSP, 2023.
13


Reviews:
[{"review_id": "vU140vPOOB", "reviewer": "Reviewer_bCYj", "summary": "This paper argues that CLIP is limited due to the semantic ambiguity of similar classes. To overcome this, the paper proposes to construct a class hierarchy with large-language models and introduces a novel image classification framework based on hierarchical comparisons that compare embeddings at each level. Extensive experiments show promising results on multiple datasets under the zero-shot image classification task.", "questions": "I do not have additional questions at this stage.", "limitations": "Limitations are discussed in the paper. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "-This paper is well motivated. Using a class hierarchy to disambiguate the semantic embeddings of similar classes make a lot of sense.\n\n-A novel hierarchical classification framework is proposed. To my knowledge, constructing a class hierarchy from LLMs is new. It is interesting to see this new application of LLMs. I also like the proposed plug-and-play hierarchical classification method which does not need to retrain the CLIP model.\n\n-Empirically, it consistently improves the CLIP and Open-CLIP on 6 datasets under the zero-shot image classification task. The interpretability is another interesting property of the method.", "weaknesses": "-Comparisons between the class hierarchy generated by LLMs and the ones labeled by humans e.g., WordNet are missing. \n\n-What is the impact of the hyperparameters, or example, the threshold to construct the hierarchy? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for their valuable insights and recognition of our motivation, innovative hierarchical classification framework, and the significance of our experimental results and interpretability. In the subsequent sections, we address the concerns raised by the reviewers.\n\n### [Q1] Comparison with WordNet\n\nWe are grateful for your attention to WordNet, as it offers pre-defined descriptions for a wide variety of classes.\n\nHowever, one primary emphasis of our research centers around the generation of visually comparative descriptions. This emphasis stems from our utilization of the CLIP model, which aligns the feature space of image and language. On the contrary, WordNet descriptions \n\n  - subsist in isolation, devoid of direct interdependencies with other definitions (similar to [26]), and\n  - often lacks any provision for visual descriptions.\n\nTherefore, we argue that, compared to our hierarchical and comparative descriptions, WordNet descriptions are less suitable for zero-shot open-vocabulary image classification. This elucidates our rationale for excluding WordNet-based descriptions from the initial submission.\n\nEmpirically, we gather WordNet descriptions for each category and append them to the original text, and evaluate the performance across two distinct model configurations (ResNet-50 and ViT-B/32) by computing the average accuracy. The corresponding outcomes are tabulated in Tab. A. Consistent with our analysis, the employment of WordNet-based descriptions exhibits a performance degradation compared to the inherent effectiveness of the CLIP default prompt, \"a photo of {class_name}\".\n\n### [Q2] Impact of hyperparameters\nThank you for your constructive suggestion! Ablation experiments are carried out to examine the impacts of $\\lambda$, $\\tau$, $N$, and prediction depth.\n- $\\lambda$.  As illustrated in Tab. C, as $\\lambda$ increases from 0 to 1, the mean progressively ascends from 61.25 to 62.01. This observed trend underscores a link between greater $\\lambda$ values and higher accuracy. The peak accuracy of 62.12 is at $\\lambda = 0.9$, prior to a marginal reduction to 62.01 at $\\lambda = 1$. \n- $N$. $N$ determines the k-means algorithm's group count. Excessively large $N$ (e.g., 8) yields shallow hierarchies and coarse-grained descriptions; smaller $N$ prolongs hierarchy construction. An ablation on the ImageNet with ViT-B/32 for varying $N$ values follows. Notably, $N=4$ slightly improves accuracy at fourfold evaluation time. Conversely, $N=8$ accelerates sixfold but compromises by 1.64% in accuracy. Typically, $N$ falls within 4 to 6 and is proportional to the number of classes.\n| $N$ | Hierarchy Construction Time (Hours) | Accuracy (%) |\n|---|---|---|\n| 4 | 97 | 64.34 |\n| 6 | 23 | 63.88 |\n| 8 | 4 | 61.72 |\n- _The depth of the prediction_. The utilization of average aggregation facilitates score computation across different depths. The outcomes evaluated on the ImageNet with ViT-B/32, in the following table, highlight an enhancement in the second layer relative to the first, which is due to the introduction of contrasting descriptions. Subsequently, a consistent performance improvement is observed with increasing depth. The results are illustrated in Fig. 4 in the main paper.\n| Depth | Accuracy(%) |\n|---|---|\n| 1 | 57.91 |\n| 2 | 59.24 |\n| 3 | 59.57 |\n| 4 | 60.96 |\n| 5 | 61.72 |\n| 6 | 63.88 |\n- $\\tau$. As shown in Tab. D, as $\\tau$ increases from 0 to 1, the mean performance initially descends from 62.12 ($\\tau=0$) to 61.95 ($\\tau=0.6$), and then rebounding to 62.10 ($\\tau=1$). The overall peak accuracy is 62.12 observed at $\\tau = 0$, the value we use in the main paper.\n- $l$. As we stated in Line 201 (main paper), the threshold for leaf nodes is typically set to 2 or 3. Our observation indicates that ChatGPT tends to provide a less distinct differentiation of descriptions when handling more than 3 categories concurrently.\n- Across the majority of our ablation experiments, our approach consistently maintains commendable performance. These outcomes further underscore the robustness and efficacy of our approach, which remains steadfast even under diverse hyperparameter settings.\n"}, {"review_id": "c3Efaur17y", "reviewer": "Reviewer_5GjG", "summary": "This paper shows an example of introducing the power of LLMs in the zero-shot classification. This paper proposes a framework (or a process flow) rather than proposing a new mathematical algorithm. However, as shown in the experimental results section, it is demonstrated that the classification accuracy is about 1% point better than the state-of-the-art.", "questions": "Please see “Wekness” part.", "limitations": "Please see “Wekness” part.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ It is impressive to see how the integration of grouping and Language Model-generated descriptions to build knowledge trees can boost zero-shot classification performance. This is a compelling application of Large Language Models.", "weaknesses": "- I am afraid that the classification accuracy would depend on the performance of LLMs, but the detailed discussion on the accuracy of LLMs and its relation to the final classification performance is missing. It would also be interesting to compare multiple LLM models.\n- I admit that the classification performance can be improved as can be seen in Tables 1 and 2. However, only the numbers are compared in those tables. The authors might want to discuss under what circumstances the proposed LLM-based one works better than the baselines and under what circumstances it does not. In other words, please discuss the remaining problems or limitations of the proposed method.\n\nThis is just a comment to improve the paper.\nThe texts in figures are too small, especially in Table 4. The authors might want to make them the same size as the text in the main body.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere gratitude to the reviewer for their recognition of our work's innovation in integrating grouping and Language Model-generated descriptions to enhance zero-shot classification performance.\n\n### [Q1] Impact on the choice of large language models\n\nWe deeply appreciate the valuable suggestion from the reviewer to explore alternative large language models beyond ChatGPT in our experiments. We illustrate the performance of our hierarchical comparison approach using a representative sample of Large Language Models (LLMs) in Fig. A. \n\nIn all examined LLMs except for the small Multilingual Pre-Trained Transformer (MPT), the integration of descriptions resulted in performance gains that differed from the baseline. From a broader perspective, better LLMs usually lead to better descriptions and performance gains for CLIPs, and the relationship between LLMs' capability and CLIP accuracy is close to a linear correlation. However, the effect of the ChatGPT-based method jumps out of this linear relationship assumption and exhibits the most significant improvement. As a result, our approach shows efficacy in the vast majority of tested LLMs, and the description-generating power of ChatGPT is outstanding among them.\n\n### [Q2] Remaining problems of the proposed method\n\nPlease kindly refer to Section 2 of the supplementary material, specifically the subsection on \"Failure Cases\" where we delve into the types of descriptions that might adversely affect performance and pose challenges for the CLIP encoder.\n\n### [Q3] Text size\n\nThanks for pointing this out! We will enlarge the texts in the final version."}, {"review_id": "0cKArFJq9p", "reviewer": "Reviewer_ufWY", "summary": "This paper presents a hierarchical comparison method for zero-shot open-vocabulary image classification. By leveraging CLIP and Large Language Models (ChatGPT here), the authors build class hierarchies by recursively comparing and grouping classes. This hierarchy enables image classification by comparing image and text embeddings at different levels, improving accuracy and interpretability. The proposed method outperforms existing methods, achieving state-of-the-art performance on various datasets.", "questions": "Please see the weaknesses.", "limitations": "The utilization of ChatGPT/GPT-4 for generating descriptions incurs a cost, which could potentially restrict the reproducibility and impact of this work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method exhibits high interpretability, as it draws inspiration from how human beings hierarchically organize visual and linguistic concepts in the real world. This intuitive approach enhances the understanding and explainability of the image classification process.\nThe efficient utilization of the extensive pretraining knowledge from ChatGPT/GPT-4 significantly improves the quality and granularity of the text descriptions.\nThe evaluations are conducted on 6 datasets and 4 different backbone settings, ensuring a robust assessment of the framework's effectiveness and generalizability.\n", "weaknesses": "It is worth considering recent open-source LLMs like LLaMA, Alpaca, and Vicuna for generating descriptions and conducting comparisons. These LLMs could provide alternative options and insights for the generation and evaluation of descriptions in the proposed framework.\nThe evaluation in the paper only includes one baseline method [26], which may limit the assessment of the proposed approach.\nAll the datasets used in the evaluation are in the general domain, which might not fully demonstrate the generalizability of the proposed method across various domains. Including datasets from diverse domains would further validate the effectiveness and robustness of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our heartfelt appreciation to you for acknowledging the high interpretability of our proposed method, which derives inspiration from human conceptual organization, and your recognition of the efficient utilization of ChatGPT's extensive pretraining knowledge and the comprehensive evaluations across multiple datasets and backbone settings.\n\n### [Q1] Impact on the choice of large language models\n\nWe deeply appreciate the valuable suggestion from the reviewer to explore alternative large language models beyond ChatGPT in our experiments. We illustrate the performance of our hierarchical comparison approach using a representative sample of Large Language Models (LLMs) in Fig. A. \n\nIn all examined LLMs except for the small Multilingual Pre-Trained Transformer (MPT), the integration of descriptions resulted in performance gains that differed from the baseline. From a broader perspective, better LLMs usually lead to better descriptions and performance gains for CLIPs, and the relationship between LLMs' capability and CLIP accuracy is close to a linear correlation. However, the effect of the ChatGPT-based method jumps out of this linear relationship assumption and exhibits the most significant improvement. As a result, our approach shows efficacy in the vast majority of tested LLMs, and the description-generating power of ChatGPT is outstanding among them.\n\n### [Q2] Datasets from special domains\n\nThe outcomes derived from the EuroSAT dataset are showcased in Table B. Despite the distinctive nature of EuroSAT images being satellite-based and divergent from the majority of prevalent image classification datasets, our proposed approach manages to secure a substantial improvement of approximately 1% in performance across varied backbone architectures. This observation not only reaffirms the consistency of our findings as highlighted in the main paper's Tab. 2 but also underscores the adaptability and robustness of our approach across diverse imaging modalities."}, {"review_id": "7U16hdcIo6", "reviewer": "Reviewer_Y2ri", "summary": "The article presents an approach for refining the classification performance of a vision-language model (i.e. CLIP), using descriptions derived from a large-language model (i.e. GPT-4). Different from previous attempts in this direction [26], directly extracting a discriminative description for each of the classes, this work proposes a hierarchical approach where more specific class descriptions are available at each layer. Starting from the class descriptions of [26], the text embeddings are used to perform clustering, creating groups of similar classes. Then, the LLM is used to obtain further descriptions to discriminate classes within each similar group. This process can be repeated iteratively until the number of classes in a group is low. Experiments on various benchmarks show that the proposed model consistently outperforms CLIP  and the direct descriptions of [26].", "questions": "Following on the previous points:\n1. Why is the gap between [26] and the proposed approach low? \n2. What is the computational cost of the proposed model?\n3. What is the impact of $\\lambda$, and $N$?\n4. How does performance vary w.r.t. the depth of the prediction?\n6. Is $\\tau$ used? If so, how does the performance vary?\n5. What is the motivation behind using the average as the aggregator in Eq. 8?\n\nPlease, while answering, clarify also how the potential additional analyses/discussions would be included in the revised manuscript.", "limitations": "Section 5 discusses various limitations of the model, highlighting a fundamental one, i.e. the assumption that a category is easy to describe via natural language. Overall, I find the assessment of the limitations fair and thorough.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The idea of splitting classification into multiple stages by asking for increasingly more specific descriptions is interesting as it allows the model to focus on easy separation first and increasingly more fine-grained ones later in the process. It also inherently gives interpretability, as we can check which type of descriptions lead to high scores for a certain class (Figure 4).\n\n- Section 3.2 describe the type of query used, while the supplementary (Section 1.1) shows examples of failure cases of naive querying strategies.\n\n- Despite the scores being very close among compared classes, Figure 4 provides a clear view of the type of descriptions involved in the classification process. Similarly, Figure 6 (despite being hard to interpret without the caption) shows the step-by-step changes in the model predictions, further complementing the info in Figure 5 on the hierarchical process. \n\n- The framework is flexible, as it can be readily applied to new versions of large-language models and vision-language ones.", "weaknesses": "1. Despite consistently outperforming [26], the gap between the two models is relatively low: i.e. in Table 2, the average gain is always below 1% in accuracy. This comes unexpectedly as the model is initialized using descriptions extracted as in [26] (lines 134-136), and the article stresses the advantages of the proposed approach (e.g. captions of Fig. 1, lines 75-77,  lines 139-142, lines 171-174). Moreover, the proposed approach increases the computational cost, by increasing the number of text-image comparisons (i.e. the summation in Eq. 5). The claims on the performance/advantages should be revised to avoid overstating the contribution, and the computational cost discussed/reported.\n\n2. The article reports limited analyses w.r.t. the method's hyperparameters. For instance, the model uses a factor $\\lambda$ to balance the default CLIP prediction with the obtained hierarchical one (Eq. 6) but the impact of $\\lambda$ is not analyzed. Similarly, this applies to the number of clusters $N$ and the size of the leaf nodes for the stopping criterion, $l$. It would also be interesting to show how the performance varies depending on the depth in the hierarchy, to confirm that deeper predictions lead to better performance. \n\n5. $\\tau$ is defined as the tolerance score to mitigate the above issue (Eq. 5, line 194). However, this value is set to 0 in the experiments (line 201). If $\\tau$ is not used, it should not be defined, otherwise, it is important to specify the cases where it is used and analyze its impact.\n\n3. [26] tests also on the EuroSAT dataset of satellite images, where providing natural language descriptions is challenging. This dataset is not included in the current experiments, but would further show the potential advantages of the approach in difficult classification settings.\n\n4. Minor point: lines 185-194 describe the need to avoid that unrelated objects receive high scores for very specific descriptions. A strategy to avoid potential differences could be to just perform classification group-wise at each level of the hierarchy. This would work by \"filtering\" the classes by excluding every class that is not part of the predicted group for the next level of the hierarchy. Would this strategy be feasible? and in general, is there a reason for using the average (Eq. 8) rather than other aggregation mechanisms of the hierarchical scores?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of our approach's innovative concept of classification through specific descriptions, insights into our visualizations (Figs. 4, 5, and 6), and acknowledgment of the framework's adaptability to new large-language and vision-language models.\n### [Q1] Improvement over [26]\nThank you for highlighting the consistent improvement achieved across diverse datasets, which reinforces the efficacy of our approach. However, rather than pursuing comprehensive enhancements, we focus on refining semantic clarity within closely related classes. Hence, the extent of performance improvement varies based on the prevalence of challenging test samples in each dataset. Our results and visualizations compellingly validate the achievement of this objective through our hierarchical comparison approach.\n### [Q2] Computational cost of the proposed model\nWhile our approach is indeed relatively more time-consuming than [26], it's important to note that the time-intensive aspect, constructing hierarchies, is a one-time operation for each dataset. Conversely, the subsequent stages, reasoning with CLIP models, are not slower than [26].\n- _Hierarchy construction_. Given the random latency of ChatGPT API calls, we report the time used to construct hierarchies in our experiments: 23 hours for ImageNet, 4 hours for CUB, 2.5 hours for Food101, 10 hours for Places365, 40 minutes for Oxford Pet, and 52 minutes for Describable Texture.\nBuilding hierarchies, while potentially time-consuming, is a one-time task per dataset.\n- _Reasoning_. The reasoning process has comparable runtime with our baseline [26]. The main difference is that we use Multi-level Score Fusion to aggregate scores, but since we implemented it in a parallelized way, the inference is about the same speed.\n\nIn summary, even though our approach has a hierarchy construction procedure for each dataset, the overall efficiency is practical and acceptable.\n### [Q3/4/5] Impact of hyperparameters and ablation study\nThank you for your constructive suggestion! Ablation experiments are carried out to examine the impacts of $\\lambda$, $\\tau$, $N$, and prediction depth.\n- $\\lambda$.  As illustrated in Tab. C, as $\\lambda$ increases from 0 to 1, the mean progressively ascends from 61.25 to 62.01. This observed trend underscores a link between greater $\\lambda$ values and higher accuracy. The peak accuracy of 62.12 is at $\\lambda = 0.9$, prior to a marginal reduction to 62.01 at $\\lambda = 1$. \n- $N$. $N$ determines the k-means algorithm's group count. Excessively large $N$ (e.g., 8) yields shallow hierarchies and coarse-grained descriptions; smaller $N$ prolongs hierarchy construction. An ablation on the ImageNet with ViT-B/32 for varying $N$ values follows. Notably, $N=4$ slightly improves accuracy at fourfold evaluation time. Conversely, $N=8$ accelerates sixfold but compromises by 1.64% in accuracy. Typically, $N$ falls within 4 to 6 and is proportional to the number of classes.\n| $N$ | Hierarchy Construction Time (Hours) | Accuracy (%) |\n|---|---|---|\n| 4 | 97 | 64.34 |\n| 6 | 23 | 63.88 |\n| 8 | 4 | 61.72 |\n- _The depth of the prediction_. The utilization of average aggregation facilitates score computation across different depths. The outcomes evaluated on the ImageNet with ViT-B/32, in the following table, highlight an enhancement in the second layer relative to the first, which is due to the introduction of contrasting descriptions. Subsequently, a consistent performance improvement is observed with increasing depth. The results are illustrated in Fig. 4 in the main paper.\n| Depth | Accuracy(%) |\n|---|---|\n| 1 | 57.91 |\n| 2 | 59.24 |\n| 3 | 59.57 |\n| 4 | 60.96 |\n| 5 | 61.72 |\n| 6 | 63.88 |\n- $\\tau$. As shown in Tab. D, as $\\tau$ increases from 0 to 1, the mean performance initially descends from 62.12 ($\\tau=0$) to 61.95 ($\\tau=0.6$), and then rebounding to 62.10 ($\\tau=1$). The overall peak accuracy is 62.12 observed at $\\tau = 0$, the value we use in the main paper.\n- $l$. As we stated in Line 201 (main paper), the threshold for leaf nodes is typically set to 2 or 3. Our observation indicates that ChatGPT tends to provide a less distinct differentiation of descriptions when handling more than 3 categories concurrently.\n- Across the majority of our ablation experiments, our approach consistently maintains commendable performance. These outcomes further underscore the robustness and efficacy of our approach, which remains steadfast even under diverse hyperparameter settings.\n### [Q6] Motivation behind using the average aggregator in Eq. 8\nThank you for asking. We employ the average as an aggregator for enhancing efficiency during inference. The inherent tree structure associates each class with distinct descriptions spanning varying levels of granularity. Thus, sequential representations manifest heterogeneity. The utilization of an average aggregator renders these features homogeneous, thereby significantly accelerating inference through parallelization.\n\nOur Multi-level Score Fusion maintains sequential arrangement across different levels, concurrently employing coarse-to-fine reasoning. The relevant code excerpt can be found in the Sup. Moreover, the effectiveness of our mean aggregator in capturing finer details with increasing layer depth is supported by Fig. 4 in the main paper.\n### [Q7] Evaluation on EuroSAT\nThank you for your suggestion. As shown in Tab. B, we evaluate our method on the EuroSAT dataset, which consistently attains an approximate 1% improvement.\n### [Q8] Revision\nThank you again for your constructive feedback! We plan to revise our manuscript (main paper) as follows:\n- Q1: Add a new paragraph in Sec. 4.2 after L242.\n- Q2: Add a new paragraph in Sec. 4.3 after L273.\n- Q3/4/5: Add new tables and paragraphs in Sec. 4.3 after L256.\n- Q6: Explain the motivation of average aggregation in detail after Sec. 3.3 at L185.\n- Q7: Add results to Tab. 2 and incorporate the analysis into the paragraph in Sec. 4.2 at L243."}]

Author rebuttal:
We would like to express our sincere gratitude to all the reviewers for their invaluable time and expertise invested in the meticulous review of our paper. Your thoughtful feedback has immensely contributed to refining our work. We are also encouraged by the recognition of our paper's strengths:

- Reviewer `bCYj`: The well-motivated approach, novel hierarchical classification framework, empirical effectiveness, and interpretability.
- Reviewer `5GjG`: Integration of grouping and language-model-generated descriptions and compelling application of large language models.
- Reviewer `ufWY`: High interpretability, efficient utilization of pretraining knowledge, and comprehensive evaluations.
- Reviewer `Y2ri`: Staged classification approach, inherent interpretability, comprehensive content, and flexibility of the framework.

These insights resonate with our aim to present a training-free, intuitive, effective, and explainable approach for zero-shot open-vocabulary image classification. We are deeply committed to incorporating all the valuable feedback received and refining our manuscript accordingly in the revised version.

Meta-review: The paper proposes a method to improve classification with CLIP using LLMs to create a discriminative class hierarchy, and an algorithm top use this hierarchy to create classification decisions. The reviews agree that the method is well motivated, interpretable, and novel, plus the results are convincing. The scores are unanimous in recommending acceptance. Concerns were raised about the depth of the analysis and ablation, but these have been addressed in the rebuttal. Therefore, I recommend acceptance. The authors should include the data presented in the responses to reviewers `bCYj` and `Y2ri` in the camera ready version of the paper.
Decision: Accept (poster)