SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Comparison with the zero-shot SoTA method under fair settings. Overall, we gain  $\sim 1\%$  improvement across different backbones and datasets.
Backbone	Method	ImageNet	CUB	Food101	Place365	Pets	Texture	Mean
ResNet-50	[29]	60.20	49.10	74.94	35.79	78.58	44.63	57.20
Ours	60.63	49.86	75.44	36.83	79.99	45.67	58.07
ViT-B/32	[29]	63.48	53.80	82.79	40.15	81.60	45.69	61.25
Ours	63.88	54.18	83.02	40.73	82.69	48.19	62.12
ViT-B/16	[29]	68.43	58.85	87.68	40.87	86.56	49.15	65.25
Ours	68.95	59.25	88.13	41.52	87.19	51.26	66.05
ViT-L/14	[29]	75.36	65.02	93.00	40.13	92.10	55.95	70.26
Ours	79.64	65.45	93.17	41.13	92.84	58.04	71.71
often rely on comparing the same attributes, which is not one of the strengths of CLIP or OpenCLIP. For more experiments and analyses please refer to the supplementary materials.
Additionally, we compare our method to the state-of-the-art zero-shot approach. To ensure a fair comparison, we employ the same model weights for each distinct backbone setting. Moreover, we utilize the same LLM, ChatGPT [3], to generate descriptions, where the different factor lies in our hierarchical description structure. As shown in Tab. 2, our approach can consistently outperform the current state-of-the-art method in zero-shot classification.
Notably, we surpass the state of the art [29] on Describable Textures by a considerable margin. This is because, without the information of other classes as the context, LLMs generate class descriptions that lack discriminability. ChatGPT [3] characterized the gauzy class as "a thin, translucent fabric" with "a light, airy feel" and "a delicate or lacy appearance" and the lacelike class as an "intricate, detailed pattern" "often used for decoration". Even though correct, these descriptions do not provide any discriminative information, which renders them ineffective in classifying the two textures. On the other hand, in our hierarchy, gauzy and lacelike are two leaf nodes with a common parent. When comparing them, our approach describes the gauzy class as "a more transparent or sheer appearance" and "a more delicate texture with a softer hand-feel" and the lacelike as "a lightweight
Figure 4: We show two inference examples and their score distribution by hierarchically querying the description with the image embedding. The description corresponds to the left input image and the right image is the sample from the confusing categories relative to the left one (zoom in for small fonts).
and almost transparent appearance” which has “visible holes or gaps throughout the material”. Such descriptions accurately capture the subtle differences between these two materials and facilitate zero-shot classification between these two categories, which manifests in the superiority of our proposed hierarchical comparison approach for zero-shot image classification.
4.3 Interpretability and Analysis
Fig. 4 illustrates two groups of examples that demonstrate the inference and score calculation in our method. Specially, we input two images and compute their similarity to the hierarchical descriptions associated with the left image. Due to the weak comparative information in the early descriptions, similar images tend to yield similar scores at the beginning. As we progress towards a more comparative description, the disparity between the scores of the two images gradually increases. Besides, this also highlights the interpretability of our method, as it provides the user insights regarding the attributes that elicit stronger responses to the input. Additionally, it indicates the specific layer of description at which the scores of different images diverge, widening the gap.
In order to further investigate the factors contributing to the improved performance of our model, we conduct a comparison of the confusion matrices between our model and CLIP, using the same backbone. As shown in Fig. 5, the number of cases where Ragdollis is misclassified as Birman is reduced from 54 to 37, as contrasting descriptions bring out distinguished text features.
Fig. 6 illustrates the step-by-step progression of our comparative hierarchical description, pushing the image embedding closer to its corresponding ground-truth text embedding. Simultaneously, this process gradually distances the confusing label from the image embedding. We visualize this process using t-SNE [42] reduction to two dimensions, while also regularizing them into unit vectors.
Figure 5: We highlight the difference between the confusion matrices at the position where Ragdoll samples are misclassified into Birman. At the right part, we show the comparative descriptions which explain where the gain comes from.
Figure 6: T-SNE visualization of similarity between the image embedding and the text embedding. Row 1 shows how our hierarchical descriptions help pull the image embedding closer to the groundtruth text embedding; Row 2 shows how our hierarchical descriptions push the confusing text embedding away from the current image embedding.  $\nearrow$  represents the image embedding;  $\nearrow$  represents the text embedding (groundtruth in Row 1; confusing category in Row 2); Other points represent the remaining text embeddings in the latent space.
We ablate on four distinct hyperparameters:  $\lambda$ ,  $N$ , depth, and  $\tau$  utilizing a ViT-B/32 architectural foundation. Metrics derived from mean accuracy are employed for  $\lambda$  and  $\tau$  across datasets including ImageNet [10], CUB [43], Food101 [2], Place365 [26], Oxford Pets [50], and Describable Textures [7]. Alternatively, depth and  $N$  are exclusively evaluated based on ImageNet [10] performance.
Score Momentum  $\lambda$ . As illustrated in Fig. 7a, there exists a direct relationship between increasing  $\lambda$  values and performance enhancement. However, solely relying on scores from the proposed methodology without offset utilization results in a decline in performance. Optimal performance is achieved when  $\lambda$  is set to 0.9.
Group Number  $N$ . Larger group numbers, such as 8, yield rudimentary hierarchies and generalized descriptions, whereas smaller values extend the hierarchy's development phase. Represented in Fig. 7b, it reveals that moderate increases in accuracy corresponded to fourfold evaluations. Inversely, a sixfold acceleration is observed, albeit at a cost of a  $1.64\%$  dip in accuracy. Typically,  $N$  oscillates between 4 to 6 and directly correlates with the class count.
The Depth of the Prediction. Incorporating average aggregation is advantageous for score computation across variable depths. Results presented in Fig. 7c, when evaluated on ImageNet with ViT-B/32, underscore the superior performance of the next layer over its predecessor, attributed to the incorporation of divergent descriptions. A successive rise in performance is evident with depth augmentation.
Tolerance for Score Reduction  $\tau$ . As delineated in Fig. 7d, the evolution of  $\tau$  from 0 to 1 maintains nearly consistent performance, peaking at 1.0. Although optimal performance is observed at  $\tau = 1$ , marginal variations across datasets necessitate the retention of this hyperparameter.
(a) Ablation study of  $\lambda$ . We fix  $\tau$  as 0, use ViT-B/32 as the backbone, and compute the mean accuracy across six datasets.
(c) Ablation study of Depth. Average aggregation is used in computing scores across various depths on ImageNet.
Figure 7: Ablation study across different hyperparameters.
(b) Ablation study of N. An ablation on the ImageNet with ViT-B/32 for varying N values follows.
(d) Ablation study of  $\tau$ . We fix  $\lambda$  as 0.9, use ViT-B/32 as the backbone, and compute the mean accuracy across six datasets.
4.5 Failure Cases
By comparing confusion matrices, we find that there are indeed some classes whose classification accuracy decreases when using descriptions at greater depths. We pick representative classes and their descriptions of where they have lower classification accuracy than only using the previous depth.
We observe that when ChatGPT differentiates between closely related classes in its deeper layers, it often references attributes such as length and size, which can be challenging to represent visually. Consider the statement: "Newfoundland dogs are typically larger than Chows. Male Newfoundlands can weigh up to 150 pounds and reach heights of 28 inches, whereas Chows generally weigh between 50-70 pounds and stand 20-25 inches tall." The CLIP model's encoder, however, struggles to interpret this length information accurately due to variations in camera angles and positions. This observation suggests that detailed text descriptions might pose comprehension challenges for CLIP. Thus, enhancing the synergy between LLMs and CLIP presents itself as a promising avenue for future research.
In this work, we propose a training-free, explainable, and effective zero-shot image classification method with enriched hierarchical descriptions powered by LLMs, such as ChatGPT. The knowledge tree constructed by alternately grouping descriptions and generating descriptions allows us to achieve SoTA in different settings. A central theme of our work is to inspire the field by emphasizing that LLMs possess the potential to enrich semantics and make them useful for contemporary machine learning tasks. How to extract semantics worth expanding in the current machine learning tasks is the key to exploring the potential of LLMs.
**Limitations.** Our approach is unfortunately still limited by some limitations of CLIP. Although we have good performance overall, as discussed in Sec. 4.2, our approach is hindered by the inability of CLIP to reason about relative relations in extremely fine-grained classification tasks. In addition, the entire approach is based on the premise that natural language descriptions can be helpful for image classification given enough context. Our experiments demonstrate that this is the case in general, but in exceedingly nuanced scenarios, the differences between different classes might be too intricate to be described by natural languages.
[52] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
[53] He Zhu, Ce Li, Haitian Yang, Yan Wang, and Weiqing Huang. Prompt makes mask language models better adversarial attackers. In ICASSP, 2023.