Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification
Authors: Reza Esfandiarpoor
Stephen Bach
Venue: ICLR 2024 poster (2024)
Keywords: representation learning for computer vision
image classification
vision-language models
large language models
CLIP
GPT-3
LLAMA
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose a novel zero-shot approach that uses natural language to provide vision-language models with differentiating information about classes in downstream image recognition tasks.

Abstract: A promising approach for improving the performance of vision-language models like CLIP for image classification is to extend the class descriptions (i.e., prompts) with related attributes, e.g., using brown sparrow instead of sparrow. However, current zero-shot methods select a subset of attributes regardless of commonalities between the target classes, potentially providing no useful information that would have helped to distinguish between them. For instance, they may use color instead of bill shape to distinguish between sparrows and wrens, which are both brown. We propose Follow-up Differential Descriptions (FuDD), a zero-shot approach that tailors the class descriptions to each dataset and leads to additional attributes that better differentiate the target classes. FuDD first identifies the ambiguous classes for each image, and then uses a Large Language Model (LLM) to generate new class descriptions that differentiate between them. The new class descriptions resolve the initial ambiguity and help predict the correct label. In our experiments, FuDD consistently outperforms generic description ensembles and naive LLM-generated descriptions on 12 datasets. We show that differential descriptions are an effective tool to resolve class ambiguities, which otherwise significantly degrade the performance. We also show that high quality natural language class descriptions produced by FuDD result in comparable performance to few-shot adaptation methods.

Published as a conference paper at ICLR 2024
FOLLOW-UP DIFFERENTIAL DESCRIPTIONS: LANGUAGE MODELS RESOLVE AMBIGUITIES FOR IMAGE CLASSIFICATION
Reza Esfandiarpoor & Stephen H. Bach
Department of Computer Science
Brown University
Providence, RI 02906, USA
(reza sefanddiapporr stephen bach @brown.edw
ABSTRACT
A promising approach for improving the performance of vision-language models like CLIP for image classification is to extend the class descriptions (i.c., prompts) with related attributes, e.g,, using brown sparrow instead of sparrow. How- ever, current zero-shot methods select a subset of attributes regardless of com- monalities between the target classes, potentially providing tion that would have helped to distinguish between them. For instance, they may use color instead of bill shape distinguish between sparrows and wrens, which are both brown. We propose Follow-up Differential Descriptions (FuDD), zero-shot approach that tailors the class descriptions to each dataset and leads additional attributes that better differentiate the target classes. FuDD first iden- tifies the ambiguous classes for each image, and then uses Large Language Model (LLM) to generate new class descriptions that differentiate between them. The new class descriptions resolve the initial ambiguity and help predict the correct label. In our experiments, FuDD consistently outperforms generic de- scription ensembles and naive LLM-generated descriptions on 12 datasets. We show that differential descriptions are effective tool resolve class ambi- guities, which otherwise significantly degrade the performance. We also show that high quality natural language class descriptions produced by FuDD) result in comparable performance to few-shot adaptation methods. Code: https: //github. htt.abbb.coo//atteeeeeeerr/////a fudd
useful informa-
INTRODUCTION
What is the most distinguishing characteristic of sparrow? It depends. To distinguish it from what? To distinguish from a goldfinch, is the brown color. But, to distinguish from a wren, it the conical bill (Fig. [. Here, we propose zero-shot approach adapt the class representations of vision-language models based on other classes in an image classification task. We use natural lan- guage descriptions (called prompts) to provide visually differentiating information for target classes.
(A)
Color
Bill
(B)
Sparrow vs. Wren
Sparrow vs. Goldfinch
Sparrow
Brown
Conical
Brown color
Conical bill
Brown color
Conical bill
Goldfinch
Yollow
Conical
X Sparrow
Sparrow
Sparrow
Sparrow
X
Wren
Wirnn
C Goldfinch
Goldfinch
Wren
Brown
Slender
Figure 1: A) Attributes for three different classes. B) Two sample classification tasks involving the wren class. The distinguishing characteristics of each class vary based other classes. Our ap- proach selects the class descriptions based on other classes in the dataset to provide the information that differentiates the target classes.

---

Published as a conference paper at ICLR 2024
(A): Sencal,
Best Shartheir. Alyssinnin,
(Britteh Shorthair, Bengel
beitish thoothete with blue coppez eye coloc. beitish shortheiz with soundedd
(Bricich Shorthots, Abyssinton)
Boxez,
(fergal, Anyssiniar)
tenpal cet, adca goeen gold eye colce.
bangel cas, with spectad cool palleer.
LLM
abyssintan, aith lazge, pointedd eazs, alyssiniin with Licked coal pellurn.
beitish sheechste, with Blus coppec eye coloc bengel oat, with ateen gold eye color.
Beitish Shorchais
beitish
seall, abyssinian, with Large,
0. Abyosinian Brieish Shorthatr Bergal
Bengal
Anyssiriian
bengel with spocted abyasintan out, with ticked cont
n. Boxer
Figure 2: FuDD overview. A) Using the model's initial prediction, we identify the potentially ambiguous classes. B) We use a large language model to generate class descriptions that differentiate the ambiguous classes. C) We use the new differential descriptions in a follow-up classification task to resolve the initial ambiguity and select the correct label.
Large Lision-Language Models (VLMs) use natural language as a source of supervision, which allows to easily create new classifiers by describing the classes in natural language (e.g., class names) and labeling each image with the closest class. As a result, we can efficiently transfer the learned visual representations downstream tasks by adapting the class descriptions (i.e., prompts)
to the model's pre-training distribution (Chen et al. 2023] Khattak et al. 2023 Menghini et al.,
2023
Mirza et al.] 2023 Novack et al. 2023 Patashnik et al. 2021 Radford et al. 202 Zang
et al. 2022 Zhang et al. 2023a).
The performance prompting depends on careful prompt engincering find class descriptions that provide the most helpful information for each task (Menon & Vondrick 2023 Novack et al. (2023) Previous works have used class attributes for better zero-shot learning performance (Lampert al,
2013 Parikh & Grauman, [011 Romera-Pareder & Torr 2015 Socher et al., [2033] Xian et al.
2018 . Several recent works have adapted this idea for image classification with VLMs and propose to describe the classes by their attributes, such as color and shape (Menon & Vondrick 2023 Pratt ct al. 2022 Yang et al. (023). Specifically, they prompt a large language model (LLM) with queries like What does an image of a (class name } look like?. ?, and use the responses as the new class descriptions. The main idea is to enable the model to use the described attributes in addition the class names identify each class. This should lead to better class representations since models can better detect the attributes because of their prevalence during pre-training.
The additional information is helpful if it discriminates the class from all other classes in the tar- get dataset. At least, the provided information should differentiate the class from other classes in the dataset that are frequently mistaken for it. Despite this crucial requirement, current zero-shot approaches generate descriptions solely based on the class itself without considering other classes in the dataset (Menon & Vondrick 2023, Pratt et al. 2022). As a result, although the descriptions provide additional details about the class, they might not contain any information that differentiates the class from other classes in the dataset. Thus, current methods might generate class descriptions that are not helpful for the target task.
In this paper, we propose Follow-up Differential Descriptions (FuDDT) a novel approach that tai- lors the class descriptions to each dataset and provides additional details that resolve the potential ambiguities in the target task. For each image, we first use a set of basic class descriptions as usual (Radford al. 2021) ) and identify subset of classes that, according to the VLM, are likely to be the true label and thus are considered ambiguous. Then, for each such ambiguous class, we gen- erate set of descriptions that include the details that differentiate from other ambiguous classes. We rely on the extended world knowledge of pre-trained large language models to generate such differential descriptions scale (Brown al. 20200: Petroni et al. 2019). Finally, we use these differential descriptions in a follow-up classification task to resolve the initial ambiguity and predict the correct label. By customizing the class deseriptions based on other classes in the dataset, FuDD aims to provide the most effective information for separating the classes of the target dataset.
Prooounced like food.
2

---

Published as a conference paper at ICLR 2024
We evaluate our method 12 datasets and show that FuDD consistently outperforms naive de- scriptions for all datasets. FuDD outperforms naive -LMegenrrtted descriptions by 2.41 percentage points on average, with up to 13.95 percentage points improvement for the EuroSAT dataset. In our experiments, we show that not all descriptions resolve ambiguities, and effective class descrip- tions should provide differentiating information about ambiguous classes. Moreover, differentiating the highly-ambiguous classes is the most important factor, accounting for most of FuDD's ' perfor- we experiment with the smaller, publicly available Llama model (Touvron et al. 2023) to study the impact of further fine-tuning, and find that the 7b 7b-parameter model can provide helpful information for everyday concepts. also benefits from further fine- tuning, especially for rare and abstract concepts in the EuroSAT and DTD datasets, with up to 23.41 percentage points boost in accuracy. Finally, we show that the performance when using high-quality class descriptions from FuDD is comparable to using few-shot methods, achieving performance competitive with 16-shot VLM adaptation methods (Yang et al. 2023 Zhou et al. 2022b) for some datasets. Our results uncover the potential of using natural language to tailor the class represen- tations to each dataset by providing information that differentiates the ambiguous classes. These results motivate future work on creating effective class descriptions for each downstream task.
mance gains. In addition GPT-3.5 P
RELATED WORK
There is an increasing body of work on adapting VLMs to wide range of downstream tasks (Gao ce al. 202T Guo et al., 2203 Tia et al. [0222] Novack et al.) 2023 Patashnik et al.) 2021: Rao et al.
2022 Udandarao et al. 2022, Zeng et al. [022; Zhang et al.) 2021 2022). Here, we describe the
related work and highlight their differences with our method.
Prompt Tuning Prompt tuning an efficient approach for few-shot adaptation of VLMs to down- stream classification tasks. Instead of updating the model parameters, prompt tuning methods add learnable parameters to the input image or text (i.e., prompt) and learn these parameters through gradient descent for each dataset (Huang et al. 2022 Jia et al.. 2022, Menghini et al. 2023 Nayak ct al. 2022, Zhou et al., 2022a b). For instance, CoOp adds set of parameters to the class descrip- tions to represent the dataset context; and then uses few labeled samples for training (Zhou al. 2022b). Although prompt tuning methods achieve good accuracy, they require additional labeled examples, which limits their applications. On the other hand, our method is zero-shot and adapts to each dataset without any additional samples, with competitive performance to prompt-tuning meth- ods in low-shot scenarios.
VLMs with Other Foundation Models One line of work uses the capabilities of other foundation
models (Brown al., 2020; Caron et al. [202]] Ramesh al. [222] in combination with VLMs to
better adapt to downstream tasks (Chen et al. 2023] Gupta & Kembhavi 2023] Menon & Vondrick
2023 Mirza et al., [003 Pratt et al.] 2022 Suris et al. [2023] Zeng et al., 2022] Zhang et al. [023bb)
For example, one can use the extended world knowledge of large language models (LLMs) in com-
bination with VLMS to solve more complex visual tasks. Our approach is closely related to this line of work; we discuss the differences further in the next paragraph. Several other methods use text-to-image generation models (Rombach et al.) 2022) on top of LLMs (Brown et al., 2020) to further improve the performance (Udandarao et al. 2022 Zhang et al. 2023a). For instance, SuS-X first uses an LLM to generate class descriptions and then uses a text-to-image generation model to generate synthetic images for each class based on these descriptions (Udandarao et al., (022). Our experiments show that despite using no images, FuDD's performance is comparable to SuS-X for most datasets while avoiding the complexities of text-to-image generation models.
Adaptation Through Description A specific approach for improving class representations with- out additional samples is provide more informative class descriptions (Menon & Vondrick 2023 Novack et al., 2023 Pratt et al.) 2022 Roth et al., (003).. For example, WaffleCLIP adds high- level category names class descriptions to avoid ambiguities caused by class names with multiple
meanings (Roth et al.) 2023). Another approach is to describe the classes with their attributes so
the model can rely on attributes in addition to class names to identify images of each class (Menon & Vondrick, 2023 Pratt et al. 2022). For example, Menon & Vondrick (202)) propose to generate such class descriptions by querying an LLLM about the most important attributes of each class. How- ever, the generated descriptions might provide no useful information for separating the class from
https::/openal
3

---

Published as a conference paper at ICLR 2024
other classes in the dataset. For example, the attribute color is not useful for separating sparrows and wrens since both are brown (Fig. T. To address this issue, LaBo uses additional labeled examples to learn the importance of each attribute (Yang et al. 2023). Then, it selects a set of attributes that are the most discriminative for each class. Unlike LaBo, FuDID generates class descriptions that effectively separate the target classes in the first place, eliminating the need for further optimiza- tion. Despite using no labeled data, FuDD's performance is comparable to few-shot LaBo for most datasets.
FOLLOW-UP DIFFERENTIAL DESCRIPTIONS (FUDD)
Here, we describe the components of our proposed method, FuDD. Section 3.1 we explain how VLMs are used for image classification. In Section 3.2 we use the model's initial predictions to identify potentially misrepresented classes that could lead to misclassifications, i.c., are ambiguous (Fig. 2 In Section (.33] we use large language models to generate class descriptions that explain the visually differentiating information for the ambiguous classes (Fig.Ep). In fection[3.4] we use these differential descriptions in follow-up classification task to resolve the initial ambiguity (Fig.
3.1 BACKGROUND
Following previous work (Radford al. 2021), given set of classes, C, and set of descriptions, Do. for each class, we calculate the class embeddings as:
Σ oT(d), he = [De dED.
where ΦT is the VLM text encoder, and he is the embedding vector for class c. Since VLMs trained to minimize the distance between related image-text pairs, we select the closest class to image embedding Φ1 (z) as the label for image x, where Φ the VLM image encoder.
3.2 DETECTING AMBIGUOUS CLASSES
Enumerating the differences between all class pairs is prohibitive for large datasets with thousands of classes. Instead, we focus on a small subset of potentially ambiguous classes that can lead to misclassifications. For example, in Fig. Zp. the model confident that boxer (a dog breed) is not the label. However, any of the three most similar classes (british shorthair, bengal, and abyssinian cat) likely to be the true label. Therefore, differentiating visual information for these classes sufficient for selecting the correct label. For an image x, we define the set of ambiguous classes CA, as the most similar classes:
CA = arg max
Σ (o)s()((z))he)))
(e
where or is the VLM image encoder, and cos is the cosine similarity operator.
3.3 DIFFERENTIAL DESCRIPTIONS
To help the model distinguish between ambiguous classes, we generate set of class descriptions that explain their visual differences. We take advantage of the extended world knowledge of LLMs to generate such descriptions at scale. Despite being uni-modal, LLMs acquire knowledge of the visual world through massive pre-training datasets (Jiang al. 2020 Petroni et al.) 2019). For example, LLM can learn how sparrow looks by reading the related Wikipedia page.
For each pair ambiguous classes, we condition the LLM to select the visually differentiat- ing attributes and describe them for both classes. We use the in-context learning capabilities of LLMs (Brown et al. 2020) to guide the model to focus on visual characteristies by providing two fixed examples as part of the prompt. Similarly, semble photo captions, which shown to better adapt to VLMs' pre-training distributions (Radford) et al. 202T) (refer to appendix for more details). We use the following prompt template:
guide the LLM to generate descriptions that
4

---

Published as a conference paper at ICLR 2024
Table I: Accuracy of FuDD in comparison with baselines. B/32 and L/14" represent the ViT-B/32 and ViT-L/144 336px vision backbones. ANaive(k) is the improvement of FuDD with & ambiguous classes over the Naive LLM-generated descriptions proposed by Menon & Vondrick 2023
Description
Cub
DTD
EuroSAT
FGVCAircraft
Flowers102
Food101
B/32
L/14"
B/32
L/14
B/32
L/14"
B/32
L/14"
B/32
L/14"
B/32
L/14
Single Template
51.21
63,48
43.14
$4.04
40.87
56.82
20.88
37.08
63.80
75.12
82.63
Template Set
51.52
64.07
42.71
55.32
46.76
54.27
21.15
38.31
63.44
74.14
83.16
93.77
Naive I.IM
52.92
65.15
45.90
55.37
44.18
46.69
21.09
38.79
66.12
75.98
84.02
94.26
FuDD (k=10) 10)
53.97
65.90
45.43
57.66
45.18
60.64
21.87
38.82
67.80
78.76
84.05
94.05
FuDD (k=|C) [CD)
54,30
66.03
44.84
57.23
45.18
60.64
22.32
39.63
67.62
79.67
84.36
94.27
A Naive (k=10)
11.05
10.75
1-0.47
12.29
11.00
113.95
10.78
10.03
1.66
12.78
10.03
4-0.21
A Naive (A= ())
11.38
10.88
4-1.06
11.86
11.00
113.9
11.23
10.84
1.50
13.69
10.34
10.01
ImageNet
ImageNet V2
Oxford Pets
Places365
Stanford Cars
Stanford Dogs
B/32
1//14
B/32
1/14"
R//2
I//1"
B/32
1/14"
B/32
1/14"
B/32
I/14"
Single Template
62.04
74.85
54.77
68.79
84.98
92.86
39.10
40.70
60.37
78.06
58.01
73.61
Template Set
63.37
76.54
55.91
70.85
84.55
92,70
40.91
42.54
60.38
79.12
57.79
74.01
Naive LLM
63.52
76.37
55.96
70.47
83.76
93.08
40.58
41.43
59.63
77.90
57.86
74.02
FuDD (k=10) 10)
64.05
76.70
56.62
70.60
86.92
93,40
42.12
43.95
60.86
78.25
60.03
75.99
FuDD (k= [CD
64.19
77.00
56.75
71.05
89.34
93.51
42.17
44.09
61.46
78.96
60.28
76.34
A Naive (k 10)
10.53
10.33
10.66
10.13
13.16
10.32
11.54
12.52
11.23
10.35
12.17
11.97
A Naive (A CI)
10.67
10.63
10.79
10.58
15.58
10.43
11.59
12.66
11.83
11.06
12.42
12.32
For the following objects, generate captions that represent the distinguishing visual differences between the photos of the two objects. Generate as many captions you can. Object 1: (class name 1) Object 2: (class name 2)
Following the provided samples, the model generates several responses similar to:
Visual characteristie: Bil1 color
Caption 1: photo of plack-focted albatross, with yellow bill.
Caption 2: photo of laysan albatross, with a pink bill.
Given pair of classes 01 and we define the pairwise differential descriptions for class D as all the values for Caption in the LLM response, and similarly define D As result, D contains all the descriptions that visually distinguish from c2. For each ambiguous class we combine all its pairwise descriptions to obtain set of differential descriptions D'.
D'd
U
D
CECAMec
The new set of differential descriptions, D', contains all the information necessary for separating class from other ambiguous classes.
3.4 FOLLOW-UP CLASSIFICATION
Since this visually differentiating information resolves the initial ambiguity, after the first round of classification based on the original class descriptions, we create a follow-up classification task with only the ambiguous classes, CA. and the new differential descriptions, D'- Finally, we follow the steps in Section 3.1 to predict the label.
4 EXPERIMENTS
In this section, we show the effectiveness of FuDD through extensive experiments. We show that FuDD outperforms both generic and naive L.L.-generated description ensembles. We design further
5

---

Published as a conference paper at ICLR 2024
Table Accuracy of differential and non-differential descriptions for ambiguous classes. B/32 and L/14" represent the ViT-B/32 and ViT-L/14@: 336px vision backbones. A is the improvement of differential over non-differential descriptions.
Descriptor
CUB
DTD
FGVCAireraft
Flowers102
Food101
B/32
I./1"
B/32
I/14"
B/32
1/14"
B/32
1/14"
B/32
1//1"
Differential
53.62
65.79
45.37
56.91
22.17
39.06
67.62
79.54
84.17
94.34
Non- mineDiffrrental
52.28
64.38
42.82
56.44
22.14
36.90
65.73
77.74
83.92
94.02
11.35
11.42
12.55
00.47
10.03
12.16
11.89
11.81
10.25
10.32
Oxford Pets
Places365
Stanford Cars
Stanford Dogs
B/32
L/14"
B/32
L/14"
B/32
L/14"
B/32
L/14"
Differential
87.24
93.68
42.45
44.26
60,90
79.39
60.31
75,96
Non-Differential
86.24
93.62
41.73
43.98
60.74
78.55
59.30
75.41
11.01
10.06
10.73
10.28
10.16
10.85
11.01
10.55
analytical experiments to show that not all semantic information resolves class ambiguities, and
effective class descriptions should provide information that differentiates the ambiguous classes. Additionally, we find that describing the differences between highly ambiguous classes is the most important, accounting for most of FuDD's performance gains.
Datasets.
We evaluate our method on 12 image recogmition datasets. We use the CUB200-
2011 Wah et al., (01T) (fine-grained bird species), Describable Textures Dataset (DTD) Cimpoi 2014) (texture classification), EuroSAT (Helber et al., 2019) (satellite image classification), CAirerat (Maji et al. 2013) (aircraft model classification) Flowers (Nilsback Zisserman Food101 (Bossard et al. 2014), ImageNet (Deng et al. (2009), ImageNetV2 (Kornblith al.
et al.
FGV
2008
2019
2017). Stanford Cars (Krause Oxford IIIT Pets (Parkhi al. 2012)) Places365 (Zhou et al.
al.
2013), and Stanford Dogs (Khosla al. 2011) datasets.
Setup. We use an instruction-tun tundd GPT-3 model (Brown et al. 3pt-3.5-turie 5tt.5-ttuoo0--- 301,, which is available through OpenAI API ford et al. 2021) as our VLM (refer to appendix for results with other VLMs). Image Net Descrip- tions: Because of the large number of classes in ImageNet, accommodate the API limitations, we cache the pairwise descriptions only for ambiguous classes detected by the ViT-B/32 backbone. In all experiments, we limit the available differential descriptions these cached values (refer to appendix for details).
2020 Ouyang et al. 2022 as LLM, and CLIP (Rad
Baselines. We use three baselines. Single Template: adapt CLIP's pre-training distribu- tion. we adopt A photo of {class name), ). as the class description (Radford et al. 2021) Template Set: we use the 80 generic templates proposed by RRafoord al. (2021) to study the ben- efits of FuDD's better semantic information beyond simple prompt ensembling. Naive LLM: we follow Menon & Vondrick (2023) create naive -..M--geerrted descriptions with the same LLM as ours, which uses prompt like What are useful features for distinguishing (class name in a photo? with a few in-context examples.
4.1 RESULTS
The benefits of using FuDD's semantic information exceed simple description ensembling (Table [ When descriptions are provided for all classes (k=|C"), FuDD outperforms the generic template set on 11 out of 12 datasets with ViT-B/32 (base) and ViT-L/14@@ 336px (large) backbones. Moreover, FuDD is more effective than naive LLM--gncrated descriptions at resolving class ambiguities. On average, FuDD outperforms naive LLM- descriptions by 1.44% and 2.41% 41% with base and large vision backbones, respectively, with up to 13.95% improvements on EuroSAT. Notably, when using the base vision backbone, naive LLM --ge-eertttedd descriptions perform worse than the generic template set on the FGVCAircraft, Oxford Pets, Places365, and Stanford Cars datasets. On the other
https::/openal httt://ccoo/...oce/eeeee

---

Published as a conference paper at ICLR 2024
Cas
EuroSAT
PGVC Ainraf
Flowers
Avcenary
X
K
Fao0101
Grfonl Patx
Placs355
Stanford Cans
Soanked Dops
22
320
3.0
130
Figure 3: Impact of differential descriptions for & most ambiguous classes with VIT-L/14@336pm
k=1 is accuracy with single template. Providing differentiating details for the most ambiguous classes accounts for most of FuDD's gains, with diminishing gains for less ambiguous classes.
hand, FuDD improves accuracy compared to the generic template set by providing differentiating information that resolves ambiguities. Importantly, we observe similar improvements with k=10,, where FuDD only describes the differences between the 10 most ambiguous classes. This empha- sizes the significant impact of class ambiguities on accuracy, which allows computational efficiency, that we will discuss in more detail later.
Importance of Differential Descriptions To study the importance of differentiating information, we compare differential descriptions with non-differential descriptions, which describe character- istics that do not separate the ambiguous classes. We select non-differential descriptions from at- tributes not used by differential descriptions. To control for the number of descriptions, we augment the descriptions with approx. 80 prefixes like image and snapshot, with minimal impact on semantic non-differential descriptions per- form worse than differential descriptions. Non-differential descriptions lead to lower accuracy by at least 1 1% for six datasets, with up to 2.16% and 2.55% for FGVCAircraft and DTD. These results confirm that not all semantic information resolves class ambiguities, and effective class descriptions like FuDD should provide the necessary information to differentiate the ambiguous classes
information (refer to appendix for details). As shown in Table 2
Role of Class Ambiguities Since FuDD mainly focuses ambiguous classes, here, we examine the importance of resolving class ambiguities. Figure plots the accuracy of FuDD for the most ambiguous classes for different values of k, which corresponds varying levels of ambiguity (Sec- tion 3.2p. k = 1 is accuracy with a single template. We find that describing the differences between the five most ambiguous classes accounts for most of FuDD's performance gains, with diminish- ing benefits for less ambiguous classes We can thus get most of the benefit while avoiding high computational costs, especially in the case of diverse datasets and open-set problems.
PUBLICLY AVAILABLE LANGUAGE MODELS
The inaccessibility of proprietary L.MSs like GPT-3.5 hinders further research into fine-tuning LLMs for visual classifica- tion. Here, we use publicly available LLMs to generate the descriptions and study the impact of fine-tuning. Specifically, we fine-tune the 7b-parameter Llama 2 model (Touvron et al. 2023) on the descriptions generated by GPT-3.3 for the Ima- geNet dataset. We find that even the original Ilama 2 model provides useful semantic information for visual classification. Moreover, fine-tuning improves Llama performance, espe- cially for rare concepts like satellite images, achieving compa- rable performance to GPT-3.5.
The percentage of de- scriptions in a random sample that are correct and visually differenti- ating for EurOSAT.
Table 3:
Model
Correct
Useful
Llama 2
82.26
19.35
Llama 2 FT
90.52
48.28
As reported in Table 4 the original Llama 2 provides helpful semantic information for everyday ob-
jects like flowers, outperforming the generie template set. However, it struggles to describe the visual differences for datasets with rare objects like EuroSAT or abstract concepts like DTD. Although the fine-tuned model is not trained on test datasets, it learns the structure of the task and provides differ-
7

---

Published as a conference paper at ICLR 2024
Table 4: FuDD's accuracy with Llama generated descriptions before and after fine-tuning. B/32 and L/14" represent the ViT-B/32 and ViT-L/14@ #136/ 336px vision backbones.
Model
Cub
DTD
EuroSAT
FGVCAireraft
B/32
I/14'
B/32
I/14"
B/32
I/14"
B/32
I/14"
Llama (k=10)
53.14
64.12
41.91
54.47
27.71
37.78
21.03
38.64
Llama (k=|(C))
53.33
64.65
41.91
54.89
27.71
37.78
20.76
37.86
Llama FT (k=10) 10)
53.45
64.81
42,66
56.54
39.14
61.19
22,
38.31
Llama FT (k=|CI)
53.37
64.43
43.14
56.17
39.14
61.19
22.44
39.57
Flowers 102
Food101
Oxford Pets
Stanford Cars
B/32
I/14"
B/32
I/14"
B/32
I/14"
B/32
I/14"
Llama (k==1)
66.03
77.88
83.54
94.00
87.19
93.19
60.27
77.53
Llama (k=[C)
66.16
78.00
84.08
94.15
89.34
93.24
60.48
77.95
Llama FT (k=10)
65.98
77.67
84.32
94.28
86.05
92.67
60.07
78.20
Llama FT (k=|C))
66.55
77.51
84.52
94.25
87.49
92.31
61.05
79.06
entiating visual information for these datasets. Using the ViT-L/14@ backbone, fine-tuning improves the accuracy by 3.25% on average, with up 23.41 % for EuroSAT.
To better understand the impact of fine-tuning, we manually evaluate random subset of pairwise differential descriptions before and after fine-tuning. Through visual inspection, for each pair, we check 1) if each description is correct and 2) if the pair helps differentiate the images of the two classes. Although fine-tuning helps with both measures, it significantly improves the usefulness of shown in Table 3 after fine-tuning, 48% of pairwise descriptions help differ- entiate the two classes, compared to only 19% before fine-tuning. As illustrated Fig. 5 unlike the original model, the fine-tuned model describes attributes that more diverse and focused on low-level visual features rather than higher-level semantic concepts. For more robust analysis, Fig. 4 plots the top-5 most common attributes before and after fine-tuning for EuroSAT. Similarly, the original model describes a limited set attributes with mostly high-level semantic information, while the fine-tuned model generates diverse set of visually differentiating attributes based on the input classes.
the descriptions:
6 FUDD VS. OTHER AUXILIARY INFORMATION
To put the importance of differential descriptions in perspective, we compare FuDD against other approaches for VLM adaptation. We show that FuDD provides more helpful information through differential descriptions compared to simple heuristics like using high-level category names. We find that FuDD can better use the potential of natural language descriptions and achieve comparable performance to other methods that use text-to-image generation models or labeled samples in low- shot settings.
Few-Shot Description Selection We compare FuDD against LaBo, alternative method that uses few-shot learning to select subset of naive L.M-ggenerted class descriptions that are more discriminative (Yang et al. 2023). As reported in Table 5 although FuDD is zero-shot and uses no
building type, 12.5%
texture, 27.9%
color, 28.6%
landscape, 10.7%
Llama 2
land use, 5.4%
Llama
FT
building color, 5.4%
purpose, 7.1%
land cover, 66.1% 1%
shape, 25.0%
size, 11.4%
Figure 4: Top-5 most common attributes described by Llama 2 before and after fine-tuning. The fine-tuned model describes diverse and visually differentiating set of attributes.
8

---

Lake or sea Published as a conference paper at ICLR 2024
Table 5: Accuracy of FuDD compared to LaBo Yang et al. 2023). which uses few-shot learning to select the most effective LLM-generated descriptions for each class with ViT-L/14 backbone. #S is the number of labeled samples.
Method
Cub
DTD
Aircraft
Flowers102
Food101
ImageNet
Acc.
#S
Acc.
#S
Acc.
#S
Acc.
#S
Acc.
#S
Acc.
#S
FaDD(k=10)
64.26
0
56.76
0
37.38
0
78.61
0
93.28
0
75.67
0
FaDD(k=(C))
64.14
57.07
0
37.89
0
79.48
0
93.40
75.99
0
LaBo
54.19
55.26
2
37.71
2
82.05
1
92.45 Full
72.60
16
labeled images, it performs better than 16-shot and full-shot (training on all samples) LaBo on Ima- geNet and F0od101, respectively. For the other four datasets, FuDDD's performance is comparable to LaBo in low-shot scenarios. Unlike LaBo, FuDD encourages the descriptions to be discriminative as part of the generation process, eliminating the need for further optimization.
High Level Concepts WaffleCLIP (Roth et al., 2023 uses high-level category names to address class ambiguities by specifying the dataset context. As shown in Table O in appendix, FuDD per- forms better than WaffleCLIP for seven of the eight datasets. Although high-level category infor- mation helpful, the additional details provided by FuDD are necessary resolve more complex class ambiguities beyond what is caused by similar class names.
Additional Images In Table in appendix, we compare FuDD with CoOp (Zhou et al.) (002b). which uses additional labeled images to learn a set of parameters part of class descriptions. Without us- ing any images, FuDD performs better than 16-shot CoOp Food101 and Oxford Pets datasets and bet- ter than 4-shot CoOp on the ImageNet dataset. On the other five datasets, FuDD's performance is com- parable to CoOp in low-shot settings. We also com- pare FuDD with SuS-X (Udandarao al., (2022). which avoids the additional labeled images by using a pre-trained LLM (Brown et al. 2020) and a text- to- to-image generation model (Rombach et al., [2022) to generate additional images for each class. As re- ported in Table 6 in appendix, despite using im-
Class
Permanent crop land
name
A type of land type of land covez water
Llama
cover with cropland
Browmish color
Blueish color Smooth texture
Rougg texture
Llama FT
geonetric
natural,
ages, FuDD achieves a performance comparable to
shape
irregular
SuS-X by only relying on the LLM-generated de- scriptions. FuDD uses the potential of natural lan- guage more effectively through differential deserip- tions and achieves comparable performance without additional labeled data complexities of using text- to-image generation models.
Shows agriculturat
shape
fields and irrigation systems
Shows islands
and coastlines
Figure 5: Descriptions generated by Llama 2 before and after fine-tuning.
7 CONCLUSION
In this work, we introduce FuDD, a novel zero-shot approach that uses natural language to provide vision-languaa language models with differentiating information about classes in downstream image recog- nition tasks. FuDD identifies potentially ambiguous subset of classes and uses a large language model to generate visually differentiating descriptions that resolve the ambiguity. We show that not all information helps resolve class ambiguities, and effective descriptions should provide dis- criminative information about the ambiguous classes. Well-designed class descriptions, such as the ones produced by FuDD, can achieve comparable performance to few-shot prompt tuning methods in low-shot settings. Our results uncover the potential of natural language for tailoring the class representations to each dataset by providing differentiating information about ambiguous classes. These results motivate future work on creating effective natural language class descriptions for each downstream task.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This material is based on research sponsored by Defense Advanced Research Projects Agency (DARPA) and Air Force Research Laboratory (AFRL) under agreement number FA8750-19-2- 1006. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the offi- cial policies or endorsements, either expressed or implied, of Defense Advanced Research Projects Agency (DARPA) and Air Force Research Laboratory (AFRL) or the U.S. Government. We grate- fully acknowledge support from Google and Cisco. Disclosure: Stephen Bach is an advisor to Snorkel AI, a company that provides software and services for data-centric artificial intelligence.
REFERENCES
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative com- ponents with random forests. In European Conference on Computer Vision, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan. See, think, confirm: Interactive prompting between vision and language models for knowledge-baid visual reasoning. arXiv preprint arXiv:2301.05226, 2023.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-imann learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 2818-2829, 2023.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Peng Gao, Shijie Geng, Renrui Zhang. Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2/ 2021.
Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui. Calip: Zero-shot enhancement of clip with parameter-fiee attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 746-754, 2023.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14953-14962, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp- 770-778, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12())2217-2226, 2019.
10

---

Published as a conference paper at ICLR 2024
Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language mod- els. arXiv preprint arXiv:2204.03649, 2022.
Gabriel Ilbarco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori,
Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi. Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. U httpthts: /ttts:///dee //cci//cci.oo .rg. 1t.p:/10..211.... 5281/ If you use this software, please cite it as below.
zenodo. 5143773
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-433 438,, 2020.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Mudti-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1911 13-1222 2023.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2661- 2671, 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE transactions on pattern analysis and machine intel- ligence, 36(3):453-465, 2013.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.515., 2013.
Cristina Menghini, Andrew Delworth, and Stephen Bach. Enhancing CLIP with CLIP: Exploring pseudolabeling for limited-label prompt tuning. In Thirty-seventh Conference Neural Infor- mation Processing Systems, 2023.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In The Eleventh International Conference on Learning Representations, 2023. URL https:
7/ opoeneeviiew wwwpenrwwiewwwet/ffoeoooeeeeee wttt://oroumiddd111laaaa....
M Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Mateusz Kozinski, Horst Possegger, Rogerio Feris, and Horst Bischof. Lafter: Label-free tuning of zero-shot classifier using language and unlabeled image collections. arXiv preprint 230. 2023.
Nihal Nayak, Peilin Yu, and Stephen H Bach. Learning to compose soft prompts for compositional zero-shot learning, arXiv preprint arXiv:2204.03574, 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision. Graphics and Image Processing, Dec 2008.
Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg, Chils: Zero shot image classification with hicrarchical label sets. In International Conference on Machine Learning, pp- 223222 PMLR, 2023.
11

---

Published as a conference paper at ICLR 2024
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Devi Parikh and Kristen Grauman. Interactively building discriminative vocabulary of nameable attributes. In CVPR 1, pp. 1681-1688. IEFE, 2011.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Puttern Recognition, 2012.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 2085-20944 2021.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Sehastian Riedel. Language models as knowledge bases? arXiv preprint arXiv: arXiv: 1w199999990060 2019.
Sarah Pratt, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. arXiv preprint arXiv:2209.0333 033220000000 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 7748-87655 PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning. pp. 8821 8831. PMLR, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guides dense prediction with context-aware prompting- In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 18082-18091, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings the ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
the IEEE/CVF confer-
Bernardino Roreres and Philip Torr. An embarrassingly simple approach to zero-shot learn- ing. In International conference machine learning. pp. 2152-2161. PMLR, 2015.
Karsten Roth, Jae Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. arXiv preprint arXiv:2306.07288 07282, 2023.
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. Advances in neural information processing systems, 26, 2013.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. Sus-x: Training free name-only transfer of vision-language models. arXiv preprint arXiv:2211.16195 2022.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
12

---

Published as a conference paper at ICLR 2024
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, (((9)2251-22666 2018.
Yue Yang. Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable im- age classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19187-19197, 2023.
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXiv:2210.0722 2022.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, al. Socratic models: Com- posing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.0053 2022.
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free free clip-adapter for better vision-language modeling. arXiv preprint arXiv:21 11.0111..0,0, 2021.
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8552 2262,, 2022.
Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Yu Qiao, Peng Gao, and Hongsheng Li. Prompt, generate, then cache: Cascade of foundation models makes strong few- shot learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15211-15222 2023a.
Zhuosheng Zhang, Aston Zhang. Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023b.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 2017.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1681 (6816-16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-234) 2348, 2022b.
13

---

Published as a conference paper at ICLR 2024
Table 6: Accuracy of FuDD compared to other approaches that use high-level category names, WaffleCLIP (Roth et al. 2023), labeled samples. CoOp (Zhou et al. 2022b), and text-to-image generation models, SuS-X (Udandarao al., (022). with ResNet-50 al. (2016) backbone. #S the number of labeled samples. SuS-X-SD uses synthetically generated images.
Method
Cub
DTD
EuroSAT
Aircraft
Flowers 102
Food101
Acc.
Acc.
#$
Acc.
Acc.
#$
Acc.
#S
Acc.
FuDD(k=10)
49.45
43.51
0
39.42
19.77
0
67.39
80.65
FuDD(k=|CC) [C)
49.26
43.51
0
39.42
19.92
0
68.76
80.95
WafflcCLIP
48.34
39.25
0
35.08
81.38
CoOp
44.39
1
50.63
18.68
2
68.
74.67
16
SuS-X-SD"
49.10
51.00
4
47.69
15
19.92
79
67.32
31
77.02
34
ImageNet
ImageNetV2
Oxford Pets
Places365
Stanford Cars
Acc.
#S
Acc.
Acc.
#S
Acc.
Acc.
FuDD(k=10)
60.69
0
53.19
86.86
0
40,64
0
56.62
FuDD(k CD)
60.78
53.60
87.52
40.69
56.77
WafflcCLIP
60.12
0
52.89
85.80
0
39.03
CoOp
59.99
87.01
16
55.59
SuS-X-SD' 61.65
36
85.09
71
57.14
5
A DIFFERENT VISION ENCODERS
In general, the benefits of FuDD over the generie template are more significant for smaller models (Fig. (6a). However, larger vision encoders can better take advantage of the nuanced information provided by FuDD beyond naive LLMeggeeeateed descriptions (Fig. 6b). We believe as image-text representations improve, VLMs can better take advantage of available semantic information, making natural language class descriptions even more important for vision tasks in the future.
B LLM PROMPTING DETAILS
We use gpt-3. 5-..---uu-o333 0301 to generate the differential descriptions.
3pt-3.5-turbbr 5..-urbu-o-0000 0301 is GPT-3 model that is fine-tuned to follow instructions (Brown et al. [2020] Ouyang et al. 2022). In this form, the model is given a sequence of user and assistant messages and expected generate the next assistant message. In our experiments, we encode two fixed sample outputs as assistant messages and ask the model to generate similar output for a given pair of classes. Specifically, we use the template messages Table 9 and replace class name and class name 2 with the desired classes to get the corresponding pairwise differential descriptions.
ImageNet Descriptions. As mentioned in Section 4 because of the large number of classes in the ImageNet dataset, it is not possible to generate the differential descriptions for all class pairs. Since
Dook
Aay. Aec. Baoe
3.25
3.5
2.00
3.0
3,25
2.80
3.83
8800
yrp-819
vmn4-114
yrr-B/te
(a)
(b)
Figure 6: FuDD's average accuracy boost for different vision backbones compared to a) generic template set and b) naive LLM-ggnerated descriptions
14

---

Published as a conference paper at ICLR 2024
Table Total API costs for each dataset in Dollars
Dataset
Cost
Cub
15.47
DTD
0.84
EuroSAT
0.03
FGVCAircraft
3.85
Flowers
4.00
Food101
3.93
ImageNet
60.24
Pets
0.52
Places365
51.63
Stanford Cars
14.85
Stanford Dogs
5.55
differential descriptions are the most effective for ambiguous classes, we generate and cache all the pairwise descriptions for ambiguous classes and limit the available pairwise differential descriptions to this set. Following Section 3.2 we use ViT-B/32 vision backbone with k 5 to detect the most ambiguous classes. For all pairs in each set ambiguous classes, we generate and cache the corre- sponding pairwise differential descriptions, D explained in Section 3.3) In all our experiments, we use the cached pairwise differential descriptions, D If the cache does not exist for a class pair, use single template description instead, i.c., D {A photo of class name.
C DIFFERENTIAL VS. NON-DIFFERENTIAL DETAILS
For the non-differential experiments in Table 2 we select set of descriptions that describe set of details for each class that does not differentiate it from other ambiguous classes. For some class c, to create non-differential descriptions. we first collect all the available differential descriptions, which equal to the differential descriptions with K CC|. Next, we create the normal set of diffcrential descriptions, D', with K 10 as explained in Section 3 As result of description generation method, we know the corresponding attribute for each of the differential descriptions. Now, we filter the set of all available differential descriptions exclude all the descriptions that their corresponding attribute is similar to the attributes explained by descriptions in D' The remaining descriptions do not include any attribute that helps to separate the ambiguous classes.
We consider two attributes to be similar if they share a common word. For instance, color and coat coler are similar since they share the word color. We split the attributes by white space and use simple string matching to check for this criteria. Because of the lack of diversity in the available descriptions for DTD, Oxford Pets, and Stanford Dogs datasets, using this criteria leads to a small number of remaining non-differential descriptions for each class. Therefore, for fair comparisons, we relax the criteria for these three datasets and compare attributes without splitting by white space, i.e., color and coat color are not considered similar for these three datasets.
As mentioned by previous work, the number descriptions for each class also impacts the aceu- racy (Radford et al., 2021; Roth al. (023). We use description augmentation to control for the number of prompts for both differential and non-differential descriptions. Description augmentation creates a large number of descriptions from the original class descriptions with minimal impact semantic information. Specifically, we create an augmented set of descriptions for each class de- scription by replacing the original prefix (e.g. photo of a) with a of similar prefixes like an image of and a snapshot of a. We use the prefixes used in Radford et al.) (2021) Now calculate the augmented description embedding, we average over the embeddings of all the descriptions in the corresponding augmented description set.
D COSTS
Although FuDD queries the LLM more than naive approaches, the API calls are very affordable and do not hinder wider adoption of FuDD. On average, one input prompt and model response
15

---

Published as a conference paper at ICLR 2024
is 380 and 199 tokens, respectively. With OpenAl pricing at the time of writing ($0.001/1k and S0.002/1k tokens for imput and response), the cost is $0.78 per 1000 queries, leading to affordable prices reported in Table FuDD also accommodates datasets with large number of classes like ImageNet by recognizing the more significant role of ambiguous classes, reducing the costs for ImageNet dataset from $388 to $60 (see Appendix [ for details).
In addition, as studied extensively in Section 5 we can use off-the-shelf or fine-tuned LLMs like Llama generate differential descriptions using in-house hardware avoid API costs accom- modate other issues like working with private and sensitive data.
E ADDITIONAL VLMS
we repeat our main experiments with different VLMs. We choose Open- CLIP (Cherti et al. 2023; Ilharco al., 2021) because of its superior performance to CLIP. For example, OpenCLIP with ViT-L/14 backbone trained on datacomp x1 s13b b90k improves the performance of its CLIP counterpart by percentage points on ImageNet dataset using Single Template descriptions. We also run experiments using BLIP-2 (Li et al. (023) because it is trained using an entirely different strategy with combination of image-text contrastive loss, image-text matching loss, and generation loss. To calculate image-text similarity using BLIP-2, we follow similar procedure to et al. (2023). As reported Table these other VLMs can also use ad- ditional information provided by FuDD and improve the performance beyond naive LLM --geeerrateed descriptions.
To further evaluate FuDD,
16

---

Published as a conference paper at ICLR 2024
Table FuDD accuracy using OpenCLIP and BLIP2 models. DComp is the checkpoint trained on datacomp.xl..oo _ttppp.11_113bbb00000 and Laion is the checkpoint trained on lalon2b_el 6 dataset.
Description
Cub
DTD
VIT-B-32
ViT-L-14
BLIP2
ViT-B-32
VIT-L-14
BLIP2
DComp
Laion
DComp
DComp
Laion
DComp
Single Template
72.63
63.93
85.31
23.52
54.95
51.12
63.19
53.24
Template Set
72.52
63.12
85.00
27.36
55.80
52.71
64.79
55.53
Naive
73.63
63.79
85.66
27.51
58.99
56.86
66.22
55.48
FuDD (k-10) 10) 73.42
63.91
85.69
28.51
58.24
55,43
67.23
$6.70
FuDD (k= (I=IC)
73.82
64.19
86.16
28.77
57.45
54.95
67.77
56.86
EuroSAT
FGVCAircraft
ViT-B-32
ViT-L-14
BLIP2
ViT-B-32
ViT-L-14
BLIP2
DComp
Laion
DComp
DComp
Laion
DComp
Single Template
38.03
41,73
61.14
52.14
29.94
26.31
51.82
14.19
Template Set
41.02
41.44
61.62
51.01
30.75
26.46
51.88
14.46
Naive
49.28
45.37
69.72
63.63
31.41
25.77
52.09
16.35
FuDD (k=10) 10)
55.74
57.27
74.05
70.84
31.59
26.67
50.89
15.54
FuDD (k=
55.74
57.27
74.05
70.84
31.38
26.43
51.25
16.44
Flowers 102
Food101
ViT-B-32
VIT-L-14
BLIP2
ViT-B-32
VIT-L-14
BI.IP2
DComp
Laion
DComp
DComp
Laion
DComp
Single Template
72.13
67.49
80.60
55.91
85.89
81.31
94.49
85.07
Template Set
72.06
67.67
81.12
57.85
85.75
81.24
94.39
86.74
Naive
73.18
66.53
79.90
60.81
85.49
81.48
94.05
87.71
FuDD
73.98
69.
83.35
61.60
86.40
81.50
94.43
88.50
FuDD (k=IC)
75.05
69.91
83.02
61.21
86.49
81.86
94.51
88.63
ImageNet
ImageNet V2
VIT-B-32
VIT-L-144
BLIP2
VIT-B-32
VIT-L-14
BI.IP2
DComp
Laion
DComp
DComp
Laion
DComp
Single Template
68.44
65.20
78.83
60.93
60.33
56.91
71.96
56.20
Template Set
69.13
65.61
79.15
66.07
60.76
57.36
72.05
60.60
Naive
68.60
65.42
79.03
66.15
60.06
57.19
71.92
61.00
FuDD
69.13
65,91
79.25
67.31
60.94
57.70
72.08
61.28
FuDD JCD
69.35
66.20
79.50
68.55
61.28
57.90
72.39
62.53
Oxford Pets
Places365
VIT-B-32
VIT-L-14
BLIP2
VIT-B-32
VIT-L-14
BLIP2
DComp
Laion
DComp
DComp
Laion
DComp
Single Template
89.40
87.54
94.74
76.70
41.52
41.88
43.21
43.72
Template Set
88.47
87.49
93.51
76.91
43.20
42.84
44.73
43.67
Naive
89.86
89.07
94.79
81.17
42.24
42.61
44.06
43.51
FuDD
90.71
89.04
94.90
81.96
42.79
43.16
44.98
45.30
FuDD (CD
90.95
90.00
95.18
83.51
43.13
43.55
44.89
45.52
Stanford Cars
Stanford Dogs
ViT-B-32
ViT-L-14
BLIP2
VIT-B-32
ViT-L-14
BLIP2
DComp
Laion
DComp
DComp
Laion
DComp
Single Template
88.42
86.82
93.67
79.97
63.92
59.76
79.23
47.70
Template Set
88.66
87.02
93.71
80.48
64.93
59.50
79.22
47.76
Naive
87.38
86.76
93.56
80.08
65.02
60.12
79.22
50.31
FuDD (A= 10)
88.35
86.87
93.77
80.25
65.13
59.84
79.39
52.42
FuDD (k=IC) CCD
88.45
86.84
93.65
81.06
65.96
60.22
80,29
52.91
17

---

Published as a conference paper at ICLR 2024
Class
Class O: Tennessee Warbler
Black-footed Albatross
Attribute: size
A photo a tennessce warbler, small songbird only about 4 inches long. A photo of black-footed albatross, a large seabird with a wingspan of up to feet.
Attribute: coloration
O: A photo of tennessee warbler, bright yellow bird with olive-green wings and back.
1: A photo of black-footed albatross, dark-colored bird with a white head and underparts.
Attribute: bill shape
A photo of tennessce warbler, bird with small, pointed bill. large, hooked bill.
A photo a black-footed albatross,
Mangrove Cuckoo
Attriburc: length
A photograph of a tennessee warbler, type of bird, with a short, pointed bill.
I: A photograph of a mangrove cuckoo, type of bird, with a long, curved bill.
Attribute: tail length
0: A photograph of lennessee warbler, type of bird, with short, square tail,
A pholograph of mangrove cuckoo, type of bind, with long, graduated tail.
Attribute: wing bars
A photograph of a tennessce warbler, type of bird, two white wing bars. A photograph of mangrove cuckoo, type of bird, with no wing bars.
Caspian Tern
Attribute: size
0: A photogruph of a tennessce warbler, type of bird, which is smaller in size. 1: A photograph of a caspian tern, type of bird, which is larger in size.
Attribute: bill color
photograph of tennessco warbler, type of bird, with black bill. A photograph of caspian ter,, type of bird, a red bill.
Attribute: wing shape
A photograph of a tennessce warbler, type of bird, with short, rounded wings.
1: A photograph of caspian ter,, type of bird, with long, pointed wings.
Black-throated Sparrow
Attribute: head pattern
0: A photograph of tennessee warbler, type of bird, with yellow and gray head pattern,
1: A photograph of black-throated sparrow, type of bird, with black and white striped head pattern.
Attribute: eye ring
photograph of tennessee warbler, type of bird, with eye ring. A photograph of a black-throated sparrow, a type bird, with white eye ring.
Attribute: breast color
0: A photograph of a tennessee warbler, type of bird, with ycllow brcast color.
1: A photograph of black-throated sparrow, a type of bird, with black and white striped breast color.
Figure 7: Various pairwise differential descriptions for sample class "Tennessee Warbler" and four different classes.
18

---

Published as a conference paper at ICLR 2024
Table Prompt template for generating pairwise differential descriptions. We use two fixed sample pairs of classes to guide the model to generate the appropriate output.
Type
Content
System
You are knowLedgeable teacher.
User
For the following objects, generate captions that represent the distinguishing visual differences between the photos of the two objects. Generate many captions you
Object 1:
Field sparrow
object 2: Acadian flycatcher
Assistant
<START>
Visual characteristic: Bill shape
Caption 1: A photograph of a field sparrow, type of bird, with cone bill shape. Caption 2: A photograph of acadian flycatcher, type of bird, with all-purpose bill shape.
Visual charactoristic: Breast color
Caption 1: A photograph of a field sparrow, type of bird, with buff breast color. Caption 2: photograph of acadian flycatcher, type of bird, with white breast color.
Visual characteristie: Back pattern
Caption 1: photo of field sparrow, type of bird, with striped back pattern. Caption 2: A photo of acadian flycatcher, type of bird, with solid back pattern. <END>
User
For the following objects, generate captions that represent the distingulshing visual differences between the photos of the two objects. Generate as many captions you
Object 1: Cornet
Object 2: Flute
Assistant
<START>
Viaual characteristic: Shape
Caption 1: photo of a cornet, a type of musical instrument, with a conical bore. Caption 2: A photo of flute, a type of musical instrument, with cylindrical bore.
<END>
User
For the following objects, generate captions that represent the distinguishing visual differences between the photos of the objects.
Gonerate as many captions you can.
object 1: {class name 1)
Object 2: (class name 2)
19

---

Published as a conference paper at ICLR 2024
Table 10: Sample descriptions generated by GPT-3.5 for Cub
Classes
Descriptions
Attribute: size
0: photograph of a caspian tern, type of bird, which is larger in size.
1: photograph of tennessee warbler, type of bird, which is smaller in size.
Attribute: bill color a photograph of a caspian tern, a type bird, with red bill. 1: a photograph of a tennessee warbler, type of bird, with black bill.
Caspian Tern
Attribute: wing shape 0: a photograph of a caspian tern, a type of bird, with long, pointed wings.
1: a photograph of a tennessee warbler, type of bird, with short, rounded wings.
Attribute: tail shape
0: a photograph of a caspian tern, a type of bird, with forked tail. 1: a photograph of a tennessee warbler, type of bird, with square tail.
Tennessee Warbler
Table 11: Sample descriptions generated by GPT-3.5 for DTD
Classes
Descriptions
Attribute: texture
0: a photo of an interlaced surface, with
woven texture.
1: a photo of a stratified surface, with layered texture.
Attribute: pattern
0: a photo of an interlaced surface, with criss-cross pattern.
Interlaced
1: a photo of stratified surface, with horizontal pattern.
Attribute: material
0: photo of interlaced surface, made of woven fibers.
1: photo of stratified surface, made of layered sediment or rock.
Stratified
20

---

Published as a conference paper at ICLR 2024
Table 12: Sample descriptions generated by GPT-3.5 for EuroSAT
Classes
Descriptions
Attribute: color
0: a satellite image of industrial or commercial buildings, with mix of grey, white, and black colors.
a satellite image of a river, with blue and green colors.
Attribute: texture
0: a satellite image of industrial or commercial buildings, with a rough and angular texture. 1: a satellite image of river, with a smooth and flowing texture.
Industrial or commercial
building
Attribute: shape
0: a satellite image of industrial or commercial buildings, with rectangular and square shapes. 1: satellite image of river, with winding and curvy shape.
Attribute: pattern
0: satellite image of industrial commercial buildings, with a grid-like pattern.
1: satellite image of river, with meandering pattern.
River
Table 13: Sample descriptions generated by GPT-3.5 for Flowers
Classes
Descriptions
Attribute: color
0: a photo of bird of paradise, a type of
flower, with bright orange and blue colors.
1: a photo of a globe thistle, a type of flower, with muted blue and green colors.
Attribute: shape
0: a photo of bird of paradise, a type of flower, with unique bird-like shape.
Bird of paradise
1: a photo of globe thistle, type of flower, with a spherical shape.
Attribute: texture
0: a photo of bird of paradise, a type of flower, with smooth and glossy petals.
1: photo of globe thistle, type of flower, with spiky and rough texture.
Globe thistle
21

---

Published as a conference paper at ICLR 2024
Table 14: Sample deseriptions generated by GPT-3.5 for Food101
Classes
Descriptions
Attribute: type of food a photo
hot dog, type of fast food, with a sausage in bun. 1: a photo of sashimi, a type of japanese cuisine, with raw fish slices.
AAG Cilleimeaa
Attribute: cooking method
0: a photo of hot dog, type of fast food, with grilled sausage.
Hot dog
1: a photo of sashimi, type of japanese cuisine, with raw fish slices.
Attribute: serving style
0: a photo of hot dog, type of fast food, served with ketchup and mustard.
1: photo of sashimi, type of japanese cuisine, served with soy sauce and wasabi.
Attribute: texture
0: photo of hot dog, type of fast food, with a chewy texture.
1: a photo of sashimi, type of japanese cuisine, with soft and tender texture.
Sashimi
Table 15: Sample descriptions generated by GPT-3.5 for ImageNet
Classes
Descriptions
Attribute: number of wheels 0: a photo of minivan, which has four wheels. a photo rickshaw, which has three wheels.
Attribute: size
0: a photo of a minivan, which is larger in size and can accommodate more passengers.
1: a photo of a rickshaw, which is smaller in size and can accommodate fewer passengers.
Minivan
Attribute: propulsion
0: a photo of minivan, which is powered by an engine.
1: a photo of rickshaw, which is powered by human pedaling.
Attribute: type of vehicle
0: photo of minivan, which is modern automobile.
1: photo of rickshaw, which is a traditional asian vehicle.
Rickshaw
22

---

Published as a conference paper at ICLR 2024
Table 16: Sample descriptions generated by GPT-3.5 for Oxford Pets
Classes
Descriptions
Attribute: body type
0: a photo of german shorthaired, type of dog, with a muscular and athletic body type.
1: photo of sphynx, type of cat, with slender and sleek body type.
Attribute: coat
0: a photo of german shorthaired, type of dog, with a short and dense coat. 1: a photo of sphynx, type of cat, with no coat or hair.
German shorthaired
Attribute: ears
0: a photo a german shorthaired, a type dog, with floppy ears.
1: a photo of a sphynx, a type of cat, with large and pointed ears.
Attribute: facial features
0: a photo of german shorthaired, a type of dog, with snout and prominent nose. 1: a photo of sphynx, type of cat, with flat face and no visible nose bridge.
Sphynx
Table 17: Sample descriptions generated by GPT-3.5 for Stanford Cars
Classes
Descriptions
Attribute: body type
0: a photo of 2012 hyundai veracruz suv, large vehicle with a high ground clearance and a boxy shape.
1: a photo of a 2009 spyker c8 convertible, a sleek and low-slung sports car with a convertible top.
Attribute: number of doors
0: photo of 2012 hyundai veracruz suv, vehicle with four doors. 1: a photo
2012 Hyundai Veracruz SUV
2009 spyker c8 convertible, vehicle with two doors.
Attribute: wheel design
0: a photo
2012 hyundai veracruz suv, with standard alloy wheels.
1: a photo of a 2009 spyker c8 convertible, with unique and intricate spoke wheels.
Attribute: grille design
0: a photo of 2012 hyundai veracruz suv, with large and prominent grille.
2009 Spyker C8 Convertible
1: a photo of 2009 spyker c8 convertible, with small and distinctive grille.
23

---

Published as a conference paper at ICLR 2024
Table 18: Sample descriptions generated by GPT-3.5 for Stanford Dogs
Classes
Descriptions
Attribute: size
0: photo of schipperke dog, small breed of dog.
1: a photo of saint bernard dog, large breed of dog.
Attribute: coat color
photo schipperke dog, with black coat color.
Schipperke
a photo of saint bernard dog, with white and brown coat color.
Attribute: ear shape
0: a photo of a schipperke dog, with pointed ears.
1: photo of saint bernard dog, with droopy
ears.
Attribute: tail length
0: photo of schipperke dog, with a short tail. 1: photo of saint bernard dog, with a long tail.
Saint bernard
Table 19: Sample descriptions generated by GPT-3.5 for FGVC Aircraft
Classes
Descriptions
Attribute: size
0: a photo of an airbus a319 aireraft, commercial airliner that is much larger than a cessna 172.
1: a photo of cessna 172 aircraft, a small, single-engine plane that is much smaller than an airbus a319.
Attribute: wing shape
0: a photo of an airbus a319 aircraft, with swept-back wings. 1: a photo of cessna 172 aircraft, with straight wings.
Airbus A319
Attribute: engine placement.
0: a photo of an airbus a319 aircraft, with
engines mounted under the wings.
BRZS
1: a photo of cessna 172 aircraft, with
single engine mounted on the nose of the plane.
Attribute: cockpit windows
0: a photo of an airbus a319 aircraft, with large cockpit window that extends over the top of the plane.
Cessna 172
1: a photo of a cessna 172 aircraft, with a small, single-piece windshield in the cockpit.
24

---

Published as a conference paper at ICLR 2024
Table 20: Sample descriptions generated by GPT-3.5 for Places365
Classes
Descriptions
Attribute: location
0: a photo of a pier, which is located near a body of water.
1: a photo of plaza, which is located in city or town center.
Attribute: puzpose
0: a photo of pier, which is used for docking boats and ships.
1: photo of plaza, which is used for public gatherings and events.
Pier
Attribute: design
0: photo of pier, which is typically long and narrow with a flat surface.
1: photo of plaza, which is typically open and spacious with various features like fountains, benches, and sculptures.
Attribute: surroundings
0: a photo of pier, which is surrounded by water and may have views of the ocean or other bodies of water.
1: a photo of plaza, which is surrounded by buildings and may have views of city streets and architecture.
Plaza
25


Reviews:
[{"review_id": "94kc6k8DdA", "reviewer": "Reviewer_RXgc", "summary": "Recent work shows that evaluating vision-language models with descriptions on object attributes improves image classification performance. This work proposes to prompt LLMs to specifically give “differential” descriptions (i.e., distinctive attributes between ambiguous classes) and using such more targeted descriptions outperforms generic descriptions. The method shows performance improvement over the baselines on a range of image classification benchmarks.\n\nAfter rebuttal: I appreciate the author's rebuttal and have raised my score to 5. However, I still believe that the contribution seems thin given that only a prompting technique is proposed.  In addition, compared to the baseline, the performance improvement is oftentimes not significant (Table 1); thus the generalization ability of the method is not very clear.", "questions": "- For the equation in 3.2, what does the summation mean? I guess the equation simply means getting the top-k classes with the highest similarity with the image features? Then in this case, do we need to sum over c_i?\n\n- Would it be possible to give examples of differential descriptions for each of the evaluation datasets? It is especially interesting to see the huge performance on EuroSAT.\n\n- Would it be possible to prompt the LLM to come up with differential attributes given >2 classes in a single prompt?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea is simple and natural. Directly prompting LLM to give attributes for a certain class is an underspecified task; there are so many attributes of different granularities to enumerate. Generating differential descriptions given two classes is a smart way to generate targeted attributes.", "weaknesses": "**1. No significant method improvement**\n\nThe idea of prompting LLM to give more differentiating descriptions is effective but not necessarily a significant innovation. \n\n\n**2. Reliance on pre-trained vision-language models**\n\nThe paper focuses only on getting differential descriptions; however, it is not guaranteed that CLIP can accurately interpret them. The effectiveness of such models hinges critically on and is bounded by the vision-language models. I suspect that this could be why the performance improvement is not very significant and why different CLIP variants could exhibit huge performance variations.\n\nI would appreciate an error analysis: how many of the mistakes are due to insufficient / erroneous differential descriptions and how many of the mistakes are due to CLIP’s insensitivity to descriptions? This could be informative for future work.\n\n\n\n**3. API cost for a single inference on an image**\n\nIt is not immediately clear how many times we need to query LLM for a single inference on one image. For k=10, does it mean we need to create 55 queries? For experiments where k = |C|, will the cost become prohibitive (even with caching)?\n\nIt would be better if such details are discussed in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kf2gPgr0Ey", "reviewer": "Reviewer_sQgp", "summary": "This paper proposed a simple data augmentation technique for zero-shot image classification tasks. Specifically, the idea is to utilize large language models (LLM) to generate more descriptive and informative class labels for ambiguous classes, with the hope that they can help the base image-text retrieval/classification model to classify the image. Experimental results suggest promising gains over baseline, in this case a CLIP model without this augmentation technique.", "questions": "NA", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is intuitive. It is expected that using more descriptive class labels can improve zero-shot tasks.\n2. Experimental results show consistent gains across different datasets.", "weaknesses": "1. The method is incremental, as it is a small trick to boost zero-shot image classification results.\n2. The baseline is not strong enough. It is therefore not clear if the method can still improve state-of-the-art models on these tasks.\n3. More diverse and challenging tasks are missing from the experiment, e.g. image-text retrieval. It will be nice to show wider applicability of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7xhABard4y", "reviewer": "Reviewer_Y5Zb", "summary": "This paper studies the prompt engineering. Instead of directly asking LLM or designing a set of prompts, the author proposed to let the model decides which sets of categories are ambiguous and then ask the LLM to clarify those sets of categories. This is actually a quite interesting and smart idea. Comparing with the other prompt engineering approach, the proposed one achieves higher performance.", "questions": "1. The section 5: I wonder why would the Llama2 can generate informative sentences for satellite images? Especially given the model only been finetuned on the ImageNet dataset sentences.\n\n2. What if the caption is so long and so complex that the image-text model could not understand? I wonder if the author encountered this scenario, if so, how to address it?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed idea makes a lot of sense and achieves quite impressive performance.\n\n2. The paper showed that by finetuning the open-sourced model, the open-sourced model (LLAMA2) could also generate the descriptions with discriminative capability across ambiguous categories.\n\n3. The proposed approach achieves much better performance than simple prompt engineering.\n\n4. If this is the first paper presented in this field, I would rate a strong acceptance. This is a little bit out of my domain, I would rate acceptance.", "weaknesses": "I think this paper is interesting and the proposed approaches make sense. I would not be able to state the weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sqWRQUhUZ7", "reviewer": "Reviewer_LDou", "summary": "This paper studies how to extend the class descriptions to improve the performance of CLIP-style vision-language models (VLMs) in image classification. It points out that the class descriptions generated by existing methods may not provide useful information that differentiates the class from other classes. \n\nTo address this problem, this paper proposes the Follow-up Differential Descriptions (FuDD). FuDD first identifies a set of ambiguous classes using the basic class descriptions. Then, for each ambiguous class, a LLM is prompted to generate descriptions, which involve information that can distinguish it from other ambiguous classes. The generated class descriptions are then used to obtain the final classification results.\n\nThe empirical studies show that FuDD outperforms the basic class descriptions and naïve LLM generated descriptions that do not consider differential attributes between classes. The experiments include two kinds of LLMs, i.e., the GPT-3.5 (accessed via API) and the open-sourced LLAMa2. It is shown that LLAMa2-based FuDD can achieve effective results after being fine-tuned on the descriptions generated by GPT-3.5.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The idea of generating differential class descriptions is well-motivated and reasonable.\n* The experiments are well-designed and the results are convincing, which demonstrates the effectiveness of using differential class descriptions.\n* The paper is well-written and easy to follow.", "weaknesses": "* The experiments only include one type of VLM, i.e., CLIP. The reviewer would like to know whether FuDD is compatible with more advanced VLMs.\n* There is no analysis on the trade-off between performance gain by FuDD and additional (computational or financial) cost. Since generating pair-wise class description is more expensive than generating independent description for each class, such analysis is important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)