What does a platypus look like? Generating customized prompts for zero-shot image classification
Authors: Sarah M Pratt
Rosanne Liu
Ali Farhadi
Venue: Submitted to ICLR 2023 (2023)
Keywords: zero-shot
image classification
prompts
open vocabulary models

Abstract: Open vocabulary models are a promising new paradigm for image classification. Unlike traditional classification models, open vocabulary models classify among any arbitrary set of categories specified with natural language during inference. This natural language, called "prompts", typically consists of a set of hand-written templates (e.g., "a photo of a {}”) which are completed with each of the category names. This work introduces a simple method to generate higher accuracy prompts, without relying on any explicit knowledge of the task domain and with far fewer hand-constructed sentences. To achieve this, we combine open vocabulary models with large language models (LLMs) to create Customized Prompts via Language models (CuPL, pronounced "couple"). In particular, we leverage the knowledge contained in LLMs in order to generate many descriptive sentences that are customized for each object category. We find that this straightforward and general approach improves accuracy on a range of zero-shot image classification benchmarks, including over one percentage point gain on ImageNet. Finally, this simple baseline requires no additional training and remains completely zero-shot.

Under review as a conference paper at ICLR 2023
WHAT DOES A PLATYPUS LOOK LIKE? GENERATING CUSTOMIZED PROMPTS FOR ZERO-SHOT IMAGE CLASSIFICATION
Anonymous authors Paper under double-bline review
ABSTRACT
Open vocabulary models are a promising new paradigm for image classification. Unlike traditional classification models, open vocabulary models classify among any arbitrary set of categories specified with natural language during inference. This natural language, called "prompts", typically consists of of hand-writte templates (e.g.. "a photo of ()") which are completed with each of the cate- gory names. This work introduces simple method to generate higher accuracy prompts, without relying on any explicit knowledge of the task domain and with far fewer hand-constructed sentences. To achieve this, we combine open vocab- ulary models with large language models (LLMs) to create Customized Prompts via Language models (CuPL, pronounced "couple"). the knowledge contained in LLMs in order to generate many descriptive sentences that are eustomized for each object category. We find that this straightforward and general approach improves accuracy benchmarks, including over one percentage point gain on ImageNet. Finally, this simple baseline requires additional training and remains completely zero-shot.
particular, we leverage
range of zero-shot image classification
Standard Zero-shot
Customized Prompts Language models (CuPL)
"A platypus looks like
Text encoder
Image encoder
Text encoder
Image encoder
GPT-3
"A photo of goldfish"
"What does a platypus look like?"
"Gokdfish are smell orange fish with shiny scales
platypus"
"A platypus looks like beaver duck's bill
"A photo of spatula!
Figure 1: Schematic of the method. (Left) The standard method of a zero-shot open vocabulary image classification model (c.g., CLIP (Radford et al.] (202I)) (Right) Our method of CuPL. First, an LLM generates descriptive captions for given class categories. Next, an open vocabulary model uses these captions as prompts for performing classification.
INTRODUCTION
Open vocabulary models (Pham et al., [002 Jia et al. 2021 Radford et al. 2021 Yu et al. 2022a) achieve high classification accuracy across large number of datasets without labeled training data for those tasks. To accomplish this, these models leverage the massive amounts of image-text pairs available the internet by learning to associate the images with their correct caption, leading to greater flexibility during inference. Unlike standard models, these models classify images by pro- generate viding similarity score between image and caption. To perform inference,

---

Under review as a conference paper at ICLR 2023
caption or "prompt" associated with each of the desired categories, and match each image the best prompt. This means that categories can be selected ad hoc and adjusted without additional training.
However, this paradigm poses challenge:
How can we best represent image category through natural language prompts?
The standard approach is to hand write a number of prompts templates (Radford et al. 2021) (e.g.. "a photo of a ( compile natural language label for each category in the dataset, and create a set of prompts for each category by filling in each of these templates with the natural language labels. Then, image embeddings are matched to the nearest set of prompt embeddings and labelled with the category associated with that set of prompts (more details in Section 2
This method has three major drawbacks. Firstly, each prompt template has be hand-written, so having twice as many prompts for category requires twice much human effort. This can become costly as each new dataset typically has a different set of prompt templates (Radford al.) 2021). Secondly, the prompt templates must be general enough to apply to all image categories. For example, a prompt for the ImageNet (Deng et al., 2009) category "platypus" could only be as specific "a photo of {platypus)" and could not be something like "a photo of {platypus). type of aquatic mammal" as that template would no longer be relevant for other image categories. Lastly, writing high performing prompt templates currently requires prior information about the contents of the dataset. For example, the list of hand-written ImageNet prompts (Radford et al.) 2021) includes "a black and white photo of the ().", "a low resolution photo of UJ., and "a toy f all of which demonstrate prior knowledge about the type of representations present in the dataset. This information is not generalizable to other datasets, as ImageNet contains "black and white" and "toy" representations of its categories, but other datasets do not (e.g., FVGC Aircraft (Maji al., 2013)).
To overcome these challenges, we propose Customized Prompts via Language models (CuPL). In this algorithm, we couple large language model (LL.M) with zero-shot open vocabulary image classification model. We use the LLM to generate prompts for each of the image categories in dataset. Using an LLM allows us to generate an arbitrary number of prompts with fixed number of wand-written sentences. Additionally, these prompts are now customized to each category and can contain rich visual descriptions while still remaining zero-shot (e.g., "A platypus looks like beaver with duck's 's bill" sentence generated by LLM).
We find these customized prompts outperform the wand-written templates on 15 zero-shot image classification benchmarks, including greater than 1 percentage point gain ImageNet (Deng let al., 2009) Top-1 accuracy and greater than 6 percentage point gain on Describable Textures Dataset (Cimpoi al. (014), with fewer hand-written prompts when compared to the standard method used in Radford et al. (202I). Finally, this method requires no additional training or labeled data for either model.
METHODS
The CuPL algorithm consists two steps: (1) generating customized prompts for each of the cate- gories in given dataset and (2) using these prompts to perform zero-shot image classification.
2.1 GENERATING CUSTOMIZED PROMPTS
This step consists of generating prompts using LLM. For clarity, distinguish between two different kind of prompts. The first are the prompts which cue the LLM to generate the descriptions of the dataset categories. These prompts do not describe object, but rather prompt the description of an object (c.g.. "What does platypus look like?"). We will refer to these as PLLM-prompts"
Secondly, there are the prompts to be matched with images in the zero-shot image classification model. These are the prompts that describe a category (e.g., "A platypus looks like ..."). We call them "image-prone prompts. These are the output of the LLM, examplified in Figure
In this work, we use GPT-3 (Brown et al. 2020) as our LLM. To generate our image- mnageeprompps, we must first construct number of M...-prompt templates. While this does require some engineering
2

---

Under review as a conference paper at ICLR 2023
ILM-prompts:
Image-prompte
"What does (lorikeet, marimba, viaduct, papillon) look like?"
"A lorikeet is a small to medium-sized perrot with a brightly colored plumage.
GPT-3
"A marimba is a large wooden percussion instrument that looks like a xylophone.
"A viaduct bridge composed of several spans supported by piers or pillars"
'A papillon is small, spaniel-type dog with long, silky coat and fringed ears."
Lorikeet
Marimba
Viaduct
Papillon
Example CuPL LLM-prompts and Image-prompts. LLM-promppss are filled in with class name and then used input GPT-3, which then outputs image-prompts. Example LLM generated image-prone prompts and associated images from ImageNet are shown. Only image-prompts are used for the downstream image classification.
Figure 2:
by hand, it is significantly less than the amount of hand-engineered sentences used in the standard method creating imaee-promp templates for CLIP. For example, in our ImageNet experiments, we construct 5 LLM-prompt templates compared to the 80 image-prompts used by CLIP for zero- shot ImageNet classification.
After constructing these LLM-prompts. we generate 10 different image-prompts for each of the L.M.-proppts. This means for ImageNet we use an L.L to generate total of 50 customized image-prompts for each image category. For each of these, generate maximum of 50 tokens, but halt generation early it produces a period. Additionally, we generate with a high temperature of 0.99, which encourages more diversity among the 10 generated image-prompts, We also clean each generated sentences by deleting any blank lines and adding a period at the end.
2.2 UTILIZING CUSTOMIZED PROMPTS
After generating image-prompts for each of the categories, we then perform zero-shot image classi- fication. While there are a number of open vocabulary models (Pham et al.) [2021] Jia et al. 2021 Radford et al. 2021 Yu al., 2022a), we report our results using CLIP (Radford et al., 2021) as this is the most popular publicly available open vocabulary model.
CLIP consists of text encoder and and image encoder (schematic on the left side of Figure Tp. In the standard setting, there are number of hand-written templates which can be completed with the relevant category names (e.g. "A photo of (}", "A photo of many ())). To classify the images in dataset, each of these templates is filled in with given category name. Then each of these sentences embedded via the text encoder, and all sentences completed with the same category name are averaged and normalized. This results in embeddings where is the number of categories in the dataset. Each of these n embeddings is the mean many different sentence embeddings. Then each image in the dataset is embedded using the image encoder. This embedding is compared to each of the n text embeddings using cosine similarity and is labeled with the most similar one.
CuPL requires only small adjustment from this standard practice. Instead of filling in the hand- written templates for each category, we simply replace these altogether with the sentences output by GPT-3. This means for CuPL, hand-written templates are only used input for the LLM, while the prompts for CLIP are entirely generated text. We present 2 different setting of CuPL (as shown in Table T each representing a different trade-off between accuracy and hand- engiinernn.....
1. CuPL (base). This setting uses three hand-written sentence across all 15 examined datasets. We do this by constructing general I.L ------rompt templates which are filled in with the category names for each dataset. Our three general templates are as follows:
Describe what a/the looks like: Describe a/the
What the identifying characteristics of a/the
3

---

Under review as a conference paper at ICLR 2023
The blank portion of this template is either filled in with the category type plus the category name (e.g. "pet" + +0 for the Oxford Pets dataset (Parkhi et al., (0122) or "aircraft" + { for FGVC Aireraft (Maji et al. 2013)) just the category name for more general datasets like ImageNet (Deng et al. 2009). Type specification is necessary because of words that have multiple meanings. For example "boxer" from the Oxford Pets dataset can also mean person who boxes, as opposed to dog breed, so it is necessary to specify "Describe a pet boxer:". Similarly, "Tornado" from the FGVC Aireraft dataset can be a type of aireraft or type of weather.
2. CuPL (full). In this setting we use different LLM-prompt templates for each dataset, just as Radford et al. (202T) uses different image-prompp templates for each dataset. However, we use fewer hund-written templates overall and also contain less specific information about each dataset in the templates. For this work, each dataset has between and 9 -rompps which generate between 20 and 90 image-prompt per category (10 generated sentences per L..M-prompt). For ImageNet, we use the following L.LM-proppi:: (1) "Describe what a(n) { looks like", (2) "How can you identify a(n) ()?", (3) "What does a(n) {} look like?", (4) "A caption of image of a(n) {J. (5) "Describe an image from the internet of a(n) {)"" Example generations for each these LLM-prompts given for two ImageNet categories in Figure 3 Full LLM- --pommppsss for all datasets
as well as example image-prompts are given in Sections and Koof the Appendix.
3 EXPERIMENTS AND RESULTS
We first discuss the details of our experimental setup. We next show improvements on a wide range of image classification benchmarks. We then examine the scaling behavior with respect to the model size and report observations regarding hyperparameters such as the LLM sampling temperature. Finally, we consider and compare with other methods of obtaining descriptive captions, and provide analysis of CuPl's improvements over the standard method.
3.1 SETUP
Unless specified otherwise, we use CLIP with a backbone of ViT-L/14 (Dosovitskiy et al.. 2020) and the GPT-3 DaVinii-002 model. Additionally, in order to perform open vocabulary image clas- sification, each image category needs a natural language label. This is sometimes provided by the dataset, but not always (e.g. ImageNet categories are described by an id number which can map to multiple synonyms). For this work, we use the same natural language labels specified Radford et al. (2021).
We report our findings on 15 zero-shot image recognition benchmarks: ImageNet (Deng et al. 2009). Describable Textures Dataset (DTD) (Cimpoi et al., 2014). Stanford Cars (Krause et al.
2013 Scene UNderstanding (SUN397) (Xiao et al. 2010). Food101 (Bossard et al. (014)). FGVC
Aireraft (Maji al. 2013), Oxford Pets (Parkhi et al. 2012). Caltech101 (Fei-Fei et al. 2004).
A moped typically has pecaas, like bicycle, but also has motor.
A moped has a small engine, usually under 50cc, and generally has a maximum speed of 30mph. A moped looks like a cross between bicycle and matorcycle.
The image is of a blue moped with a white seat.
A moped on a city street.
add-looking animal with duck-like bill, beaver-like tail, and otter-like body. The easiest way identify a platypus by its beak, which wide and flat like duck's.
The platypus
A platypus looks like beaver that has duck bill.
The image shows platypus swimming in body of water. This adorable creature is a platypus, unique mammal found in Australia.
Slide rules are rectangular, and have number of lines and markings of different lengths on them. Siide rules have linear scale on one edge of the rule and logarithmic scale on the other edge. A slide rule is a straight ruler with two scales that can slide past one another. A slide rule is a mechanical analog computer. This slide rule belongs to my grandfather.
Figure 3: Example image-promper for each of the L.M-prompts. For three ImageNet classes (moped, platypus, and slide rule), we give an example image-prompt for each of the 5 L...-prompie used CuPL (full) for ImageNet.
4

---

Under review as a conference paper at ICLR 2023
Table 1: Performance of CuPL prompts compared to the standard, hand-written prompts in CLIP (Radford et al. 2021) on 15 zero-shot image classification benchmarks. "Astd" stands for the difference; green shows improvement. In addition to accuracy, we show number of prompt templates ("# hw") that are hand-written for each dataset using each method, as well as the total and unique number of hand-written templates for each method (unique number only counts templates once even if used for multiple datasets). Note that CuPL (base) uses just three hand-constructed sentence across all datasets compared to 175 in the standard method.
ImageNet
Stanford Cars
FGVC Aireralt
Oxford Pets
Cattech101
Flowes 102
Kinsties- 700
RESISCAS
CIFAR-10 10
CIFAR 100
SUN297
Food101
UCF101
Birdsnap
Unique
DTD
Total
std
75.54 55,20
77.53
69.31
93.08
32.88
93.33
93.24
78.53
77.45
60.07
71.10
95.
95.59
78.26
50.43
73.43
hw
80
8
1
34
48
28
18
18
18
268
175
CuPL (base)
76.19
58.90
76.49
72.74
93.33
36.69
93.37
93.45
78.83
77.74
60.24
68.96
95
78.47
51.11
74.15
A std hhw
+0.65
+3.70
-1.04
+3.13
+0.25
+3.81
+0.04
+0,21
+0.30
+0.29
+0.17
-2.14
+0.22
+0.21
+0.63
CuPL (full)
76.69
61.70
77.63
73.31
93.36
36.11
93.81
93.45
79.67
78.36
60.63
71.69
95. 95.84
78.57
51.11
74.80
A std
+1.15
+0.28
+0.48
+1.14
+0.25
+0.51
+0.63
hw
5
9
3
2
2
59
45
Flowers 102 (Nllsback & Zisserman 2008 ). UCF101 (Soomro al. 2012), Kinetics-700 (Carreira
et al. 2019), Remote Sensing Image Scene Classification (RESISC45) CCheng et 2017), CIFAR- 10 (Krizhvssyy et al.) (2009), CIFAR- 100 (Krizhevsky et al.) 2009), and Birdsnap (Berg al. 2014). For the two video datasets, we extract the middle frame of the video, as is done in Radford et al. 2021.
3.2 RESULTS
Our results for the base prompts setting and the full prompts setting are in Table T We present our method's performance on 15 different image classification benchmarks, comparing both the classification accuracy and the number of hand-written sentence templates needed for each method. Note that for the standard method (Radford al. 2021), the hand-writter sentences refer to the while for CuPL the hand-written sentences refer to the LLM-prompts, with which generated.
image-prompp
1. CuPL (base). In this setting, we see performance gains in 13 out of the 15 examined datasets. Note this setting uses just three hand-constructed sentence across all datasets. This is in comparison to the nearly 175 unique image-promp templates that are hand-written across all of these datasets in the standard setting. Additionally, in the standard setting these hand-constructed prompts must be very specific to the dataset (c.g.. "a black and white photo of a { "a plastic ()."). In comparison, CuPL (base) requires only the category type of the overall dataset and still outperforms the hand- written, domain specified baseline in almost all cases. Thus, we present this base prompt setting as a simple standard that matches or exceeds prompt engineering open vocabulary models.
2. CuPL (full prompts). Here we see improvements on all examined datasets. This includes large (over percentage point) gains on ImageNet Top- DTD (texture classification), SUN397 (scene classification), FGVC Aircraft (fine-grained aircraft classification), and Flowers 102 (flower classification). While this setting requires more hand-written prompts than setting (1), it still requires significantly fewer than the baseline method (5 sentences versus 80 sentence for ImageNet), and does not include knowledge about the image domain. The full list of construuud sentences for CuPL (full prompts) and the baseline method (Radford et al. 202T) can be found in Section Af the Appendix.
3.3 ANALYSIS AND ABLATIONS
Model Size. In Figure 4 we show CuPL (full prompts) at different model scales. As there are two different zero-shot models in the CuPL. algorithm, show the effects of varying each model
5

---

Under review as a conference paper at ICLR 2023
76-
CuPL
CuPL
ImageNer Accuracy
74
standard
ImageNet Accuracy
76.4
standard
72-
76.2
70
76.0
68
75.8
66
75.6
75.4
10
20
40
80
0.02 Language Model Size (OpenAl API cost, log scale)
0.0004
0.002
Image Classification Model Size (GMACs, log scale)
Figure 4: Performance of CuPL models scale. (Left) ImageNet Top-1 accuracy for various scales of CLIP. prompts remain consistently better than standard prompts even we adjust CLIP model size (ViT-B/32, ViT-B/16, ViT-L/14), GPT-3 model set as Vanei-0002 (Right) ImageNet Top-I accuracy for various scales of GPT-3 (ada, babbage, curie, davinci-002). Larger models produce higher accuracy. CLIP model set as ViT-L/14.
individually, On the left hand side, vary the CLIP model used while holding the LLM constant. We see consistent gains across all model sizes. On the right hand side, we vary the size of the LLM. We plot the accuracy of the baseline well, which does not vary it does not utilize LLM. We find larger models lead to higher accuracy, though the 2nd and 3rd largest models perform similarly.
Number of Prompts. In Figure 6 we present ablations on the number of M.MM-rroppts and image-prompts for CuPL (full prompts). On the left side, we show ImageNet accuracy as
CuPL
ImageNet Accuracy
75.4
we increase the number of LLM-prompts. This
75.2
also corresponds to the number of sentences that have be hand-written. Notably, this methods outperforms the baseline even when using prompts generated from a single hand- written sentence. On the right hand side, we hold the number of LLM LLM-prompts constant at 5 gen-
75.0
74.6
74.4
74.
0.4
0.6
0.8
1.0
and adjust how many emage-prompe erate per -.ommpt We plot the accuracy given the total number of image-prompts (so 10 generated image-prompt per LLM-prompt cor- responds 50 total image-prompts). We see
GPT-3 temperature
Figure Effect of LLM temperature. More prompt diversity leads to higher performance.
that CuPL begins to outperform the baseline at just 25 image-prompts, well below the 80 image- prompts used in the baseline.
Diversity of Prompts. We also examine the impact of the diversity of imagc-prompts on ImageNet accuracy. We adjust this parameter by changing the temperature of the GPT-3 model. This value changes the likelihood of selecting lower probability tokens and makes sentences more diverse from each other. As demonstrated in Figure 5 more diverse prompts lead higher ImageNet accuracy. Note these comparisons are done with single LLM-prompt to save computational cost.
WordNet Definitions and Wikipedia Descriptions. We also consider two additional methods of obtaining descriptive sentences for each ImageNet category, other than using LLM. Firstly, compare CuPL (full) image-prompts with image-prompts generated using definitions each Im- ageNet category. Because each ImageNet category is derived from the WordNet database (Miller 1995), we can use the WordNet definition of each word.
We preprocess these definitions they of the form "A(n) O is ..."" as all WordNet defi- nitions contain the name of the word itself. We also add a period to the end of each definition, as find this increases performance. As shown in Table 2 ImageNet Top-1 accuracy with WordNet

---

Under review as a conference paper at ICLR 2023
77.00
78
CuPL
CuPL
76.75
standard
standard
ImageNet A Accuracy
ImageNet Accuracy
76.25
76
76.00
75.75
75
75.25
75.00
73
20
80
100
Number LLM-prompts
Number of Image-prompts
Figure 6: Ablation on number of LLM-prompts (left) and image-prompts (right). (Left) As number of hand-written L..M-rropps increases, does accuracy. 10 image-prompts are generated for each L..M-prompt. Note that CuPL. outperforms the baseline even with just one hand-written sentence. We add the prompts in greedy manner, at each step adding the 10 prompts which lead to the largest performance gain. (Right) We adjust the number of image-prompts generated by fixed number (5) of LLM-prompts. Even at 5 image-prompts per LLM-prompt (25 prompts total), outperform the baseline which uses 80
definition prompts is below that CuPL or standard prompts. In addition to lower accuracy, this method uses significantly more sand-constructed sentences as it requires 1000 unique hand-written definitions compared to 175 unique hand-written mnage-prompt templates for the standard method and 45 unique hand-written M.I.-prompt templates for CuPL. (full).
Secondly, we compare against prompts generated from Wikipedia articles corresponding each Im- ageNet category, as collected in Bujwid & Sullivan (202Ib). Note that the Wikipedia article does always exactly match the natural language name of the class used by Radford et al. (202T). Addition- ally, 80 categories map to more than one Wikipedia article (e.g. the category associated with the natural language word "patio" is mapped to the articles for "patio" and "terrace"). In this case, we select the first associated article. We preprocess these by removing the first line (the name of the article), and then extracting the first sentence, including the final period. We find both of these preprocessing steps lead to increase in accuracy. We also truncate this sentence to the maximum allowed input length of CLIP. For the 24 ImageNet categories that do not have an associated Wikipedia page, we use the name of the category as the image-prompts. As shown in Table 2 find Wikipedia be less effective than standard prompts, CuPL. prompts, or WordNet definitions.
Table 2: ImageNet Top-1 accuracy for different methods generating image- prompts.
Standard
CuPL.
WordNet
Wiki
75.54
76.69
73.44
68.20
Ensembling with Standard Prompts. We also consider using LLM generated prompts from CuPL (full) in addition to hand-written prompts. We do this by averaging together all the text embeddings of the CuPL prompts and hand-written prompts. As shown in Table we find that for some datasets, ensembling both types of prompts outperforms CuPL prompts on their own, while for others CuPL prompts perform better. For all datasets, this ensemble performs better than standard prompts alone. However, this ensembling method requires all the hand-written effort and domain knowledge of the standard approach.
Analysis of Accuracy Gains. In addition to total accuracy gains, we present the per class accuracy shift between the image-prompts used Radford al. (202I) and CuPL, shown in Figure 7 As demonstrated, the accuracy gains seen in CuPL are not distributed uniformly through the ImageNet classes, with some classes seeing <40 percentage point accuracy gains, and others seeing 440 percentage point accuracy losses when compared against class accuracy with standard prompts. In other words, while CuPL sees a higher accuracy overall when compared to the standard method, the images which correctly predicted by the standard prompts not subset of the images
7

---

Under review as a conference paper at ICLR 2023
Table Performance of the ensemble of CuPL (full) and the standard, in CLIP (Radford et al., 2021). This ensemble outperforms the standard hand-written prompts for all examined datasets (difference shown with A std), and outperforms CuPL (full) for 11 datasets (difference shown with A CuPL)
prompts
FGVC Aireralt
Stanford Cars
ImugeNNt
SUN297
Food
Oxford Pers
Calteh1111
Howers 102
UCF101
Kinetics-
RESISCAS
CIEAR
CIFAR- 10AAR-100
Birdsnap
DTD
monn
Ensemble
76.51
61.60
77.66
73.51
93.42
36.47
93.71
93.87
79.73
78.16
61.50
73.03
95.88
79.33
51.09
75.03
A std
+0.97
+6.40
+0.13
+4.00
+0.36
+3.59
+0.38
+0.63
+1.20
+0.71
+1,43
+1.93
+0.29
-1.07
+0.06
A CuPL
40.38
40.36
40.00
+0.42
+0.06
+0.20
40.77
+1.34
+0.34
which are correctly predicted by CuPL. In fact CuPL sees just over a 1 percentage point gain when compared to standard prompts, but differs in it's predictions from the standard method for 11.50% of predictions (with CuPL correct for 4.48% of these, standard correct for 3.32%, and neither correct for 3.70%).
Figure shows the classes with the 20 greatest accuracy gains and losses when comparing class accuracy with standard image-prompts (Radford et al., (02I) versus with CuPL protges Interestingly, for many of the classes which see a large accuracy gain, we see a corresponding class with a large accuracy loss that is either similar to the initial class or likely to co-oceur with it (e.g. agaric/mushroom, academic gown/graduation cap, military uniform/Pickellaube, desk/monitor).
4 RELATED WORK
4.1 NATURAL LANGUAGE DESCRIPTIONS FOR IMAGE CLASSIFICATION
Several prior works use text-based knowledge of image categories improve classification accu- racy, Elhoseiny et al. (017)) extract visual information from unstructured text descriptions collected collectee from the internet to recognize parts of object and classify them a zero-shot way. RRed et al. (016) and He & Peng (0017) use natural language descriptions of bird types to train a multimodal classification model. Huang et al. (202T) use hand-collected attribute tags to attend over relevant features in images. Paaz-rgaman et al. (2200) extract visual information from Wikipedia descrip- tions to enable zero-shot bird classification. Additional works (Shen et al. 2022, Bujwid & Sullivan, 202 show improvements on large datasets (e.g.. ImageNet) using external information from ex- ternal databases such as Imagenet wiki and Wordnet. While these works show the effectiveness of augmenting zero-shot models with descriptive text, all of these prior works rely external natural language databases for descriptions. This often limits the possible categories that can be classified and can require extensive preprocessing to extract visual descriptions from noisy natural language.
4.2 GENERATED TEXT FOR DOWNSTREAM TASKS
Recent work has utilized text generated from LLMs in number of ways. Santurkar al. (2022) use an LLM to paraphrase existing image captions to use as data augmentation for CLIP. Liu et al. 2022) use GPT-3 generate knowledge is then used to improve accuracy on common sense reasoning questions. Hu et al. (2022) use a LLM to add labels text to improve text classification accuracy. In Yu al. (2022b). the outputs of GPT-2 model are used to train an encoder on top of a vision model to generate multimodal image representations for variety of tasks. [Su et al. (222) utilize a language model to perform image captioning by iteritively generating candidate image captions with LLM and then using feedback from an open vocabulary model to align it to a given image. Similarly, Yang et al. (002)) use GPT-3 along with text descriptions of images for the Visual Question Answering (VQA) task. However, unlike CuPL these prior works are either purely language tasks (common sense reasoning, text classification) multimodal with some language component (image captioning, VQA). In our work, we demonstrate how LLM generated text can be used to improve purely visual image classification tasks across number of benchmarks.
topic when given number of demonstrations, which
8

---

Under review as a conference paper at ICLR 2023
Higher Acouracy with Standard prompts
Higher Accuracy with CuPL prompts
Baharan horned viper Pambroka Welsh Corgl Australionn ee vouted oeling
mushroom
Scothsh Deerhound
Pickebwubbe
gradusllon cao
Treeing Via Coomouud
gown player
promontory
swim
Austrolian millarr unifom
oclie
timaran
Sgarii
Groenenced
stsworm
nematode
scuba diver
la through arch bridge
Siberia Prssky
filled nocked Izard
montor
agartc
patas monkey
cance
laptop computer
class accuracy difference of CuPL Standard image-prompts
Figure 7: Distribution of ImageNet per class accuracy difference of CuPL image-prompts sus standard image-prompts. As shown, the accuracy gains of CuPL are not uniform across all classes. Rather, we see large gains for some classes, and losses for others. In addition, we list the classes which see the largest accuracy gains when switching to CuPL prompts (with "mushroom" having the largest gain), and the 20 classes with the largest accuracy losses (with "canoe" having the largest loss.)
4.3 PROMPT ENGINEERING
Previous efforts have explored methods for obtaining successful natural language prompts. For both open vocabulary image classification models as well as LLMs, the format of prompts is known to highly affect accuracy (Schick Schütze 2021 Radford al. 2021 Brown al. 2020 Gao al., 2020). This has led to large effort to find optimal prompt formats. Proposed methods include crowd-sourcing high performing prompts (Bach et al. 2022 as well as framing prompts to induce models to give explanations
well as answers (Wei et al. 2022 Kojima et al. 2022 Nye et al.,
2021) Additional works have proposed learning prompts via gradient based methods (Zhang et al,
2021 Qin & Eisner 2021: Li & Liang 2021 Lester et al.) 2021: Shin et al. 22020), retrieval from
database (Rubin et al. 2022), or reformatting/ephrrsingg existing prompts (Jiang et al. 2020 Rubin
al. 2022).
Most relevant to this work are a number of methods for designing optimal prompts for zero-shot image classification with open vocabulary models. These methods learn prompts formats which yield high accuracy for image classification using either supervised (Zhou et al. [022] Rao et al. 2022) unsupervised (Huang et al. 2022) methods. However, unlike these prior works this work requires no additional training or labeled data.
CONCLUSION
We demonstrate that leveraging knowledge from LLM can immediately improve zero-shot curacy a variety of image classification tasks, with much less hand-engineering efforts to craft natural language prompts. Furthermore, prompts can customized the desired categories, rather than general template that applies to all existing image categories. Finally, using prompts gener- ated by L.LMs lowers the barrier of prior knowledge about the dataset, which is often required when crafting prompt templates.
Querying an LLM for prompt construction is simple, straightforward and as our results suggested, immediately beneficial. The hypothesis that joint force of LLMs and open vocabulary models would improve zero-shot image classification is thoroughly tested in this work. We hope these findings serve useful tool towards understanding and improving zero-shot image classification, and more generally, the consolidation of model capacities and modalities through natural language.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY
We have number of measures to ensure the reproducibility of this work. First, in the supplementary material we include the code to generate image-prompts for ImageNet and evaluate the aceuracy of these prompts. In Section 2 we note all hyperparameters used for the LLM. Additionally, in the appendix we include LLM-prompts used to generate image-prompts for each the 15 datasets. In the supplementary material, include all generated image-prompts for all dataset, for both CuPL (base) and CuPL (full).
REFERENCES
Stephen H. Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M SAIFUL BARI, Thibault Févry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Mike Tian-Jian Jiang, and Alexander M. Rush. Promptsource: integrated development environment and repository for natural language prompts. ArXiv, abs/2202.01279, 2022.
Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L Alexander, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 201 1-2018. 2014.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov. Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint (xXiv:2006.07/52 2020.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative com- ponents with random forests. European Conference on Computer Vision, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sebastian Bujwid and Josephine Sullivan. Large-scale zero-shot image classification from rich and diverse textual descriptions. ArXiv, abs/2103.0966)) 2021a.
Sebastian Bujwid and Josephine Sullivan. Large-scale zero-shot image classification from rich and diverse textual descriptions. arXiv preprint arXiv:2 2021
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv: 1907.06987, 2019.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, 105(10):1865-185) 2017.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pottern Recognition (CVPR), 2014.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint MrXiv:2010.11922 2020.
Mohamed Elhoseiny. Yizhe Zhu, Han Zhang, and A. Elgammal. Link the head the "beak": Zero shot learning from noisy text description at part precision. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6288-6297, 2017.
10

---

Under review as a conference paper at ICLR 2023
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Tianyu Gao, Adam Fisch, and Dangi Chen. Making pre-trained language models better few-shot learners, arXiv preprint arXiv:2012.15723, 2020.
Xiangteng He and Yuxin Peng. Fine-grained image classification via combining vision and lan- guage. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7332- 7340, 2017.
Shengding Hu, Ning Ding, Huadong Wang. Zhiyuan Liu, Juan-Zi Zi Li, and Maosong Sun. Knowl- edgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification. In ACI, 2022.
Siteng Huang, Min Zhang, Yachen Kang, and Donglin Wang. Attributes-guide and pure-visual attention alignment for few-shot recognition. In AAAI, 2021.
Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language mod- els. arXiv preprint arXiv:2204. 03649, 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Zhengbao Jiang, Frank F. Xu, J. Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. ArXiv, abs/2205. 1011/2055119166,1 1916, 2022.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13), Sydncy, Australia, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. ArXiv, his/2104408691. 2021.
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I1th International Joint Conference on Natural Language Processing (Volume I: Long Papers), hbs/2101.001990 2021.
Jaachng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannanch Hajishirzi. Generated knowledge prompting for commonsense reasoning. In ACL, 2022.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aireraft. Technical report, 2013.
George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pp. 722-729. IEFF, 2008
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Au- gustus Odena. Show your work: Scratchpads for intermediate computation with language models. ArXiv, abs/21 12.001 14, 2021.
11

---

Under review as a conference paper at ICLR 2023
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505 IEEE, 2012.
Tzuf Paz-Argaman, Yuval Atzmon, Gal Chechik, and Reut Tsarfaty. Zest: Zeoo learning from text descriptions using textual similarity and visual summarization. FINDINGS, 2020.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for open-vocabulary image classification. arXiv preprin! arXiv:2111.100500 2021.
Guanghui Qin and Jas' Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. ArXiv, abs/2104.06599 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. $778-8766. PMLR, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18082-18091, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International Conference on Machine Learning, pp. 5389-5400. PMLR, 2019.
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Learning deep representations of fine-grained visual descriptions. 2016 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 49-58, 2016.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. NAACL, 2022.
Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. Is a cap- tion worth thousand images? controlled study for representation learning. arXiv preprint arXiv:2207.07635, 2022.
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In EACL, 2021.
Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang, Pengchuan Zhang, Anna Rohrbach, Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, Kurt Keutzer, Trevor Darrell, and Jian- feng Gao. K-lite: Learning transferable visual models with external knowledge. ArXiv, abs/2204.09222. 2022.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Elic- iting knowledge from language models using automatically generated prompts. ArXiv, abs/20110119988 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 1rXiv:1212.0400 2012.
Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu. Dani Yogatama, Yan Wang, Lingpeng Kong, and Nigel Collier. Language models can see: Plugging visual controls in text generation. ArXiv, Abs/2205.02653. 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Ben Wang and Aran Komatsuzaki. Gpt-j-6b: billion parameter autoregressive language model, 2021.
12

---

Under review as a conference paper at ICLR 2023
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In Advances in Neural Information Processing Sys- tems, pp. 10506-10518, 2019.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201. 11903, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.0377 2019.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7959-7971, 2022.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang. Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge- bbaeedeeee-eedddddd vqa. In AAAI, 2022.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917. 2022a.
Youngjae Yu, Jiwan Chung, Heeseung Yun, Jack Hessel, Jae Sung Park, Ximing Lu, Prithviraj Am- manabrolu, Rowan Zellers, Ronan Le Bras, Gunhee Kim, and Yejin Choi. Multimodal knowledge alignment with reinforcement learning. ArXiv, aws/2205.1166... 2022b.
Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable prompt makes pre-trained language models better few-shot learners. ArXiv, abs/2108. 13161, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, pp. 1-12, 2022.
13

---

Under review as a conference paper at ICLR 2023
APPENDIX: What does a platypus look like?
Generating customized prompts for zero-shot image classification
OVERVIEW
CuPL (Full Prompts) vs Standard Prompts
B CuPL Base Prompts
C
Evaluation Metric
D
Open-Source LLM
Single Sentence Baseline
Robustness
G
CuPL Improvement Analysis
Error Analysis
Image-Prompt Distribution
Tempurature Analysis
Example Generated image-prompts
A CuPL (FULL PROMPTS) VS STANDARD PROMPTS
We detail the hand-written prompt templates used for CuPL (full prompts) versus standard CLIP Radford et al. (2021) prompt templates. For CuPL, hand-written prompt templates needed for the MLr-pppppss, while for the standard method hand-written prompt templates are needed for the image-prompts.
Note that many of the hand-written templates for the standard method encode information about the datasets. For example, tay ()" demonstrates knowledge that objects are sometimes represented as a toy version of an object rather than as the literal object. CuPl. prompts remain much more general (e.g. "Describe what () looks like"").
Caltech101
CuPL hand-ward written
Standard hand-written
Describe what a(n) O looks like: Deseribe a(n) {):
photo of a +-
a photo of the {
painting of (}-
painting of the {
What are the identifying characteristics of a(n) ()?
plastic ()-
the plastic ().
sculpture a 0-
a sculpture of ()-
sketch of a (--
a sketch of the O-
tattoo of a +-
a tattoo of the {
toy ().
the toy {)- a rendition of the (}- the embroidered ().
rendition a ().
embroidered ().
cartoon {-
the cartoon (}-
O in video game.
the { in video game.
plushie ()-
the plushie O)
origami ()-
the origami 0-
art of ()-
art of the ()-
graffiti of a {
graffiti of the {
drawing of (}.
drawing of the {--
doodle of ().
a doodle of the {}.
14

---

Under review as a conference paper at ICLR 2023
Food101
CuPL hand-written
Standard hand-written
Describe what (} looks like
photo of (), type of food.
Visually describe {}
How can you tell that the food in this photo is {)?
Stanford Cars
CuPL hand-written
Standard hand-written
How can you identify a(n) {
photo of a {}-
Description of a(n) (). a type of car.
a photo of the 0-
A caption of a photo of a(n) ():
photo of my 0-
What are the primary characteristies of a(n) (
i love my O!
Description of the exterior of a(n) (}
photo of my dirty 0.
What are the identifying characteristics of a(n) {. a type of car? Describe an image from the internet of a(n) {}
a photo of my clean 0-
photo of my new ()-
What does a(n) {} look like?
a photo of my old ()-
Describe what a(n) () type of car, looks like
Oxford Pets
CuPL hand-written
Standard hand-written
Describe what pet {} looks like
a photo of a, type of pet.
Visually describe a(n) '0", a type of pet.
ImageNet
CuPL hand-written
Standard hand-written
Descrihe what ain) () looks
had photo ()-
the origami ()-
How can identify a(n) ()?
photo of many {)-
the {} in a video game.
What does a(n) look like?
a sculpture of a O-
a sketch a ()-
Deseribe an image from the internet ain)
photo of the hard to see O- a low resolution photo of the {
doodle of the ()- a origami {}-
A caption of an image of atu) ():
rendering ol'al ft-
low resolution photo (}-
graffiti of O-
the toy O-
bad photo cropped photo of the O). a tattoo of a ()-
O-
a rendition the O-
photo of the clean {))
a photo of large ()
the embroidered {).
rendition of
a photo of a hard to see (}-
a photo of a nice {
bright photo photo of clean O-
()-
photo a blurry photo of a ()-
()-
photo of a dirty O-
a cartoon ()-
dark photo the ().
art of ()-
a drawing of a {-
a sketch the {]-
photo of my the plastic 1
embroidered (-
a piselated photo of
photo of cool ()-
itap the ()
close-up photo of a O)
jpeg comupted photo of the ()
a black and white photo of the ().
a good photo of a {}-
painting of the {
plustic {)-
a painting a (--
a photo of the nice ()-
pixelated photo
{)-
photo of small ()-
a sculpture of the ()-
a photo of the weird ()-
hright photo
()-
the cartoon ()-
cropped photo of O..
art of the O- a drawing of the ()-
a plastic ().
photo of the dirty {}-
photo of the lurge ()-
a jpeg commpted photo of O-
a black and white photo of ()-
blurry photo of {)-
the plushic {
a photo of the {}-
a dark photo of ()-
good photo the (}-
itap O-
readering of the U-
grafttti of the (}. a toy ()-
a { in video game. photo of one (}-
itap {)-
a doodle of a ()-
a photo of cool O-
up photo the {)-
photo small (-.
a photo of ()
a tatton of the ()-
15

---

Under review as a conference paper at ICLR 2023
FGVC Aircraft
CuPL hand-written
Standard hand-written
Describe a(n) O aireraft
photo of O. type of aireraft. photo of the {}. a type of aircraft.
Describe the {) aircraft
DTD
CuPL hand-written
Standard hand-written
What does "("" material look like?
photo of texture.
What does a "(" surface look like?
photo of () pattern.
What does "("" texture look like?
photo of thing.
What does a "O"" object look like?
photo a {} object.
What does "[ 1m thing look like?
photo of the { texture.
What does a " "{)" pattern look like?
photo of the pattern.
photo of the thing.
photo of the O object.
SUN397
CuPL hand-written
Standard hand-written
Describe what a(n) {} looks like How can you identify a(n) ()? Describe photo of a(n) {}
a photo of a 0) a photo of the ()-
Kinetics-700
CuPL. hand-written
Standard hand-written
Describe the action "("" What does a person { look like?
photo of a person ()-
a photo of a person using 0-
What does the act of () look like? Describe "(}"
photo of a person doing (
a photo of a person during ().
a photo of a person performing (-
a photo of a person practicing U a video of (). a video of person 0
a video of a person using O-
a video of a person doing ()-
a video of a person during O-
video of person performing (
a video of a person practicing {
example of ().
a example of person f).
example of person using 0-
a example of person doing 0-
a example of person during (-
a example of person performing {-.
a example of person practicing ().
a demonstration of ()-
a demonstration of a person f)-
a demonstration of a person using {
a demonstration of a person doing ().
a demonstration of a person during ().
a demonstration of a person performing {..
demonstration of a person practicing ()-
16

---

Under review as a conference paper at ICLR 2023
UCF 101
CuPL hand-written
Standard hand-written
Describe the action {
a photo of a person {
What does the act of {} look like?
video of person {).
What does person doing {} look like?
a example of person f}-
Describe "[)"
a demonstration of a person f}.
Describe the action "("
a photo of the person ()-
video of the person ()-
a example of the person {
a demonstration of the person 0.
a photo of a person using ()-
a video of a person using 0-
a example of person using (}-
a demonstration of a person using U
a photo of the person using ()-
a video of the person using (}.
example of the person using (}-
a demonstration of the person using 0-
photo of a person doing ().
a video of a person doing {
a example of person doing (.
a demonstration of a person doing O.
a photo of the person doing f
a video of the person doing {
a example of the person doing 0-
a demonstration of the person doing 0-
a photo of a person during (..
video of person during (
a example person during (.
demonstration of a person during ()-
a photo of the person during {
video of the person during ()
a example of the person during 0-
a demonstration of the person during (
a photo of a person performing O-
a video of person performing ().
a example of person performing ()-
a demonstration of a person performing f}.
a photo of the person performing ()-
a video of the person performing (..
example of the person performing 0-.
a demonstration of the person performing 0.
photo of a person practicing ().
a video of a person practicing ()-
a example of person practicing O.
a demonstration of a person practicing {..
a photo of the person practicing ().
a video of the person practicing ().
a example of the person practicing ().
a demonstration of the person practicing O)
17

---

Under review as a conference paper at ICLR 2023
RESISC 45
CuPL hand-written
Standard hand-written
Describe satellite photo of a(n) f}
satellite imagery of (}.
Describe a(n) {} as it would appear in aerial image
aerial imagery of {
How can you identify a(n) { in an aerial photo?
satellite photo of {
Describe the satellite photo of a(n) 0
acrial photo of {):
Describe an acrial photo of a(n) {
satellite view of (}-
aerial view of 0-
satellite imagery of a 0
acrial imagery of {}.
satellite photo of a {
aerial photo of {).
satellite view of a {}.
acrial view of a 0
satellite imagery of the 0- aerial imagery the ()-
satellite photo of the {
acrial photo of the (
satellite view of the ()-
aerial view of the {}-
Birdsnap
CuPL hand-written
Standard hand-written
Describe what the bird ( looks like: Describe the bird {):
photo of a (). type of bird.
What are the identifying characteristics of the bird ()2
Flowers 102
CuPL hand-written
Standard hand-written
Describe how to identify a(n) O. type of flower What does a(n) ( flower look like?
photo of (). type of flower.'
CIFAR-10
CuPL hand-written
Standard hand-written
Describe what a(n) { looks like Describe a(n) ():
photo of O.
blurry photo of a O)-
What are the identifying characteristics of a(n) (
black and white photo of 0--
low contrast photo of a ().
high contrast photo of a {).
bad photo of a {}.
good photo of {}.
photo small {}-
photo of big ()-
a photo of the {
blurry photo of the ()-
black and white photo of the O.
low contrast photo of the (..
high contrast photo of the ().
bad photo of the ()
good photo of the {).
photo of the small ().
photo of the big O-
18

---

Under review as a conference paper at ICLR 2023
CIFAR-100
CuPL hand-written
Standard hand-written
Describe photo of a(n) ():
photo of f
What are the identifying characteristics of a(n) {}? Describe what a(n) ( looks like:
blurry photo of a {}.
black and white photo of O-
Describe a(n) ():
low contrast photo of a {
high contrast photo of a {
bad photo of {
good photo of a {}-
photo of small {}.
photo of big {
photo of the {).
a blurry photo of the ()-
black and white photo of the 0
low contrast photo of the {
high contrast photo of the (}.
bad photo of the {}.
good photo of the (..
photo of the small (
photo the big (}-
B CUPL BASE PROMPTS
The three general sentences used in the base prompt setting are:
Describe what a/the looks like: Describe a/the
What are the identifying characteristics of a/the _
Here we specify the type filled for each of the examined datasets, as well as the article used for that dataset ("a(n)' 'the");
Dataset
Base LLM-prompt type specification
ImageNet
a(n) {
DTD
the texture f
StanfordCars
the car 0
SUN397
a(n) O
Food 101
the food 0
FGVC Aircraft
the aircraft {
Oxford Pets
pet O
10
a(n) f
CIFAR-10 10
a(n)
CIFAR-100 100
a(n)
Flowers 102
the flower 0
Kinetics-700
the action of {
UCF101
the action of {
RESISC45
satellite photo of {
Birdsnap
the bird O
19

---

Under review as a conference paper at ICLR 2023
C
EVALUATION METRIC
FGVC Aircraft
Stanford Cars
ImageNet
SUN397
Food101
Oxford Pets
Caltech 101
Flowers 102
UCF101
Kinetics- 700
RESISC45
CIFAR -0
CIFAR- 100
Birdsnap
DTD
Acc.
Acc.
Acc.
Acc.
Acc.
Mean
Mean
Mean
Mean per
Acc.
Mcan (topl,
Acc.
Acc.
Acc.
Acc.
per
per class
per
class
class
class
top5)
D OPEN-SOURCE LLM
While GPT-3 (Brown et al., 2020) demonstrates higher performance on number of tasks compared to smaller open-source models, open-source mod- els are sometime more accessible. We therefore show improvement using GPT-J-6B (Wang & Ko- matsuzaai 202T], a small open-source model avail- able on HuggingFace (Wolf et al. 2019). We find that we are able to surpass human written prompts with prompts generated by this model as shown in Table, though we still fall short of those generated by GPT-3. Additionally, we employ number of strategies to increase the accuracy of the lower quality GPT-J-6B generations. First, we generate lower temperature (0.3) prevent irrelevant nonsensical generations, which find occur more frequently in smaller models. Additionally, we generate times more Image-prompts per LLM-prompt than we do when using GPT-3 (Brown et al., 2020). We also add punctuation the end of all ALM-prompte to encourage the LLM to begin new sentences. Finally, we filter out any image-prompts which do not contain the name of the ImageNet category they are meant to describe, as well as remove number of unicode characters from the generations (e.g. 'u2019'). We present these findings as a way to make CuPL a more accessible options until large high performance models become available to the public.
Table 4: CuPL with open-source model. CuPL able to improve over hridtwritten baselines even for smaller open-source mod- els.
ImageNet 75.54
standard
CuPL (GPT-J-6B)
75.62
CuPL (GPT-3)
76.69
E SINGLE SENTENCE BASELINE
One of the primary benefits of CuPL is that it de- creases the amount of necessary band-engineering. However, this could also be done by decreasing the number of total hand-written templates used, which comes at loss in performance. a 'low effort' baseline, where We present this as only the hand constructed template of 'a photo of O We com- pare this with two CuPL baselines. The first is the baseline in which we also only construct one hand written template: 'Describe what a ( looks like". We then use this to generate 10 Image-prompe prompts. The second baseline is single CuPL generated sentence, generated with the prompt 'Describe what a { looks like'. For this experiment, we generate at a temperature of 0.3 as we find that higher tempuratures only helpful when we able to ensemble many diverse prompts, not when are limited to hand-written written template under both of these settings, as one. We find that CuPL outperforms a single hand-wre
Table 5: Single sentence baselines. Com- parison single hand-written Image- prompt template with single handle written LLM-prompt as well as single CuPL gen- erated Image-pronpt.
ImageNet
photo of {
73.46
CuPL (I hand-written)
75.71
CuPL generated)
74.24
shown in Table
20

---

Under review as a conference paper at ICLR 2023
F ROBUSTNESS
In addition to the previously mentioned benefits of open vocabulary models, one of the important advances made by CLIP (Radford al. 202T) is an increased robustness on out-of-distribution data. Fine-tuning has been shown to degrade performance on out di-tiitributtion tasks (Wortsman et al., 2022), however zero-shot CLIP is robust to these distribution shifts. We show improvement on two common distribution shifts in Table demonstrating that CuPL. maintains the robustness of CLIP.
Table 6: Robustness of CuPL. CuPL accuracy two common ImageNet variants, using CuPL ImageNet image-prompts. CuPL improves performance on both of these variants, demonstrating that CuPL. improves accuracy in-distribution tasks, while maintaining robustness distribution shifts.
ImageNet
ImageNet- V2
ImageNet- Sketch
Deng et al. 2009)
Recch et al. 2019)
(Wang et al. 2019
Standard CuPL
75.54
69.86
59.60
76.69
70.85
60.05
G CUPL IMPROVEMENT ANALYSIS
G.1 VISUAL SIMILARITY ANALYSIS
In Figure 7 we provide initial analysis on the categories where the CuPL algorithm improves the most over standard hand-written prompts. We find that the improvement is not uniformly distributed, but rather some classes see a large improvement, while others see a decrease in per class accuracy. Interestingly, there are often two similar categories where one sees a large increase in accuracy and the other sees a decrease. For example, the 'mushroom' class has an approximately 40pp increase, while 'agaric' (a subclass of mushroom) is one of the classes with the largest drop in accuracy.
In order to better understand this phenomenon, we examine the change in accuracy between CuPI. and the standard method of prompting in the image embedding space. Thus we are able to visualize the close relationship between categories like 'agaric' and 'mushroom'. In order to be able to visu- alize the high dimensional CLIP image embedding in two dimensions, we utilize the 1-distributed stochastic neighbor embedding algorithm (Van der Maaten & Hinton 2008). Figure g visualizes image features (reduced into two dimensions) in relation to CuPL improvement.
As suggested by Figure 7 see in Figure & that when there is class that has large increase in accuracy with CuPL. prompts ("mushroom', 'graduation cap'. 'monitor') there is often a decrease in class accuracy for visually related class. This means that when choosing between two similar or co-occuring classes, CuPL has a different distribution of classification than the standard method (e.g. the standard method prefers 'canoe' over 'paddle' much more strongly than CuPL). This suggests that the overall accuracy improvement of CuPL over the standard method may come (at least in part) from better distinguishing between two visually similar classes. While it may be over-corres cerr-corretiin from the mistakes of the standard method (as demonstrated by the drop in accuracy in one of the two similar classes), the CuPL predictions appear to be overall more accurate, as demonstrated by the overall higher accuracy.
G.2 CO-OCCURRENCE, BETWEEN OBJECTS
Many of the frequently confused pairs in Figure are objects that are likely to occur in an image (i.e. "canoe"- "paddle' or ggraduationn -acad'ac gown' or monitorr'''esse
One potential benefit CuPL
of CuPL captions is that they are able to capture co-occurrences as well. For example, prompt for the "canoe' class is canoe is typically a narrow boat with pointed ends that is propelled with a paddle. This caption contains the word 'paddle' which is frequently confused with 'canoe' and likely to be present in images, even where the correct label is "canoe'. We therefore investigate the effectiveness of CuPL captions images which contain more than one ImageNet object.
We attain this by using the ImageNet- ReaL dataset (Beyer et al. 2020) which relabels ImageNet images with all applicable labels, so an image with both "canoe" and 'paddle" would have both
21

---

Under review as a conference paper at ICLR 2023
Image Embedding vs CuPL improvement
paddle
Net CuPL accuracy Baseline aoouracy
30
02
pickelhaube
miltary uniform
ecuba diver snorkel
graduation eap
pecddmmic gown
Ageric
mushroom
monitor
desk
-90
Figure 8: Visualizing of image embedding of ImageNet classes compared to CuPL improve- ment on that class. Each point on this figure represents the average image embedding of an Im- ageNet class, which has been reduced to two dimentions using t-sne (Van der Maaten & Hinton 2008). We see that when there is class with large improvement compared the baseline, is of- ten visually similar to class which has decrease in accuracy. This suggests that CuPL's improved accuracy may be due in part to an increased ability to distinguish similar classes compared to the baseline.
labels. We then tag images as having multiple ImageNet objects or only one ImageNet object based on the Real. dataset. Finally, we compute ImageNet accuracy across each of these two sets (using standard ImageNet labels), Results are given in Table 7
Table 7: Standard versus CuPL accuracy based number of ImageNet classes present in image. We use the ImageNet-Reall dataset (Beyer et al.) 2020) to find images which have more than one applicable ImageNet label. We then present the accuracy for standard prompts and CuPL prompts using standard ImageNet labels, split by images which contain only one possible ImageNet class and images which may contain multiple classes.
One class present (85.1% of ims)
Multiple classes present (14.9% of ims)
Standard CuPL
79.89
51.59
80.79
53.58
H ERROR ANALYSIS
In Figure we present an eтrоr analysis of our model using two different metrics. The first is an analysis between the model prediction and the correct class using the visual similarity of these labels. To capture this, we first attain an average visual embedding of each class by taking the mean of each image in that class and then normalizing that mean. Then for each class we rank how similar each of the other 999 classes are by the distance between these embeddings. If the models makes an incorrect prediction, but it predicts the class with the closest embedding to the correct label, then we refer to this an image offset of 1.
Additionally, we examine the prediction errors in terms of the linguistic similarity of the labels. We do this with the WordNet (Miller 1995) similarity of two labels. For example, if the label of the prediction and the ground-truth label share the same parent in the WordNet tree, that is a WordNer offset of 2.
While slight, there a difference in the errors made by CuPL compared to the emrors made by the baseline shown in Figure g CuPL. is more likely have error that has image offset of
22

---

Under review as a conference paper at ICLR 2023
Eroor Distance by Image Offeet
Error Distance WordNet Offset
Baseline CUPL
Baseline
CuPL
2157
Number of Model Enore
210
240
1121
11
Image Offsett from Target to Prediction
WordNet Offsctt from Target to Prediction
Figure 9: Error analysis CuPL. and baseline comparing models errors visually and linguis- tically to ground truth labels. We compare the errors made by CuPL to the errors made by the baseline methods. We find that the errors made by CuPL are more likely be the most visually similar class the ground truth label when compared to the errors made by the baseline. However, errors made by baseline are more likely to be the most linguistically similar class according to the WordNet (Miller (1955) heirarchy. This suggests that visual information is being extracted from the CuPL descriptions to make visually consistent predictions.
than the baseline. However, the baseline is more likely to have an error that has a WordNet offset of 1 than CuPL. This implies that CuPL. may be taking advantage of the visually descriptive language of the captions, as even when the model makes errors, they tend to favor categories that are visually similar the ground truth. However, the baseline method does have visual descriptions in its Image-prompts which may lead to its errors aligning more linguistically with the ground-truth.
TSNE of embeckding Image-prampls generated different LLM-prompts
Platypus
Lacybug
Tonch
Visualization of embeddings of Image-prompts generated with various LLM- prompts. Each point represents the image embedding of one Image-prompts for the stated category with has been reduced two dimensions using t-sne (Van der Maaten & Hinton 2008). Image- prompts with the same color are generated by the same LL.M-proppt..
Figure 10:
I IMAGE-PROMPI DISTRIBUTION
When generating Image-prompts, we use an ensemble of prompts generated by different LLM-
prompts (e.g. 'What does {) look like?). As we find that the diversity of Image-prompts is correlated with accuracy, it valuable to understand how different LLLM-promppt affect the diversity of proggep To accomplish this, visualize the text embedding of magee-poopps for selection of classes using t-sne (Van der Maaten Hinton 2008) dimension reduction. We then color Image-prompts that were generated by the same L...--rompt. As shown in Figure T0
23

---

Under review as a conference paper at ICLR 2023
find that while there is slight clustering of Image-prompts by LLM-prompts, there is also large amount of overlap.
J TEMPERATURE ANALYSIS
We provide several further analyses of the effect of temperature prompt generation. First, we visualize the distribution of Image-prompts that have been generated with variety of different tem- peratures. We do this by selecting prompts for 3 different ImageNet classes at different tempera- tures. We then perform dimentionality reduction on the text embeddings of these prompts in order to visualize their distribution. As shown in Figure [T] Image-promplt generated with temperature of 0.1 are clustered in few different locations. Image-prompts generated with temperature of 0.5 are more widely, and progppp generated with temperature of 0.9 have a similar, but even wider distribution.
we give all generated prompts for the ImageNet class 'Tench' at three different temperatures. the lowest temperature, the generated image-prompts are nearly identical when generated with the same L.M--proppts. As the temperature increases, so does the difference the generated prompts.
Additionally in Section K
ISNE of text embeddiing of tmage prompts at various temperatures
Platypus
Lady
Tench
Figure 1I: Visualization of embeddings of Image-prompts generated with various tempera- tures. Each point represents the image embedding of one Image-prompts for the stated category with has been reduced to two dimensions using t-sne (Van der Maaten & Hinton 2008). Prompts generated with a higher temperature cover wider distribution.
K EXAMPLE GENERATED IMAGE-PROMPTS
for subset of ImageNet categories. We give all 50 for the first ImageNet category of "Tench" and then 10 randomly selected prompts for number of randomly selected ImageNet categories.
selection L---enneraeed
K.1 ALL. GENERATED IMAGE-PROMPTS FOR "TENCH" CATEGORY
Temperature = 0.99
carp family.", "A tench is a freshwater fish that is typically brown or olive in color.",
"A tench is a freshwater fish
grow up to 2 feet in length.", "A tench is freshwater fish of the family Cyprinidae.",
"A tench is fresh water fish that
"A tench is a freshwater fish of the carp family.",
"A tench is a freshwater fish with a dark green back and light-colored
sides.".
"Tench are freshwater fish found in Europe.", tench is small Freshwater fish in the carp family.",
24

---

tench is freshwater fish that is typically grewnish-brown color Under review as a conference paper at ICLR 2023
tench is heavyset freshwater fish with mottled brown body and
small, flat head.",
"A tench is a freshwater fish that looks similar to a carp.",
"A tench is a freshwater fish in the carp family.",
"A tench is freshwater fish of the Cyprinidae family.",
"The tench is freshwater fish of the Cyprinidae family.",
"The tench is a fresh-water fish the family Cyprinidae.
"The easiest way to identify a tench is by its herringbone scales.",
tench is freshwater fish the carp family.",
"Tench are freshwater fish found in Europe.",
"Tench have a large, slimy body with scales that have a green hue.", "The tench is a freshwater fish belenging to the carp family.", "A tench is freshwater fish of the Cynoglossidae family.", tench is freshwater fish in the carp family.", "Tensch are freshwater fish with Olive Green backs, shading to Yellowish on the sidcs.",
"A tench locks like a green freshwater fish with brownish hue.", "A tench locks like freshwater fish with a dark olive-green back, fading
the sides.", orange-yel underbelly.",
bron "A tench usually has olive-green skin with dark spots, and
"Tench are freshwater fish that can grow up to 70cm long! They have olive-brown skin with dark spots, and their meat is white and firm. ", "A tench is a freshwater fish with a sturdy body and greenish- coloration.",
"A tench is a freshwater fish that can grow to about two feet long.", "A tench is freshwater fish in the carp family.", tench is large, freshwater fish with thick body and large head.", "The image is a tench fish swimming in water.",
"The image is of tench fish swimming in pond.",
"The tench is a freshwater fish native to Europe
"This image shows large, dark green tench swimming in pond.", a tench from the internet would likely show dark green fish with lighter underside.",
"An image
"The image is of a tench fish.",
"The image is of a tench fish on white background. tench is freshwater fish of the Cyprinidae family.", "The image is a tench swimming in murky pond.", "In the image, tench swims in a pond with lily pads.", A tench in a river.",
"A tench (Tinca tinca) is freshwater fish in the carp family that is found throughout Europe.",
Tench (Tinca tinca), à member of the carp family (Cyprinidac), native to Eurasia.",
A tench, freshwater fish in the family Cyprinidae.",, The tench (Tinca tinca)
freshwater fish of the cyprinid family found throughout Eurasia.",
A tench in a Finnish lake.",
"A tench (Tinca tinca) is treshwater fish belonging to the carp family (Cyprinidae).
tench in fishpond.", The common tench is freshwater fish of the cyprinid family found throughout Eurasis.", "Tench (Tinca tincs) in a pend."
Temperature 0.5
with a brassy sheen.",
"A tench is a freshwater fish that typically has dark green back, 1ight brown sices, and a white belly.",
"A tench is freshwater fish that can grow up to two feet long.",
"rench freshwater fish found in Europe.",
25

---

Under review as a conference paper at ICLR 2023
tench is freshwater fish that is typically olive green in color with dark spots.",
"A tench is a freshwater fish that is typically olive green in color with brownish tint.", ",
"A tench is a freshwater fish of the Cyprinidae family.", "Tench
freshwater fish found in Europe.",
"A tench is a freshwater fish that has a dark green back, light brown sides, and a white belly.",
"A tench is a freshwater fish that is typically olive-green in color with brownish dorsal fin.",
tench is a freshwater fish of the cyprinid family.",
"A tench is a freshwater fish of the carp family.", "A tench is a freshwater fish that is typically olive green in color with brownish back.",
"The tench is freshwater fish of the carp family Cyprinidae.",
"A tench is a freshwater fish that is typically greenish- in color.",
"A tench is a freshwater fish of the carp family.",
"A tench is freshwater fish of the carp family.", "Tench have olive green backs and flanks, with yellowish bellies.", "A tench is a freshwater fish of the carp family.",
"A tench is a freshwater fish of the cyprinid family.",
"A tench is a freshwater fish that can grow to 30 inches long.", tench is freshwater fish that is typically greenish-brom in color.",
"A tench is a freshwater fish that can grow up to about two feet long.", "A tench is a freshwater fish with a brownish-green back and sides, and yellowish-brown belly.",
"A tench is freshwater fish that can grow up to two feet long.",
"A tench is a freshwater fish that looks similar to a carp.", "A tench is a freshwater fish that is typically greenish-brow in color.",
"A tench is freshwater fish that is part of the carp family.",
"A tench is a freshwater fish of the cyprinid family."
"A tench is a freshwater fish that can grow up to two feet. long.",
"The image is of a tench fish swimming in water.",
"The image is of tench fish swimming in pond."
"The image
tench fish swimming in pond.".
"The image is a tench fish swimming in pond.".
"The image is of a tench fish swimming in a pond.",
"In the image, tench is swimming in pond with lily pads.",
"The image is of a tench fish swimming in pond.".
"The image
a tench fish swimming in pond.".
"The image is of a tench fish swimming in a pond.".
"The image is of a tench fish swimming in a pond.",
"A tench fish, native to Europe, characterized by its greenish-brown color and spets.",
tench (Tinca tinea) in pend.", "A tench (Tinca tinca) is a freshwater fish belonging to the carp family (Cyprinicae)... .",
"A tench (Tinca tinea) is freshwater fish in the carp family.", tench (Tinca tinca)
freshwater fish in the carp family
(Cyprinidae).""
A tench in a river.", "A tench (Tinca tinca) is freshwater fish in the carp family (Cyprinidae).."
tench (Tinca tinca) in pond.", A tench (Tinca tinca) in a garden pond.",
A tench in a river."
Temperature 0.1
"A tench is freshwater fish that is typically olive green in color with dark spots.",
26

---

Under review as a conference paper at ICLR 2023
"A tench is freshwater fish that can grow up to 30 inches long. ",
"A tench is freshwater fish that can grow a length over two feet.",
"A tench is a freshwator fish that can grow to two feet long.",
"A tench is a freshwater fish that can grow to two feet long.", tench is freshwater fish that is typically olive green in color with
dark spots.",
"A tench is a freshwater fish that can grow up to two feet long.",
"A tench is a freshwater fish that can grow to two feet long.",
tench is freshwater fish that can grow up to two feet long.", tench is freshwater fish that is typically olive green color with dark spots.",
"A tench is a freshwater fish of the carp family.". "A tench is freshwater fish of the carp family. tench is freshwater fish of the carp family."
"A tench is a freshwater fish of the carp family.
"A tench is a freshwater fish of the carp family.".
"A tench is a freshwator fish of the carp family.".
"Tench have dark green back, light olive sides, and yellowish belly.",
"A tench is a freshwater fish of the carp family.".
"A tench is a freshwater fish of the carp family.",
"A tench is a freshwater fish of the carp family.",
"A tench is freshwater fish that can grow up to two feet long."
"A tench is a freshwater fish that can grow up to two feet long.
"A tench is a freshwater fish that can grow up to two feet long."
"A tench is a freshwater fish that can grow to two feet long.".
"A tench is freshwater fish that is typically olive green in color with brownish dorsal side.",
"A tench is a freshwater fish that can grow up to two feet long.",
"A tench is a freshwater fish that can grow up to two feet long.",
"A tench is a freshwator fish that is typically olive green in color with brownish dorsal side.",
"A tench is a freshwater fish that can grow up to two feet long.",
"A tench is a freshwater fish that can grow up to two feet. long.",
"The image is of a tench fish swimming in pond."
"The image is of tench fish swimming in pond.
"The image
tench fish swimming
pond."
"The image is a tench fish swimming in pond.
"The image is of a tench fish swimming in a pond.
"The image is of tench fish swimming in pond.
"The image is of tench fish swimming in pond.
"The image
a tench fish swimming in pond.
"The image is of a tench fish swimming in a pond."
"The image is of a tench fish swimming in a pond.", "A tench (Tinca tinca) is freshwater fish in the carp family.", tench (Tinca tinca) is freshwater fish of the carp family (Cyprinicae).".
"A tench (Tinca tinca) is a freshwater fish in the carp family.". "A tench (Tinca tincal is freshwater fish in the carp family."
"A tench (Tinca tinca) is freshwater fish in the carp family.",
tench (Tinca tinca) is freshwater fish the carp family."
"A tench (Tinca tinca) is freshwater fish in the carp family.",
A tench (Tinca tinca) in a garden pond.",
A tench (Tinca tinca) in garden pond.",
"A tench (Tinca tinca) is freshwater fish of the carp family (Cyprinidac).
K.2 SAMPLE GENERATED IMAGE-PROMPTS FOR RANDOMLY SELECTED IMAGENET CATEGORIES
"bubble":
"A bubble sustained period of inflated asset prices.",
27

---

Under review as a conference paper at ICLR 2023
"A bubble looks like sphere of air.",
"A bubble often characterized by rapidly increasing prices in an asset
or security, followed by a sharp decrease in prices.",
"A bubble looks like a small, round, thin film of soap filled with air.",
"A bubble locks like round sphere of soap film.",
\"A bubble being blown in the park.",
"A close-up of soap bubble with thin film of water in between two layers of air.",
"A bubble is a spherical shape made up of thin film of soap water.", bubble is circle air surrounded by water.",
bubble looks like round, thin layer soap surrounding pocket of air."
"kit fox":
"A kit fex is small species fex, about the size of domestic cat.", "You can identify kit fox by its small size, its big ears, and its
long, bushy tail.",
A kit fex laying on the ground in a desort habitat.",
"A kit fox is small fox found in North America.",
kit fex is small fox with sleek coat of fur.",
"A kit fox is a small fox with large ears, a long, black-tipped tail, and pale fur.",
"A kit fex has reddish coat, with white patches en its chest and throat.",
"This kit fex has reddish coat and large ears.", "A kit fox looks like small fox with a pointed nose, large ears, and long, bushy tail.",
"A kit fox is small species of fex."
"toy terrier":
"A toy terrier looks like very small version of a terrier.", "The image is of a toy terrier that is mostly white with brown spots.", "You can identify toy terrier by looking for compact, short-legged dog with a short muzzle.",
Cute little guy.",
"Toy terriers are miniature versicns of terriers, such as the Jack Russel1 Terrier.", toy terrier a small, brown toy terrier.", "A toy terrier typically has a leng, narrow head with pointy ears, and a small, compact body.",
small, lightweight. breed dog.",
"The image
"A toy terrier is small, short-legged dog with long body, pointy nose, and large ears.",
"A small, brown and white toy terrier is sitting on a beige couch, looking at the camera."
"mousetrap":
"In the image, there is a mousetrap made of wood and metal.",
"An image of a nousetrap from the internet would most likely show a traditional wooden mausetrap with metal spring.",
"A mousetrap is a device made to catch and kil1 mice.", mousetrap is small device that is used catch mice.",
"The classic mousetrap consists of a wooden base with a metal spring mounted on one end.",
"The mousetrap is small wooden bex with metal spring inside.",
"The mousetrap is simple device that has been used for centuries to catch micc.",
"The classic mousetrap simple, effective, and deadly.", "The most common way to identify mousetrap is by its small size and rectangular shape.",
mousetrap typically has wire wooden frame that is baited with food and springs open quickly te snap shut on the mouse when it attempts to steal the bait."
"dog slea":
28

---

Under review as a conference paper at ICLR 2023
"A dog sled is vehicle runners, typically with thin frame and
flat bottom, that is used to cenvey goods or passengers over snow or ico.",
"A dog sled is toboggan pulled by dogs, typically over snow.", ",
"A dog sled looks like a bed runners that is pulled by dogs.",
"A dog sled is typically heavy frame on runners that is pulled by one or more degs.",
"A dog sled is traditionally a sled pulled by dogs, used for transportation, racing, or ether purposes.",
teamå aog sled team is group dogs that are narnessed together to pull a sled.",
"A dog sled looks like a small, open vehicle that is pulled by one or more dogs.",
"Dog slecding in winter.",
pulled by huskiesThe image is of dog sled pulled by huskies.",
"A dog sled looks like a large cart that pulled by a team of dogs."
"geyser";
"A geyser is hot spring that periedically crupts, shooting colunn of water and steam into the air.",
"A geyser locks like a hole in the ground that somet ies spurts hot water and steam into the air.",
"Geysers are hot springs that periodically spout water and steam into the air.",
"The image of a geyser erupting.",
"A geyser is a hot spring that periodically erupts, spraying water into the air.",
"A geyser typically looks like cone of rocks with small hole at the top.",
"A geyser a hot spring where water intermittently boils, sending jet of hot water and steam into the air.", ,
"A geyser is a hot spring that periedically shoots a strean. of hot water and steam into the air.",
"The image
a geyser shooting water high into the air.",
"A geyser locks like a column of water that shoots into the air and then falls back down."
:Schipperke":
"A Schipperke a small, Belgian breed dog.",
"A Schipperke is a small black Belgian dog with a rat-like tail.", "The image is of black and white dog with pointy ears and long body.",
"A schipperke a small, black, Belgian breed of dog.", "A schipperke is a small, black, Belgian breed of dog that closely resembles a fox.",
"A Schipperke is small Belgian breed of dog that resembles fox.", and pointed muzzle.", Schipperke a small dog breed with a fox-like appearance.", "It's a photo of a black and tan Schipperke dog standing in front of a brick wall.",
"Schipperkes have long, black
"Black, small, spitz-type dog with long, fox-like snout, large erect ears, and long, high-set tail."
"go-kart":
"A go-kart typically looks like a small car buggy with small engine in the back.",
go-kart is small vehicle with four wheels, steering wheel, and a gas pedal.",
"A go-kart is a small vehicle with a stecring wheel, pedals, and an engine.",
"Two young girls in go-karts race down a path in park.",
"A go-kart is a small, lightweight vehicle with four wheels and simple, open frame.",
"A go-kart is small, open-wheeled vehicle used for racing.",
"Two kias racing go-karts dirt track.",
29

---

Under review as a conference paper at ICLR 2023
go-kart is small, racing car.",
"A go-kart typically looks like a small, open-wheeled car.", "A go-kart is a small, lightweight vehicle with four wheels that is propelled by a small engine."
"black-and-whis colobus":
"The aldck-and-White Colobus a type of Old World monkey, found in Africa.",
"A black-and-white coloous has Long black fur, and a white face with black triangle around the eyes.", "Colobus Monkey in the Trees'his elegant colobus monkey swinging through the trees search of feed.", monkeyIn this image, a black-and-white colobus monkey is shown perched atop a tree branch.",
"The black-and-white colobus monkey is of the beautiful and distinctive of all the colobus menkeys." " "The black-and-white colobus monkey is à species of primate in the Colobidac family.",
"Colobus monkeys generally black with white patches their face, back, and sides.",
"The black-and-white colobus is a species of 0ld World monkey.", monkeyThe image is of a black-and-white colobus monkey sitting on tree branch.",
monkeyIn the image, the alack-and-white colobus monkey is sitting in tree."
"sock":
"A sock usually has cuff at the tep, and heel at the bottom.",
A black seck with white line running down the middle.",
"A sock is typically a garment wern on the feet and made from a soft material, such as cotton.",
"A sock normally has a heel, toe and cuff at the top.", sock is small amount of money that is given semcone without them knowing.",
"A sock is an article of clothing wern on the feet.",
"A sock is piece of clothing that is worn on the feet.", "A sock is tubular garment that covers the foot and ankle.", "This image is of blue and white striped sock.",
"There are many ways that you can identify a sock."
"Cocker Spaniel":
"A Cocker Spaniel is medium sized dog with long, floppy ears, and silky coat that is usually either brown or black.", "The Cocker Spaniel has a long floppy ears, a silky coat, and a bushy tail.",
"A Cocker Spaniel has long, black muzzle and big, brown eyes.", "The Cocker Spaniel is breed of deg.",
a Cocker Spaniel from the internet shows a small brown and white dog with long floppy ears.",
"An image
"The image is of a Cocker Spaniel with short, brown fur and long, tooppy ears.",
"Cocker spaniels have long, floppy ears and long, silky coat.", "A Cocker Spaniel is a small to medium sized dog.", "The image is of a light brown and white Cocker Spaniel standing on green grassy field with its head turned to the side.",
"A Cocker Spaniel has long, silky coat that is usually either black, brown, or gelden."
"southern black vidow":
"A southern black widow spider perched atop web.". ",
"Female southern black widows have black body with red hourglass shape on their abdomen.", "The southern black widow is black with a red hourglass shape on its belly.",
30

---

Under review as a conference paper at ICLR 2023
"Female southern black widow spiders black with characteristic red hourglass-shaped mark on their ventral abdomen.",
spiderThe image is of a large, black spider with a red hourglass shape on its abdemen." ",
"A southern black widow is spider that is black with red hourglass shape its abdomen.", "A southern black widow spider is a small, black spider with a red hourglass-
mark on its underside.",
"There's an image the internet of southern black widew that's really cool.",
southern black widow can identified by its black coloration with red hourglass shape on its abdomen.",
"A Southern black widow is a type of spider that is black with a red hourglass shape on its belly."
"catamaran":
"A catamaran is a sailboat that has two hulls, or wide bodics, that are connected by beams.",
"The casiest way to identiff catamaran is by its two hulls.",
catamaran is type of boat that has parallel hulls.", "A catamaran is a multi-hulled vessel with two parallel hulls of equal
size.",
"My dream boat! A sleek catamaran that can zip through the waves.",
"A catamaran is multi-hulled vessel with two parallel hulls of equal size.",
"The image is of a white catamaran with blue trim.",
"A catamaran is a type of boat that has two hulls, or platforms, that are parallel to each other.",
"The image is water.",
yellow catamaran with white trim, sitting in caln
"A catamaran is a type of sailing vessel that consists of two parallel hulls of equal size."
"beach":
".",
"Blue skies, white sands, and clear turquoise se waters make this beach a paradise.",
beach usually has sand and water.",
"Beach icentification can be accomplished through the identification of physical characteristics.""
"A beach is a naturally occurring feature of the landscape.", "The sun sets over the ocean, casting beautiful orange hue in the sky.",
"A beach is a large body of water with sand or small rocks at the shore.",
"In the image, the beach is Brilliant white with crystal blue waters.",
"The beach lecks like long strip of land next the ocean.", "A beach typically looks like large, flat expanse sand with sone rocks or other natural features nearby."
"rotary dial telephone":
rotary dial is device used to dial telephone numbers.",
"Rotary dial telephone from the mid-20th century.",
"A rotary dial telephone is a phene with a circular dial on the front face.",
rotary telephone, you the Flace the phone's s receiver on your ear and listen for a dial tone.",
"When looking
tell it is rotary phone by
"A rotary dial phone is an older model phone that has a circular device with numbers on it that you rotate with your finger to dial number." ",
"A rotary dial telephone is a type of telephone that uses a mechanical wishes call.",
dial to select the telephone number that
"History of the Rotary Dial Telephone.",
"A rotary dial telephone is an old-fashioned telephone that has round dial on the front of it.",
31

---

Under review as a conference paper at ICLR 2023
rotary dial telephone looks like classic telephone with rotary dial on the base unit.", "The image is of a rotary dial telephone on a black background."
32


Reviews:
[{"review_id": "zPvlPe6IhIC", "reviewer": "Reviewer_DGAE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to generate prompts for zero-shot learning inference with CLIP by querying the large language model like GPT. The results show that the class prompts generated from GPT improves the zero-shot classification performance across many datasets. ", "summary_of_the_review": "Due to the weakness mentioned above, I think this paper does not have sufficient contributions for ICLR. I would recommend a reject and suggest the authors submit it to a workshop.", "strength_and_weaknesses": "Strength\n\n+It is interesting to generate prompts from LLM. Querying the description of an object class from LLM makes a lot sense because LLM is learned from large text corpus, covering knowledge about most of vocabularies. \n\nWeakness\n\n-I found this paper has little technical novelty. The proposed method generates prompts for each class from the outputs of a LLM i.e., GPT-3 and then use those prompts for zero-shot inference with the pretrained CLIP, which is straightforward and expected to achieve some improvement. \n\n-The author's main contribution is querying GPT but the design decision of why a particular prompt is chosen to query GPT3 for CuPL base and full are not discussed or ablated.\n\n-To my knowledge, using GPT-3 is not entirely free, which makes the method less appealing. Does this approach work well with other free LLM?\n\n-In Fig. 5, increasing the temperature of LLM also increases the randomness of the outputs. What is the standard deviation at each temperature?\n\n-The performance improvement are very minor and comes with a decrease in performance on other similar classes as shown in figure 7. This insight indicates that such a simple query setup is suboptimal for finegrained setting. The authors have not tried to mitigate this basic drawback. \n\n-Detailed analysis on what makes the GPT3 prompts work better than human designed ones is missing when it is the main contribution of the work. For example in figure 5 we see that GPT3 prompts only surpass generic hand designed prompts at ~0.56 temperature. Why is this so? How do the outputs of the language model change with this temperature setting and what contributes to this change. If the method is this sensitive to temperature value, it adds to additional labelling cost of running the GPT3 model to ablate over the temperature value. How sensitive is this temperature value on different datasets? This needs to be analysed and discussed in light of labelling cost of generic human prompts.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "uaFxvqjQ7N", "reviewer": "Reviewer_ECmW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents combines CLIP with large language models like GPT3 to create customized prompts for zero-shot image classification. Specifically authors leverage the knowledge contained in LLMs in order to generate many descriptive sentences that are customized for each object category for prompting CLIP models. Experiments on 15 downstream classification datasets show the effectiveness of the proposed method over handcrafted prompts that were used in the original CLIP adaptation.", "summary_of_the_review": "I’d like to rate the current submission as a clear rejection due to very limited technical novelty and lack of convincing experiments. Despite all the changes, I still feel the paper will not have enough contributions to be accepted for ICLR.", "strength_and_weaknesses": "**Strengths**: \n\n- The paper is well-written and easy to follow.\n- The idea is very simple and performs reasonably well on standard classification datasets.\n\n**Weaknesses**:\n\n- The technical novelty of the paper is very limited. It simply combines GPT3 with CLIP without a clear motivation. It is not surprising that output of a large language model will have better descriptions/knowledge compared to vanilla prompt engineering like \"a photo of a\". So, what is the main contribution of the paper need to be clearly discussed in the paper. Why someone will adopt this approach given that it often requires higher computational resources for inferencing a GPT3 model (and sometime not accessible for all)?\n\n- Although the proposed framework uses a strong LLM like GPT3, the zero-shot classification performance on downstream datasets are very minimal. Training a continuous prompt, e.g., CoOp in Learning to Prompt for Vision-Language Models using 1-2 labeled samples performs significantly better than hand engineered prompts. It is often very practical to have 1-2 labeled samples per class inn many applications.  It is not clear why learning a single prompt using few labeled samples is not practical but the use of GPT3 can be considered practical for zero-shot classification? A thorough comparison and discussion with other prompt learning methods should be included in the paper.\n\n- How does the proposed framework with only one prompt compare to the baseline that uses prompt like \"a photo of a\" (no ensemble of prompts)? This is a very important experiment to verify the effectiveness of the proposed framework in fair manner.\n\n- What other generative LLMs besides GPT3 can be used and how are they comparable to the current one?\n\n- The experiments only show performance on CLIP, while there exists many other recent vision-langauge models, e.g., DeCLIP, FILIP, CLOOB, CyCLIP etc. Authors should perform experiments on a variety of V&L models to demonstrate the generalizability of the proposed method for generating better prompts.\n\n- Like original CLIP, CuPL still requires handcrafted prompts for LLM which is suboptimal. Identifying the right hand-crafted prompt is a non-trivial task, which often requires significant amount of time and domain-specific heuristics.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aoENYxohYyU", "reviewer": "Reviewer_LiMR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to replace hand-written prompts used in CLIP style models with GPT-generated prompts for zero-shot image classification. The authors show that their approach (CuPL) of using LLMs to generate diverse text prompts, can improve classification accuracy while removing the need for hand-crafted prompts.", "summary_of_the_review": "Overall, I like the paper. The paper presents a simple contribution and shows its effectiveness through thorough empirical evaluation. However, some analysis is still missing from the paper to build a better understanding of why the proposed approach is working. After reading the paper, apart from empirical evidence, there is little else to see the benefits of using the proposed approach. I think providing some of the analysis mentioned in my review will make the paper stronger and more insightful.\n\nUpdate after rebuttal: I thank the authors to provide answers to my questions and incorporating all feedback! I think the paper presents a simple to understand/implement/use contribution that is very effective! I quite like the paper and I am bumping my rating to 8 to reflect that. ", "strength_and_weaknesses": "Strengths\n1. The approach is simple and effective. They show that by using the knowledge baked into LLMs like GPT can be used to improve zero-shot classification accuracy. I think this contribution is neat! \n\n2. While some analysis is still missing in the paper (See weaknesses), I think the authors did a thorough empirical evaluation of their approach. I appreciated the author's effort on showing how different model sizes, number of LLM prompts, number of image prompts, LLM temperature affect performance. Per class accuracy difference (Figure 7) was also quite informative to look at! \n\nWeaknesses: \n1. Higher accuracy yes, but it doesn't remove the need for hand-designed prompts. As the authors mention in Figure 7, CuPL leads to significant accuracy boosts for certain classes (like mushrooms), but doesn't do well in other classes (canoe). This kinda means that hand-designed prompts can't be completely replaced because they bake in domain knowledge about the dataset. \n\n2. I am curious how the GPT prompts are helping improve classification accuracy. It would be nice to sort of look at which words in the image prompt (maybe by looking at attention distribution) are important. Doing this both for standard as well as LLM based image prompts might help understand why there is such a drastic jump ( +/- ~40%) in accuracy for different classes.  \n    - I found the 40% accuracy jump for `mushroom' quite surprising. My first intuition was that the proposed approach will work better for classes that have distinct parts (cars, scooters, chair, etc). I wonder why LLM based prompt helped for a category like a mushroom?\n\n3. Diversity Analysis (Figure 5) done on single LLM-prompt. It's quite possible that different LLM prompts, produces the same Image prompt. So an analysis of how different all the image prompts generated from different LLM prompts for a single image are would be nice. \n\n4. In figure 6, \n    - how did they choose which prompt to select when using fewer prompts. For instance, when only using 1 LLM prompt, which prompt did they use? \n    - I also didn't understand the caption for Figure 6 (left). The caption says that CuPL outperforms the baseline with just 3 hand-written sentences. But in the figure, CuPL is outperforming the baseline even with 1 LLM prompt. Can the authors please clarify? \n\n5. May I suggest adding a baseline that only uses 'a photo of a {}' prompt? The proposed approach lies somewhere in the spectrum of no-effort (using one standard prompt for everything), and high-effort (using standard hand-written prompts). We see where it lies with respect to high-effort. It'd be great to also see how the proposed approach compares to a _no-effort_ baseline.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7kwQ74Als1c", "reviewer": "Reviewer_UY2k", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Open vocabulary models require natural language prompts as an intermediate medium between category and visual content, thus it is important to choose a good prompt. In this paper, the authors study how to use a pretrained language model to help generate better language prompts for open vocabulary tasks. The authors feed the categories with language prompt templates to an LLM and use the output of the LLM as the image prompts. The authors show that the prompts generated by their method can achieve better performance on zero-shot image classification benchmarks. ", "summary_of_the_review": "The authors propose a novel method that is simple and has the potential to apply to many different tasks. However, the authors did not show that this method is widely applicable which makes the paper weaker. My decision is borderline accept.", "strength_and_weaknesses": "Strength:\n- The paper is well written. The method is intuitive and the performance gain is good.\n- The authors did different ablation studies to show how different design choices would affect performance including the number of prompts, diversity of prompts, model size, etc. These ablations are helpful for the audience to understand the full picture of this method.\n\nWeaknesses:\n- Although the general idea is good, the authors only consider one task in this paper which is zero-shot image classification. It is more interesting for tasks like open vocabulary segmentation/detection etc. This method can even be used for image caption retrieval where LLM is used to paraphrase the caption. It is more convincing that the proposed method can universally work on different tasks.\nEven for imagenet, it is more interesting to see the performance on the full imagenet21k instead of the 1k set. In addition, original CLIP is evaluated on different relatives of Imagenet like imagenet-v2, imagenet-R etc. which would also be good to have in this paper.\n- It would be better if there can be a clear analysis or explanation of why the standard prompts and CuPL prompts have different preferences as shown in figure 7. It is a really interesting phenomenon to me, and it may also reveal some hints on how to improve CuPL.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject