Visual Classification via Description from Large Language Models
Authors: Sachit Menon
Carl Vondrick
Venue: ICLR 2023 notable top 5% (2023)
Keywords: vision-language models
CLIP
prompting
GPT-3
large language models
zero-shot recognition
multimodal

Abstract: Vision-language models such as CLIP have shown promising performance on a variety of recognition tasks using the standard zero-shot classification procedure -- computing similarity between the query image and the embedded words for each category. By only using the category name, they neglect to make use of the rich context of additional information that language affords. The procedure gives no intermediate understanding of why a category is chosen, and furthermore provides no mechanism for adjusting the criteria used towards this decision. We present an alternative framework for classification with VLMs, which we call classification by description. We ask VLMs to check for descriptive features rather than broad categories: to find a tiger, look for its stripes; its claws; and more. By basing decisions on these descriptors, we can provide additional cues that encourage using the features we want to be used. In the process, we can get a clear idea of what the model ``thinks" it is seeing to make its decision; it gains some level of inherent explainability. We query large language models (e.g., GPT-3) for these descriptors to obtain them in a scalable way. Extensive experiments show our framework has numerous advantages past interpretability. We show improvements in accuracy on ImageNet across distribution shifts; demonstrate the ability to adapt VLMs to recognize concepts unseen during training; and illustrate how descriptors can be edited to effectively mitigate bias compared to the baseline. 

Published as a conference paper at ICLR 2023
VISUAL CLASSIFICATION VIA DESCRIPTION FROM LARGE LANGUAGE MODELS
Sachit Menon, Carl Vondrick Department of Computer Science Columbia University
ABSTRACT
Vision-language models (VLMs) such CLIP have shown promising perfor- mance on a variety of recognition tasks using the standard zero-shot classifica- tion procedure computing similarity between the query image and the embed- ded words for each category. By only using the category name, they neglect to make use of the rich context of additional information that language affords. The procedure gives no intermediate understanding of why category is chosen, and furthermore provides no mechanism for adjusting the criteria used towards this de- cision. We present an alternative framework for classification with VLMs, which we call classification by description. We ask VLMs check for descriptive fea- tures rather than broad categories: to find a tiger, look for its stripes; its claws; and more. By basing decisions on these descriptors, we can provide additional cues that encourage using the features want to be used. In the process, clear idea of what features the model uses to construct its decision; gains some level of inherent explainability. We query large language models (e.g., GPT-3) for these descriptors to obtain them in a scalable way. Extensive experiments show framework has numerous advantages past interpretability. We show improve- ments in accuracy on ImageNet across distribution shifts; demonstrate the ability adapt VLMs to recognize concepts unseen during training; and illustrate how descriptors can be edited to effectively mitigate bias compared to the baseline.
get
INTRODUCTION
Why does person recognize a hen in D If you had to justify your answer, you might name its beak, describe its feathers, or discuss any number other traits that we associate with hens. easy for people describe the visual features of categories in words, well use these verbal descriptions to aid perception. However, generating such schemata, let alone leveraging them for perceptual tasks, has remained key challenge in machine learning.
Vision-language models (VLMs) trained on large corpora of paired image-text data, such as CLIP (Radford al., 2021), have seen huge successes recently, dominating image classification. The standard zero-shot classification procedure- - computing similarity between the query image and the embedded words for each category, then choosing the highest has shown impressive performance on many popular benchmarks, such as ImageNet (Russakovsky et al. 2015). Comparing to the word
Our top prediction: Hen and say that backuse. Average
CLIP's top prediction: Dalmatian but we diort say that beceuse.
Avarago
black Ever-colore
nar
body
laga
small head!
short atiff hair
two wings
tail
long, slender muzzle
long tapering
beak
Figure 1: On the left, we show an example decision by our model in addition to its justification (blue bars). On the right, we show how CLIP classifies this image. Our model does not make the same mistake because it cannot produce a compatible justification with the image (red bars).

---

Published as a conference paper at ICLR 2023
that names category reasonable place to start because these methods rely on the fact that the word "hen" tends to show up near images of hens on the Internet.
Despite the advances classification performance, the large models often make unreasonable mis- takes or give undesired answers (Goh et al. 2021). The standard zero-shot method gives us no intermediate understanding (i.e. explanation) of the model's reasoning process. They often fail to look at cues that a human would use easily, and there is no clear way to get the right cues or provide them to the model.
Our key insight is that we can use language as an internal representation for visual recognition, which creates an interpretable bottleneck for computer vision tasks. Instead of querying VLM with just category name. the use of language enables us to flexibly compare to any words. If we have an idea what features should be used, we can ask the VLM check for those features instead of just the class name. To find a hen, look for its beak; its feathers; and more. By basing the decision on these features, we can provide additional cues that encourage looking at the features we want to be used. In the process, we can get clear idea of what the model uses make its decision; it is inherently explainable.
However, hand-writing these features can be costly, and does not scale to large numbers of classes. We can solve this by requesting help from another model. Large language models (L.LMs), such as GPT-3 (Brown al., 2020), show remarkable world knowledge on variety of topics. They can be thought of as implicit knowledge bases, noisily condensing the collective knowledge in way that be easily queried with natural language (Petroni et al. 2019). As people often write about what things look like, this includes knowledge of visual descriptors. We thus can simply ask an LLM, much like 5-year old asking their parent: what does it look like?
the Internet
We provide an alternative to the current zero-shot classification paradigm with vision-language mod- els, comparing to class descriptors obtained from large language model instead of just the class directly. This requires no additional training, and does not require substantial computational over- head during inference. By construction, this provides some level of inherent interpretability; we can know an image was labeled tiger because the model saw its stripes rather than its tail. Rather than compromising performance metrics, our approach improves accuracy across datasets and distribu- tion shifts, achieving a ~ 4-5% increase on top-1 ImageNet accuracy.
METHOD
2.1 PERFORMING CLASSIFICATION WITH DESCRIPTORS
Given image x, goal is to classify whether visual category is present in the image, where we represent a category c through a textual phrase, e.g., "school bus." To make our model both
"beach"
tiger"
beach "srrnnee
tiger"
Taand"
"syrrnge"
(a)
(b)
model compares image embeddings (white dot) to word embeddings of the category name (colorful dots) in order to perform classification. (b) We instead mine large language models to automatically build descriptors, and perform recognition by comparing to the category descriptors.
Figure 2: (a) The standard
2

---

Published as a conference paper at ICLR 2023
School bus
Barber shop
Vialin
* regge yeinow vehicce
Tws bu building with la hege open atcoefrent
in anriingdd instrument
words school wrtheen side
burther paie sign currase the strp
typsioally tree tour enrings
sop sign deploys from
bus
charrs
shop
wepden
top
Shoe store
-eo large scicetion shoes in the window
soow displey recks the stooe
building with sign that says 'shoo store"
rogister
ehrreeeuurg elourger
ourger patty
cach rogidter
ting arew for cur-cueee customens
Pirato
I-holes
coundiport
bow
Figure Examples of descriptor schema produced by GPT-3.
with
Volcano
tomato
camroona the diekk
Has large cone shaped mountain
omionn
& weodenn hull
connrr the hop of mountain
portholes
Boeinn from the erater
kotchhup
figiing
rieing
interpretable and editable, we estimate score for category e through the additive decomposition:
s(c,x)
Σ
o(d,z)
(I)
D(c)
dED(c)
where D(c) is the set of descriptors for the category c and o(d, x) is the log probability that descrip- tor d pertains to the image Our approach will represent the descriptors d also through natural language sentence; we explain how to obtain these the next section.
This model s(c,x)) will output high score when the dictionary for the category D(c) contains many descriptors that highly match the observed image œ. Figure 2 illustrates this approach to classification. We use addition so that some descriptors can be missing the image, and normalize by the number of descriptors for the class to allow different classes to have different numbers of descriptors. Since the descriptors is naturally interpretable. To understand why the model predicts category one can simply read which descriptors have high score.
both additive and expressed in natural language, the model
2.2 BUILDING DESCRIPTORS
For the classifier to work, we need to successfully estimate the descriptors D(c) of each visual category. We propose to automatically construct this set by prompting a large language model, such as GPT-3, describe the visual features that distinguish that object category in photograph. We prompt the language model with the input:
Q: What are useful features for distinguishing a (category name) in photo?
A: There are several useful visual features to tell there is (category name) in a photo:
where { (category name) is substituted for a given c. The generated list then comprises the dictio- nary D(c). Further implementation details can be found in Appendix A
Fig. 3 shows several example descriptor schemata that emerge from generative language pre- training. The descriptors often cover colors, shapes, object parts, counts, and relationships, but can be anything in natural language. While descriptors are closely related to more traditional "attributes." this flexibility distinguishes them, enabling each category's 's descriptors we observe in Fig. 3, they can be -ategory-specific, such as the "stop sign" for the "school bus" category, or more general, such as "cash register" for both "shoe store" and "barber shop."
be rich and nuanced. As
While language models do not have images in their training set, they learn to imitate visual descrip- tion successfully without visual input. The corpora used to train language models contain descrip- tions written by people with visual knowledge. These descriptions, aggregated at scale, provide strong basis for visual recognition.
2.3 GROUNDING DESCRIPTORS
We use vision-language models visually ground the natural language descriptors generated by the large language models, i.c. CLIP similarity to form ø. Since descriptors are often relative to their class, condition descriptors on the class name. For example, "long tail" for mouse
3

---

Published as a conference paper at ICLR 2023
ImageNNet
ImageNet
CUB
EuroSAT
Architecture for @
Ours
CLIP
Ours
CLIP
A
Ours
CLIP
A
Ours
CLIP
ViT-B/32
62.97
58,46
4.51
55.52
51.90
3.62
52.57
51.95
0.62
48.94
43,84
5.10
VIT-B/16
68.03
64.05
3,98
61.54
57.88
3.66
57.75
56.35
1.40
48.82
43,36
5.46
VIT-L/14
75.00
71.58
3.42
69.3
65.33
3.97
63.46
63.08
0.38
48.66
41.48
7.18
ViT-I/144 336px
76.16
72.97
3.19
70.32
66.58
3.74
65.257
63.41
1.847
48.74
44,80
3.94
Places365
Food101
Oxford Pets
Describable Textures
VIT-B/32
39,90
37.37
2.52
83.63
79.31
4.32
83.46
79.94
3.52
44,26
41.38
2.87
VIT-B/16
40.34
38.27
2.07
88,50
85.61
2.90
86.92
81.88
5.04
45.59
43.72
1.86
ViT-L/14
40.55
39.00
1.55
92.44
91.79
0.65
92.23
88.25
3.98
54.36
51.33
3.03
VIT-L/14@3366 336px
41.18
39.58
1.59
93.26
92.23
1.03
91.69
88.20
3.49
54.95
52.39
2.55
Table 1: Accuracy gains over CLIP category name embedding baseline. We see a consistent ~ 3-5%
improvement across model sizes for ImageNet and ImageNetV2, as well as up to 7% on other datasets from dramatically different domains.
will be still shorter than a "short tail" for an elephant. We estimate similarity with text of the form settegrr_nnaaee which (is hws/sacs (descriptor). These text embeddings are similar to class prototypes (Snell al. 2017 Rudin et al. (021), but instead they are obtained across modalities, as discussed further in the Related Work. If an image belongs to the class, but does not show a particular descriptor, that descriptor is activated less. We show Section 3.2 that we can nonetheless recognize new categories from descriptors that do not need such reference.
2.4 CLASSIFICATION AND EXPLANATION
Our model is able to discriminate between categories by selecting the one with the highest score:
arg max s(c, x)
rEC
where C' is the set of object categories in the dataset. Since the model is required to construct predictions by first estimating descriptor similarities Φ, the model is explainable by construction. We can understand why model picked one category by reading the descriptors that are activated in an image. The same mechanism allows us to also understand why a category was not selected.
3 EXPERIMENTS
Our experiments explore visual models with language as an internal representation. We show results for explainable object recognition, adaption to novel categories, and the reprogrammability of visual classifiers in order to correct biases and other errors. We quantitatively and qualitatively compare our method against CLIP, which is one of the most established methods for vision-language pretrain- ing. To analyze the capabilities of our approach on real images, we consider variety of domains, including everyday objects and satellite images.
3.1 EXPLAINABLE OBJECT RECOGNITION
We evaluate our method the ability to perform image classification while also providing expla- nations for its decisions. While most interpretability methods come with a compromise on the benchmark performance, we demonstrate in Table Ithat our approach improves on Compared to CLIP which compares images to embeddings of class names, our approach improves performance by over 3% on average for ImageNet, without training on it. Furtherrmor,, the improvements across a range of domains show that the advantages of our method is not limited to everyday object recogni- tion. For example, we achieve up to ~ 7% improvement on the EuroSAT dataset for satellite image recognition;
2.5% improvement and a ~ 1-2% improvement on the CUB benchmark for fine-grained classification of birds. This suggests that GPT-3 can provide some useful knowledge even for niche domains.
the Describable Textures dataset for texture recognition;
Since there is an inherent bottleneck to first construct linguistic attributes of an image, the method naturally explainable. Fig. [ shows several cases where the model justifies its explanation for the
4

---

Published as a conference paper at ICLR 2023
top prediction: Airliner
top prediction Albatroes
say bocause.
say because.
Average
Averoge
livery or paint schamn
slo powerful Flight
engines mounted on the wings
log hocked bill
landing gear with wheels and tines
long narrow wings
lrge, aircraft
block wingtips
2003
Tuselage with pointed nose
large, long-winged bird
wings
white or grey pumagge
wabbed foot
Our top prediction: Rapeseed
CLIP's top prediction: Bec
that brese
but dan't that because..
petals amanged
-hape
blackack and yellow striped
yellow greanish yallow flower
two palrs
stem with small sharp thoms
2720
chawing
hairy leaves
small round swedpod
small flying
compound eyea
Our top prediction: Valley that Average
CLIP's top prediction. Alpine ibbx that Average
mountains
limbee mammel
river or streamn running through
long curvect horns
a diepasssion in the earthi aurface
27/09
hooves
Jush vegetation
20.13
black, gray,
often with V-shaped profle
short
Our top prediction: Goldfish
CLIP's top prediction: loizen hound say becrure.
say that
Average
2005
Average
23.97
long. flowing tall
long thin laga
scalls that ahimmer the light
25.m1
leary athiaric build
bright color
shart, amooth coat
small block eyes
long narrow head
24.59
lage, pointy ears
madium airadiadd dog
brown or hazel eyea
Our top prediction: Cloak
CLIP's top prediction: Southern Black Widow but me don't aay that bessues.
and we say that bsecie 30.
Average
20.0
Average
has hood
+a small head
typically brack dark color
. black with red hourgloss
of clothing
long. black lags
by
roundi bulbous abdomen
23800
often made wool
Figure 4: (left, in blue) We show example decisions and their justifications from model. (right, in red) We show the prediction from CLIP, and the justification from our model why it did not select that answer. The bar charts show the descriptor similarity & to the image in the CLIP latent space.
prediction. For example, in the top row, CLIP incorrectly classifies the airplane as an albatross, a choice our model disregards because it cannot identify features similar to bird. Instead, model correctly identifies there is an airliner because it can identify many features related to airplanes.
3.2 ACQUTRING AND UTILIZING NOVEL INFORMATION
well-known limitation of machine learning models is that they often perform poorly on data they have not seen during training (Gulrajani & Lopez-Paz 2020 Jiang et al. 20200] Wortsman al. 2022, Djolonga al., 2021). Although foundation models are trained on large portions of the Internet representing a wide variety of data, it is impossible for them to have been trained on concepts that only came into existence after they were trained.
In contrast, our approach acquires visual descriptions from a large language model, which allows it to build new classifiers for categories that ø has not encountered yet. CLIP was originally trained in February 2021, and we added two new categories to the ImageNet validation set that widely appeared on the Internet after this date: a) the Ever Given, which is the ship that blocked the Suez Canal in March 2021 (Wikipedia, 2022a), and b) the game Wordle, online word game that went viral January 2022 (Wikipedia] 2022c). For each category, we added five images into the existing validation set of 50, 000 ImageNet images.
5

---

Published as a conference paper at ICLR 2023
Query
Descriptors
Ours
CLIP
container ship rett white and green
Ever Given
name 1FV12 GIVEN written
DONNN
the the ship
ESITIVE
of connammer the dleck ship
Wordle game
of different colors different lntters
Figure 5: Our retrieved images vs those found by CLIP for concepts popularized after CLIP's train- ing. If CLIP has not observed a particular text-image association when it was trained, it is bound to fail when retrieving images related to that text naïvely. Our approach queries the large language model obtain the necessary external information. Incorporating this knowledge provides it cues for what to look for, allowing for successful retrieval of relevant images.
Subgroup Descriptors
Recognized Images
Wedding
groom wearing tuxedo
Sub-group
Ours
CLIP
Western African
100%
40%
OR
groom wearing a dashiki
Chinese
100%
20%
Japanese
100%
0%
North Indian
100%
60%
groom wearing kimono
Figure 6: (left) CLIP only compares to the word 'wedding', yielding biased results -iaults-itae only correctly approach provides way to address the bias, by
recognizes the first row. The descriptor-basse expanding the initial set of descriptors (only the top) to be more inclusive with prior knowledge. (right) Modifying the descriptors to be more inclusive causes accuracy significant improve on sub-groups.
We quantify the recall retrieving these examples within the top ten detections of the category, and Fig. compares the performance our system versus CLIP. For both categories, our method obtains 100% recall, while CLIP obtains 0% recall for Wordle and 10% recall for the Ever Given. Even though these categories are relatively new, GPT3 is able to build deseriptors for them because enough people the Internet have visually described them. By combining these descriptors gether, the model can recognize the new category. In contrast, since these categories are novel to CLIP, baseline methods obtain poor performance. CLIP matches the "Wordle game" incorrectly category to keyboards instead of the game, likely an artifact of how CLIP tokenizes the input words. For the "Ever Given" category, CLIP retrieved just one correct example in the top ten detections. The example it retrieved correctly has its name written on the ship, and CLIP likely performed well on this example due its reading capabilities (Gihh et al., 202T). However, for the rest of the nine Ever- Given ships in the dataset, CL.IP was unable to correctly detect them with high confidence. This experiment illustrates that visual classifiers can be automatically built by leveraging the descriptive capabilities of large language models.
3.3 CORRECTING FAILURES BY DESCRIPTOR EDITING
Bias remains an unsolved challenge in machine learning, including for foundation models trained on large-scale data. Typically, diagnosing the source of biases challenging because representations usually black-box. However, linguistic attributes makes it possible to both identify which part of the system introduces a bias, and manually correct them in some cases.
We found that both CLIP and our model is biased towards Western celebrations for the "wedding" category. Inspired by the Inclusive Images Challenge (Atwood et al. 2020), we collected a chal- lenge set consisting of images of weddings from various cultures to explore this bias. There many wedding and cultural traditions in the world, and each exhibit their own identifying visual characteristics. On this dataset, CLIP tended to have Western-centric bias, primarily considering

---

Published as a conference paper at ICLR 2023
Jackfruit, which (has/is/etc)
"lion, which has tawny or golden fur" (within top 10)
large, round fruit
green or yellow skin
flesh with
seeds
sweet and sticky taste
strong smell
Vespa, which (has/is/etc)
a type wasp
black and yellow stripes
"jaguar, which is a large, stocky cat" (within top l0)
a stinger
pairs wings
six logs
narrow waist
Hair spray, which (has/is/etc)
aercsable product
aerosable product
aerosable product
(a)
(b)
Figure 7:
(a) Examples of errors in the descriptors produced by GPT-3. (b) Examples of undesired behavior in CLIP retrievals from descriptors. The text in red annotates the error.
"white weddings" (named for the white dress brides wear in this tradition) (Wikipedial 2022b). Our model showed a similar bias. When our classifier prompted GPT3 for the visual descriptors of wedding, the language model mostly prescribed that the groom should be wearing a tuxedo. For these models to be deployed and trusted, we must have ways amend them to be inclusive.
Since the decisions from our model are based on human-readable text, changing them changes the decision process. Figure iillustatts that can intervene to partially fix the bias. We can manually overwrite the attribute for "a groom wearing a tuxedo" to include other clothing traditions, such as dashiki (traditional clothing in many parts of Western Africa) in many parts of Japan). This effectively points the VLM to what it should look for instead of relying solely its biased association with the word wedding. We thus build more inclusive classifier by considering the descriptor similarity of every such sets of descriptors, then choosing the category "wedding" if any of them have the highest total descriptor similarity compared to background classes.
kimono (traditional clothing
On this dataset and in these settings, the table in Fig. Gsoows that intervention can effectively correct this bias. Compared to CLIP, our approach obtains significantly higher accuracy at recognizing the cultural variation of wedding photographs. Figure shows several qualitative examples where the model is reprogrammed with different clothing attributes.
3.4 UNDERSTANDING FAILURE MODES
We explore a few limitations of the method here. We discuss failures both in the language model D(c) and failures in the grounding Φ.
Failure in descriptor creation. noted three types of failures, shown in Figure 7 (a). The first failure mode shortcoming in understanding modalities. Despite being asked for visual features of the category, GPT-3 will occa- sionally produce descriptors that, while correct, reflect other modalities. In Figure 7 (a), we observe "jackfruit" is given descriptors pertaining to taste and smell in addition to vision. As these cannot be seen, they are not useful cues for the vision- language model.
Typically, GPT-3 produces high-quality descriptors; however, we
Another failure case is caused by word ambiguity. Many words have multiple meanings. GPT-3 produces descriptors solely from the name of the category: it cannot see any of the images in the dataset to tell which meaning of word is intended. It picks one meaning to describe. This often succeeds, as it tends to pick the most common meaning - for example, the descriptors for "St. Bernard" relate to the dog breed instead of the religious figure, matching the visual category. In some cases, however, it can lead to descriptors that are completely unrelated to the visual category. The "Vespa" in Figure 7 (a) is illustrative example of this. In ImageNet, this category refers to scooters made by the Italian designer brand Vespa. GPT-3 instead describes a wasp. "Vespa" the Italian word for "wasp" the multilingual capabilities of GPT-3 actually lead an undesired result.

---

Published as a conference paper at ICLR 2023
This type of mistake is to some extent unavoidable solely from text without further information about the category; using the name "Vespa scooter" does not lead to this error.
Finally, GPT-3 still (albeit rarely) produces syntactic errors such as verbatim repetition of the same descriptor (see "Hair spray" in Fig. (a)). As large language models progress, we expect this failure mode continue to diminish. Our framework readily adapts such advances; it will only continue to work better as language models improve further.
Failure in recognizing descriptors. While for most descriptors, CLIP successfully matches the descriptor text to images, we find it can also retrieve unexpected images in certain circumstances. We identify two interesting types of failure in recognition, illustrated in Figure (b). If a descriptor matches multiple categories, there are some cases where it can be strongly activated for categories outside the intended one even when the intended category name is included in the text to be embed- ded. For example, the descriptor "lion, which has tawny or golden fur" retrieves two Chow Chows in the top 10 activations. We hypothesize this is due to the confluence of two factors. The first is of course that it is likely Chow Chows are strongly associated with having "tawny golden fur." But typically, this is not enough on its own, judging by other descriptors (see Appendix for examples). We believe the other relevant factor is having category names that are themselves related. The Chi- nese name for Chow Chows includes the word 'lion' in it; similarly to the "Vespa" case for GPT-3, there error may have a multilingual nature. (We note that the other descriptors for "lion" do not fit Chow Chows the way "golden fur" does, and do not retrieve images of the dogs).
Another failure mode we observe stems from both word ambiguity and the ability of CLIP to solve multiple tasks. In particular, just as its ability to read can impede its ability to recognize objects (Goh et al. (02T), we find this same ability can also impede recognizing descriptors. The second example, "jaguar, which is a large, stocky cat" retrieves jaguars correctly, but is strongly activated by stock photos of jaguars over jaguars which are themselves stocky cats. This is especially true when they contain the word "stock", but also occurs for other stock photos, due to the ambiguity between the word "stock" in how it applies to images rather than cats.
4 RELATED WORK
Vision-language models have grown to be a dominant paradigm for visual recognition with the release of CLIP (Radford et al., [02T), showing strong zero-shot performance on range of bench- marks across distribution shifts. Other work such as ALIGN (Jia et al. (021). FLAVA (Singh et al.) 2022]. Florence (Yuan et al. 2021), and more have since furthered this paradigm. The hallmark of these recent models is that they are trained on large-scale datasets of image-text pairs collected from the Internet. They have seen success on variety of tasks, including classification, detection
(Kamath et al. 2021), and more. Compared to previous models, vision-language models boast the advantage of connecting visual data to free-form language rather than fixed categories. Coneur- rent work Menon et al.) (2022) shows that the visual representations learned by such models can be predisposed towards certain tasks a priori.
Interpretability and explainability for deep models in vision is field too broad to fully cover here; we discuss some relevant examples here, and direct the interested reader to (Gilpin et al. 2019) for broader overview. Much of the work in explaining deep model decisions is post-hoc, often in the form of heatmaps such as GradCAM (Selvaraju et al. 2020). Chefer et al. (2021) extend similar techniques vision transformers, such as those often used for lanion-language models. Such work can be useful for understanding low-level decision factors such as "which part of the input image was the decision based on?" However, as the decision is not constructed from these explanations but rather the explanations are constructed from the decision, there can be questions of faithfulnees,, i.c. how much the generated explanation actually reflects the decision process of the model. In addition, outputs such as heatmaps require some interpretation on the part of the user to parse, and cannot easily capture medium-to-high level factors such as "cash register" being the deciding factor for a 'store'. Other work aims to produce explanations in natural language for vision. Park et al. (2018) create multimodal explanations by training a language model to produce explanations in tandem with a visual classifier, but these text explanations need not be what the visual classifier bases its decision on, rather they intend to be plausible explanations for the given image. [Sammani et al. 2022) extends this idea more closely integrate the representations of both modalities.
8

---

Published as a conference paper at ICLR 2023
Our approach is natural when viewed through the lens of prototype learning with neural networks (Vinyals et al. 2017: Snell et al. 2017: Chen et al.) 2019). with some marked differences. In par- ticular, our work is partly inspired by Chen et al. (2019), which constructs inherently interpretable decisions for visual classification by comparison to prototypical examples in the training set. This work bridges the gap between fully black box neural networks and fully transparent white box mod- els without compromising performance; similarity to the prototypes is estimated with a black box neural network, but which prototypical images the decision was based on and how much for each of them is exposed to the user. This is similar to our use of descriptors at inference time, but requires a unique training procedure and produces only visual explanations. We can view ((c) as a support set of prototype vectors for class c. Rather than prototypes in the typical sense, however, these are text prototypes for visual data. Work such as Chen et al. (2019) has previously explored prototypes for inherently interpretable models, but these have been "visual words" rather than free text. In addition, we do not need to learn our prototypes, instead making use of pre-trained foundation mod- els. We note that one difference from ProtoNets (Snell et al. 2017) is that we compute the class score 8 as the mean similarity to support vectors (i.e., descriptor) rather than the similarity to the mean of the support vectors. Aggregating similarities enables interpretability, be decomposed into similarity with each descriptor, more akin to Chen et al. (2019); Vinyals et al. (017). Comparing the mean of support vectors is essentially the approach of typical "prompt ensembling." CLIP (Radford et al. (021) presents a remarkable ensemble of 80 prompts hand- designed for the ImageNet dataset over the course of a year. We encourage readers to see concurrent work (Pratt et al.] (022) that demonstrates that prompt ensembling with prompts obtained by large language models can improve accuracy on recognition tasks. One of the key differences between Pratt et al. (2022) and our work is that while prompt ensembling is an effective tool for increasing accuracy, it does not afford the same interpretability, editability, or adaptability on a per-descriptor basis that our approach does.
the class score can
There are several advantages to using text prototypes. Critically, using natural language allows us to obtain descriptors by leveraging the world knowledge condensed into large language models such as GPT-3, This eschews costly learning processes and incorporates external knowledge effectively. Text prototypes also are readily interpretable, whether by technical user layperson. It is easier and more natural for a person to edit text than to edit visual data to a desired prototype, including to define a previously-unsee unseen category.
Other work demonstrates that external text knowledge can provide substantial aid to vision tasks. K-LITE (Shen al., 2022) shows that such knowledge from WordNet and Wiktionary has the po- tential to enhance prompts for vision- connecting large language models to vision-language models, enabling emergent capabilities such as image captioning. Large language models, especially GPT-3 (Brown et al. 22020), have seen widespread application since their introduction due to their impressive ability to generate sequences similar to those observed from humans. PICA (Yang et al. (021) demonstrates knowledge derived from GPT-3 can aid few-shot VQA tasks.
models. (Zeng er al. 2022) is similarly motivated in
Our work is also closely related to zero-shot attribute- based classification, such as (Lampert et al.) 2014]. Parikh & Grauman (20TT) develop a system to create a vocabulary of "nameable" attributes from humans. RRomer--Paredes & Torr (2017) demonstrate a framework where connections between attributes and classes are given by the environment. [Socher et al. (003)) introduce the idea of using word embeddings to use knowledge distilled from large-scale text corpora for zero-shot visual recognition; we go a step further and use large language models to obtain the words to embed themselves. These selected works share similar motivation our use of large language models to create dictionaries of descriptors. As this area is also too large to summarize shortly, we direct the interested reader to Xian et al. (2020) for further reading.
CONCLUSION
We introduce a new framework for zero-shot classification with lasion-language models. We lever- age the linguistic knowledge about visual categories from large language models to generate textual descriptors for each category, comparing images to these descriptors rather than estimating the sim- ilarity of images directly with category names. Using GPT-3 and CLIP, we show promising results showing the capabilities of this framework to provide interpretable model decisions, improve per- formance on recognition tasks, enable adaptation to new knowledge, and mitigate bias.

---

Published as a conference paper at ICLR 2023
ETHICS
Large pretrained models such as GPT-3 and CLIP learn various biases relating to race, culture, gender, and more from the Internet Goh et al. (2021). Systems that use these models can reproduce or even exacerbate this bias. Using both in tandem has the potential to compound the biases of both models. Interpretable models, like the one we present, have the potential shed light on these biases that could otherwise remain unknown. For example, it is likely that descriptor dictionaries produced by GPT-3 could reflect its biases. We hope that the methods we present for editability and bias mitigation serve useful tools to combat said biases.
7 REPRODUCIBILITY
We use CLIP as our vision-language model and GPT-3 our large language model, both of which can be queried by anyone CL.IP having weights available, and GPT-3 having public API. Suffi cient details to reproduce the method can be found in Section (for inference) well as Appendices (for querying language models) and (for editability). We will release the data for the editability and adaptability experiments where the appropriate licenses permit and sufficient de-identification possible. We will also release code upon publication.
ACKNOWLEDGEMENTS
This research is based on work partially supported by the NSF NRI Award #2132519, the DARPA MCS program, and the DARPA GAILA program. SM is supported by the NSF Graduate Research Fellowship. We' d like to thank Dídac Surís and Rich Zemel for helpful discussions and feedback.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
James Atwood, Yoni Halpern Pallavi Baljekar, Eric Breck, D. Sculley, Pavel Ostyakov, Sergey I. Nikolenko, Igor Ivanov, Roman Solovyev, Weimin Wang, and Miha Skalic. The Inclusive Im- ages Competition. In Sergio Escalera and Ralf Herbrich (eds.), The NeurlPS '18 Competition, The Springer Series on Challenges in Machine Learning, pp. 153-186, Cham, 2020. Springer International Publishing, ISBN 978-3-003--2915--8 doi: 10. 10.1007/978.3-0002293333
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 Mining Discriminative Components with Random Forests. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision - ECCV 2014, Lecture Notes in Computer Science, pp- 446-461, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10599-4 doi: 10. 07177/97888-39--1033
29.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. arXiv:2005. 14/65 arXiv: 2005.14165.
[cs], July 2020. URLhttse /ttpaaaiiiv/../////v////aaaaaaaa http:/iwwwggg/a//////aaaaaaa/ httpp:://aaw.v.....r/a//////5050 2200. 14165
Hila Chefer, Shir Gur, and Lior Wolf. Generie Attention-model Explainability for Interpreting Bi- Modal and Encoder- Transformers. arXiv:2103. 15679 [cs], March 2021. URL http:
//axxxv.org/abs/////// 2103. 15679 arXiv: 2103.15679.
Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. This Looks Like That: Deep Learning for Interpretable Image Recognition, December 2019. URL
http:/ /tttxi//rxxiv.... org.org/////...o abt///////// www.aa/s........aae...a..ee.eeea 10574 arXiv: 10574 [cs, stat].
Mircea Cimpoi. Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and An-
drea Vedaldi. Describing Textures in the Wild. pp. 3606-3613, 2014. URL
https: thecvf wtt.:oo//c/////o content..eeetteeeco cvpr_ 2014/htm1/Cimoo) Desorribbnng_Tettueee in in__inn_000__ 2014_00122_____ CWWR__paprr.ttrr
Josip Djolonga, Jessica Yung. Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander A'mour, Dan Moldovan, Sylvain Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lucie. On Robustness and Transferability of Con- volutional Neural Networks, March 2021. URL [cs]. Number: arXiv:2007.0855: ar arXiv:2007.0855
Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining Explanations: An Overview of Interpretability of Machine Learning, February 2019. URL
abs/ 00069 1rXiv:1806.0000 1806.00069 [cs, stat].
Gabriel Goh, Nick Cammarata t, Chelsea Voss t. Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal Neurons in Artificial Neural Networks. Distill, 6(3): e30, March 2021. ISSN 2476-0757. doi: 10.23915/distill.000 pub/2021 mu t modal neurons
URL https:///ssstill
Ishaan Gulrajani and David Lopez-Paz. In Search of Lost Domain Generalization. arXiv:2007.01434 (cs, stat], July 2020. URL http:://arsv...rrg//b///////// arXiv: 2007.01434.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification, February 2019, URL [cs].
http://arxxt orgg abs / 709. 00029
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li. and Tom Duerig. Scaling Up Visual and Vision- Vision-Language Representation Learning With Noisy Text Supervision. arXiv:2102 05918 [cs], June 2021. URL http: arXiv: 2102.05918.
//arsiv.org//bs//2000 05101

---

Published as a conference paper at ICLR 2023
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust Pre-Training by Adversarial Contrastive Learning. arXiv:2010. 13337 [cs], October 2020. URL http://axxiv..rrt
abs/2010. 13337 arXiv: 13337.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. MDETR- Modulated Detection for End-to-Ena Multi-Modal Understanding. October 2021.
URL http: /ttp://axvvvv/// org/ aaaaaass/2110 12763 arXiv:2104.1270 [cs].
Simon Kornblith, Jonathon Shiens, and Quoe V. Le. Do Better ImageNet Models Transfer Better? In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2656- 2666, Long Beach, CA, USA, June 2019. IEEE, ISBN 978-1-72813-293-88 2019.00277. URL https:: ore leee. org http:///dooooetocce//oo 899443847
doi: 10.1 10.1109/CVPR.
Christoph H. Lampert, Hamnes Nickisch, and Stefan Harmeling, Attribute-Based Classification for Zero-Shot Visual Object Categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(3):453-465, March 2014. ISSN 0162-8828, 2160-9292. doi: 10. 1091/AM.P 2013.140. 140. URL pattpp//ieeexp.ie0
ieee. wwtpo///gggo/////o/////e 6571196/
Sachit Menon, Ishaan Preetam Chandratreya, and Carl Vondrick. The risks of versatile models: Resolving task ambiguity for vision- models. arXiv, 2022.
Devi Parikh and Kristen Grauman. Interactively building discriminative vocabulary of name- able attributes. In CVPR 2011, I. pp. 1681-1688, Colorado Springs, CO, USA. June 2011. IEEE. ISBN 978-1-4577-039422 doi: 10. 1119000P..201.....0 1,59954511. URL. htll.httt:/eett http:/eeeeppppp. ore.
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Dar- rell, and Marcus Rohrbach. Multimodal Explanations: Justifying Decisions and Pointing to the Evidence. 2018 IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. Salt Lake City, UT, June 2018. IEEE ISBN 978- 1-220-6---6666 doi: 10.1109) WVPR.2011.00911 URL
ieee. tttp//ocmeo///////o5/55.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498-3505, June 2012. doi: N0.11000/CVRR.22222224233
ISSN: 1063-6919.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language Models as Knowledge Bases? In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IJCNIPP), pp. 2463-2473, Hong Kong. China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D199 1250. 1250
URL
Sarah Pratt, Rosanne Liu, and Ali Farhadi. What does platypus look like? Generating cus- tomized prompts for zero-shot image classification, September 2022. URL http:://axxvv
org/abs/ 22...0.000. 03320 arXiv:2209.0333 [cs].
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. 2rXiv:2/03.0000 [es]. February 2021. URL http:://axxiv www.///..//.a.....a.a/ 00020 arXiv: 2103.00020.
Bernardino Comera-Paredeer and Philip H. S. Torr. An Embarrassingly Simple Approach to Zero- Shot Learning. In Rogerio Schmidt Feris, Christoph Lampert, and Devi Parikh (eds.). Visual Attributes, pp. 11-30. Springer International Publishing, Cham, 2017. ISBN 978-3-319-50075-1 978-3-319-50077--3 doi: 10.10077978-33311 3-1-5-07---199-077-5522 URL http::/t/hiiii springer. Series Title: Advances in Computer Vision and
com/ http110010079988888 319- 50077-5 Pattern Recognition.
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong, Inter- pretable Machine Learning: Fundamental Principles and 10 Grand Challenges. arXiv:2103.1m 11251 [cs, stat), July 2021. URL htttp::////xvvv. wwtpa/wwwwww.wr.p/a//aaa///////e
11251 arXiv: 2103.11251
12

---

Published as a conference paper at ICLR 2023
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. arXiv:1409.0573 [cs], January 2015.
URL
arXiv: 1409.0575.
Fawaz Sammani, Tanmoy Mukherjee, and Nikos Deligiannis. NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Tasks, March 2022. URL http://
arxiv. wexiv.org//aa.. abs 2ttpb///2233.0000000 05081 arXiv:2203.0555 [cs].
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das. Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Lo- calization. International Journal of Computer Vision, 128(2):336-33) February 2020. ISSN 0920-5691, 1573-1405. doi: 10. 120-7//1126--019--22222 URL
02391
arXiv: 1610.02391.
Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang. Pengchuan Zhang, Anna Rohrbach, Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, Kurt Keutzer, Trevor Darrell, and Jian- feng Gao. K-LITE: Learning Transferable Visual Models with External Knowledge, April 2022.
URL http::tttapiii/ org/ abs 2204 09222 arXiv:2204.09222 [cs].
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Mar- cus Rohrbach, and Douwe Kiela. FLAVA: A Foundational Language And Vision Alignment Model, March 2022. URL http://arxx wwwprxiwwi.rrrooooooooo httg/abb/22122 04482 [cs].
arXiv:2112.04482
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical Networks for Few-shot Learning, June 2017.
05175 arXiv:1703.05175 [cs, stat].
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, and An- drew Ng. Zero-Shot Learning Through Cross-Modal Transfer, March 2013. URL http:
//arxiv.oo org/ hat///b///// wtt.aa////11..ee. 3666 htt::3301.366 [cs].
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing Networks for One Shot Learning, December 2017. URL http::/aaxxv.orrg//////
1606. 04080 arXiv:1606.0440 660.000060000 [cs, stat].
Catherie Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD Birds-200-2011 Dataset, July 2011. URL https: http:::/rresoverr.. cawtech. edu/
CaltechAUTHORS: 22111026-122511120
Issue: 2010-001 Num Pages: 8 Number:
2010-001 Place: Pasadena, CA Publisher: California Institute of Technology.
Wikipedia. Ever Given, August 2022a. URL https: htt.p::://ewwwwiipeeeeeeeeeeeeee http://wwwkppdii...oooooo//////a
ehpettttleeEEver_Gieeeee 1110000001111000111100001011 Page Version ID:: 110501
Wikipedia. White wedding, September 2022b. URL https::/en wwkipedia. org/
wtt.eex.pppptttleWiieeee 1112110253. oeg.s:/wwheeeeeoccoooooodddddd.d
Page Version ID:
Wikipedia. Wordle, September 2022c. http:///ww///////p////aaaaaaaaaa
php?
Litte=xkorcleeooddddd--113333 Page Version ID: 1112595044.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. arXiv:2109.0 10900/903 [cs], February 2022. URL
http:://rxia www.ooggaaa... abs 220..11111.. 01903
arXiv: 2109.01903.
Yonggin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-Shot Learning A Comprehensive Evaluation of the Good, the Bad and the Ugly, September 2020. URL http: arXiv: 111i:11770000000 [cs].
//arxivvorg//bs//100 00
Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An Empirical Study of GPT-3 for Few-Shot Knowledge-Basse -assed VQA. September 2021. URL Number: arXiv:2109.0501 arXiv:2109.05014
http: /tpp:///xxvv.. arxii.v org/ hbs//bb//200 abs/2109, 05014 [cs].
13

---

Published as a conference paper at ICLR 2023
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Lu- owei Zhou, and Pengchuan Zhang. Florence: A New Foundation Model for Computer Vision, November 2021. /RL.hhtt:://rxxivoorr//
11432 arXiv:2111.11432 [es].
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong. Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Van- houcke, and Pete Florence. Socratie Models: Composing Zero-Shot Multimodal Reasoning with Language, May 2022. URL htt.:://arx...orggabss. 2204 00598 arXiv:2204.00598 [cs].
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle- moyer. OPT: Open Pre-trained Transformer Language Models, June 2022. EXXiv2225500066 [cs].
URL http:
//araav.orr/aas//2050000000
14

---

Published as a conference paper at ICLR 2023
A PROMPTING THE LANGUAGE MODEL
A key aspect of method is producing discrete, separate descriptors through the language model, rather than embedding everything together as is typical for prompting. Thus, an important consider- ation is how encourage the language model to produce descriptors in way that does require human parsing. Recall our prompt structure is of the form
Q: what are useful features for distinguishing (category name) in photo?
A: There are several useful visual features to tell there is (category name) in photo:
We find that adding the trailing '-' enough typically result in a bulleted list output. This is simple to automatically obtain descriptors from by simply removing the hyphens.
Per OpenAl's API recommendations, we the structure Q: A: for the query and the desired response. We sample from the "text-davinci-002" model with temperature of 0.7 and maximum token length of 100.
As with previous work with GPT-3, we find that the list formatting becomes more reliable when one or two examples of desired output are provided. These can be of the form
Q: What are useful visual features for distinguishing a lemur in photo?
A: There are several useful visual features to tell there is a lemur in
photo:
four-limbed primate
black, grey, white, brown, or red-brown
wet and hairless nose with curved nostrils
long tail
Large eyes
furry bodies
claweq hands and feet
which itself was constructed by GPT-3 (although such examples can easily be constructed by hand as well).
Presumably, all of this could be improved with more effort towards prompting; we did not tune these prompts at all after our initial generation of descriptors for the 1000 ImageNet classes.
B DATASET DETAILS
We consider the ImageNet dataset (Russakovsky et al. 2015) for everyday object recognition; Im- ageNetV2 (Kornblihh et al., 2019) for distribution shift from ImageNet: CUB for fine- ine-grained clas- sification of birds (Wah et al. 2011;; EuroSAT (Helber al.
2019) for satellite image recognition;
Places365 for scenes; Food10l (Bossard et al. 2014) for food: Oxford Pets (Parkhi et al.) (012) for common animals; and Describable Textures Cimpoi al. (2014) for in-the-wild patterns.
15

---

Published as a conference paper at ICLR 2023
3.00
VTT-L/14@336px
2.25
VIT-L/14
Improvement Over CUP
1,50
ViT-B/16
VIT-B/32
0.00
-1.50
-2.25
-300
200
400
600
800
1000
1200
1400
Size of LLM
Figure 8: We use the OPT /Zhang et al. (2022) series of language models to evaluate the influence of different language models at different parameter counts. We find that using descriptors from smaller language models can actually hurt performance, while after certain size performance gains appear; this trend holds across sizes of VLM. We hypothesize this is because smaller models do not contain the knowledge of the visual world that larger language models possess.
C FURTHER COMPARISON
In this section, we provide some additional interesting quantitative comparisons.
ImageNet
Wiktionary Descriptions
Wordnet Descriptions
Ours
CLIP
ViT-B/32
57.91
60.00
62.97
58.99
VIT-B/16
62.56
64.58
68.03
64.05
ViT-L/14
68.87
71.14
75.00
71.57
VYT-L/14@366p
69.21
72.16
76.16
72.96
Table 2: Comparison with auxiliary information obtained from WordNet and Wiktionary rather than GPT-3. We observe that the Wiktionary information tends to hurt performance, while WordNet sometimes slightly helps and sometimes slightly hurts. We hypothesize this is because the informa- tion contained in WordNet and Wiktionary concerns definitions more often than visual descriptions.
D EDITABILITY/BIAS MITIGATION EXPERIMENT DETAILS
We collect 10 images depicting each of four wedding traditions from Western Africa, China, Japan, and Northern India. We use Flickr to collect these images with Creative Commons licensing, with the exception of the Western African examples, which could not be found on Flickr; for these,
Aggregation Method
Mean
Max
VIT-B/32
61.34
60.00
ViT-B/16
66.45
64.58
ViT-L/14
73.15
71.14
WTT-L/14@336mg
74.19
72.16
Table 3: Comparison between aggregating deseriptor similarities for given category using their mean (as described throughout the paper) and their maximum. We find the mean to have a consistent benefit over max, suggesting using multiple justifications conjunction provides some benefit.
16

---

Published as a conference paper at ICLR 2023
ImageNet (80 Prompts)
Ours
CLIP
A
ViT-B/32
63.76
63.37
0.39
ViT-B/16
68.83
68.36
0.47
ViT-L/14
75.96
75.52
0.44
ViT-L/14@ 336px
76.85
76.57
0.28
Table 4: Comparison with ImageNet using all 80 hand-engineered prompts created for the original CLIP paper. Though the prompts are not hand-tuned to descriptors, we note that they still pro- vide some benefit; this motivates future work creating hand-tuned prompts explicitly designed for descriptor-based methods.
tiger big teeth
killer whale
chameleon
long. curved dorsal fin
head with crest of spikes or horns
broom bristles
ice cream toppings
candle
a flame
totem pole carvings of animals or humans
lighthouse
cannon large, cylindrical barrel
location a hill or cliff
Figure Top retrievals for various descriptors.
use Google Images. (We will release the images for which licenses are provided with identifying features blurred.)
To evaluate performance, we add the category 'wedding" to the 1000 ImageNet categories. We provide descriptors corresponding each of the edited versions of the original, GPT-3-constructed descriptors of white wedding. We perform these edits by identifying cross-cultural analogs in each descriptor and replacing the Western-specifie words, such as 'tuxedo' African example. This results in 5 subgroups, including each of the four additional cultures and the original Western-centric descriptors; if the average descriptor score for any of these 5 the highest, the category chosen is "wedding." We remove the existing category 'bridegroom' as this leads to category overlap. but the more general 'wedding" need not include only men. We use the CLIP RN50 model in these experiments, but find similar results across model sizes.
'dashiki' for the Western
E
TOP DESCRIPTOR ACTIVATIONS
Here show various descriptors' top retrievals in Fig.
17


Reviews:
[{"review_id": "oFaCC0mDjpc", "reviewer": "Reviewer_UNQW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple idea to improve zero-shot image classification using CLIP-like models. It uses a pre-trained LLM (GPT-3) to automatically generate text descriptors for visual categories. The text descriptors are then used by CLIP to compute image-text similarities as prediction scores for each category. The method shows improved zero-shot classification performance on ImageNet and CUB datasets over the naive CLIP prompt.", "summary_of_the_review": "This paper opens up an interesting new direction that uses LLMs to generate visual descriptors for CLIP-like models. I have two major concerns:\n- The ability of the proposed method to generalize to other domains.\n- Comparison with CLIP's prompt is not entirely clear. Comparison with knowledge-based method could be better.\n\nI will happily raise more score if my concerns can be cleared. ", "strength_and_weaknesses": "Strengths:\n- The proposed idea is simple and well-executed. The paper is also well-written.\n- Visual-prompt-engineering can be a heuristic and tedious process. It is refreshing to see that LLMs can help to automate this process. The prompt for GPT-3 is also well-designed to guide the model in generating visual descriptors. \n- The method provides interesting additional capabilities such as explainability and novel category classification.\n\nWeakness:\n- Could the proposed prompt-generation method generalize to other domains other than common objects and animals? It is necessary to evaluate on more datasets as in the CLIP paper, such as food/scene/satellite/medical images. In order for the method to be practical, it should at least not be harmful on the more difficult domains.\n- It should be made clearer (rather than in the footnote) that the CLIP results in Table 1 uses a single base prompt, which I assume is \"a photo of {class_name}\"? Does the proposed method include the same base prompt with class_name?\n- It is suggested to include the result for the hand-crafted CLIP prompts in Table 1.\n- There exist other methods to construct visual descriptors that can offer similar advantages such as explainability. For example, using WordNet (K-LITE) or wikipedia articles [a]. It would be good to see some comparison between descriptors generated by GPT-3 and descriptors retrieved from knowledge base.\n\n[a] Large-Scale Zero-Shot Image Classification from Rich and Diverse Textual Descriptions. Bujwid and Sullivan. ACL 2021.\n\nMinor suggestions:\n- One of the entry in Table 1 has an inconsistent number of digits. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "j52PJ2pQGfm", "reviewer": "Reviewer_4Ax6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a simple but effective approach to zero-shot learning and explainability for image classification. The title of the paper says it all, but for completeness here is a summary. The main idea is to query a large language model, in particular GPT-3, to obtain multiple short descriptions about a class. These short descriptions are then projected into a joint embedding space which was learned to encourage visual grounding via multi-modal training pairs (ie CLIP). Then when an image from this class is encountered it is projected to the joint embedding space and its similarity to each class is computed by aggregating the similarity of the image's encoding to each encoding of the class's short descriptions. The image is then classified as the class that gives the highest overall similarity score. A by-product of this system is that the short descriptions with the highest scores then give a ready made interpretable explanation of why the system made its decision.\n\nThe contributions of the paper:\n\n1) The paper demonstrates, and adds to the evidence, that the publicly available large language and multi-modal models, trained on large scale data such as CLIP, can be leveraged via descriptions (and prompt engineering) to perform explanability without a drop in classification performance and zero-shot learning. The interesting part is that the amount of manual prompt engineering can be minimized and can be performed by GPT-3 instead of by a human.\n\n2) The paper raises interesting questions to think about how to best leverage the large scale foundational models in vision and language to perform zero-shot learning especially in a world with ambiguities and that is evolving constantly and how best to transfer knowledge from large scale language models to visual models.", "summary_of_the_review": "My recommendation is based on the fact that I think the results achieved are interesting and stimulating for a reader with an interest in zero-shot learning. There are some weaknesses in the presentation of the technical and some of the experimental evidence seem more like tidbits as opposed to fully fleshed out ideas. However, I think overall I would be in favour of publication with some cleaning up of the presentation.  \n\n(\nTypos spotted along the way\n\n* Page 5 last line: valiation --> validation\n* In section 4 on the 2nd line of the 2nd paragraph there is an empty reference. \n)\n\nResponse to the authors' feedback:\nI would like to thank the authors for their responses and the extra information and results provided. They have addresses my concerns I have now upgraded my rating accordingly. Well done on an interesting and fun paper!", "strength_and_weaknesses": "Strengths:\n\n- The paper is generally well written and a fun read. It has a interesting experiments and shows that interesting results in explainability can be achieved almost for free from manual intervention by harnessing GPT-3 descriptions and models in the mould of CLIP. \n\n- A nice set of interesting experiments showing that the CLIP results on ImageNet can be improved upon from text descriptions automatically obtained from GPT-3 as opposed to the manually specified in the original CLIP paper. There is just one reservation here in that it is not clear which text prompts were used for the CLIP results in table 1. \n\nWeaknesses:\n\n- The original CLIP paper has already demonstrated that prompt engineering can have a significant impact on the zero-shot recognition results. So this finding in the paper is not new.\n\n\n- The experimental evidence presented is nice. But some of these experiments are a little sparse and a little artificial. This is especially true for the \"acquiring and utilizing novel information\" learning experiments. It is fun to see that \"Wordle\" and \"Ever Given\" images can be recognised from their GPT-3 descriptions but it is obviously far from a complete study of the model's capabilities on this front and it is a bit unclear what it demonstrates. \"Ever Given\" is a proper noun and the name of a specific container ship. It is not surprising that CLIP cannot do image retrieval based on a proper noun which was generally unknown at the time of training CLIP. \n\n   * The paper mentions that very little in the way of prompt engineering is done. The engineering that is performed is mainly done to ensure a list of descriptions is created by GPT-3. How sensitive are the results to the temperature and max token length used in the language model? Were the default values used or were they set using some form of hyper-parameter sweep.\n   \n\n- There are some technical details missing or brushed over. So for example:\n\n   * In equation (1) how exactly is \\phi(d, x) computed? Is a cos similarity used, does it use a softmax to compute the probabilities, etc...?\n   \n   * For table 1 for the CLIP column what is the text prompt used to acquire the classification? Is it just a classname or does it use the prompt \"A photo of a {label}\" as suggested in the CLIP paper as the basic prompt that boosts performance over just using the classname.\n\n   * In figure 1 and figure 4 there are bar charts showing the compatibility between a text description and an image. What exactly are the numbers being displayed? Also the horizontal scaling seems to be different between the different examples and thus gives an inconsistent visual cue via the length of the bars. \n\n- Figure 4 gives some qualitative results for the explainability for particular images. But it would be nice to have some more quantitative measures of how well the explainability is doing or thoughts on how to do this or show which explanations are most useful for each class etc... Also would some calibration be required to decide which descriptions are actually \"valid/present\" for a given image.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "faUGipncUDt", "reviewer": "Reviewer_LAah", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores using a list of category descriptions as a better representation for image categorization using VLMs such as CLIP. Decomposing the category word into a set of related descriptions could bring a better textural representation when doing matching. In addition, these descriptions could provide interpretability.  The paper also demonstrates more benefits such as improvements in accuracy on ImageNet across distribution shifts; recognizing unseen concepts, mitigating bias.", "summary_of_the_review": "This paper proposes a simple and yet effective method to improve current clip image categorization quality.  The results are relatively preliminary but it proposes a novel direction and clearly demonstrated the benefits (interpretability, removing biases, etc.)\n\nThis paper might be stronger if it:\n1. demonstrates if this method is generalizable to other big models\n2. explore in more systematic treatment on how to combine descriptors in addition to simply adding them. (Eq.1)\n3. provide more in-depth analysis on the potential different results on categories at different levels in a hierarchy  (e.g., dog vs husky)", "strength_and_weaknesses": "Strength:\n1. This idea is quite novel and intuitive. \n2. The experiments have clearly demonstrated the benefits of the proposed methods. \n\nWeakness:\n1. the aesthetics of the figures could be further improved. \nsome potential questions were not answered\n1. generalization: how this generalizes to other tools other than CLIP and GPT?\n2. how do you weight the prob. of a set of features linked with \"or\" logic? for example, in the appendix, the lemur could be \"black, grey, white, brown, or red-brown\", do you just sum these prob up? A better way might be using max?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "79s44FfxPp4", "reviewer": "Reviewer_nRmq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper propose an image recognition paradigm named \"classification by description\". To be specifically, instead of matching an image with its categorical description, the method first queries a large language model to obtain some textual descriptions for each category and base the recognition decision on these descriptors through consensus. In addition to interpretability, the method boosts recognition accuracy on the ImageNet and ImageNetV2. The paradigm is also able to recognize novel concepts and mitigate bias by descriptor editing. ", "summary_of_the_review": "This is generally a well-written paper with a novel approach to achieve both interpretability and performance boost. It would be better if there is more quantitative experiment to support.", "strength_and_weaknesses": "### Pros:\n\n+ It is a conceptually novel to obtain text descriptors for image recognition by querying large language models.\n\n+ The classification via text description naturally provides explanations for decision of objection recognition (Sec 3.1) and can potentially reduce existing biases in existing methods (Sec 3.3).\n\n### Cons: \n\n- The performance from the original CLIP is a weak baseline which does not use prompt ensembles. To obtain the text descriptors from GPT-3, we still needs to prompt and it is also mentioned in Appendix A that \"the list formatting becomes more reliable when one\nor two examples of desired output are provided\" which needs additional manual efforts. That's why we need to allow the CLIP baseline to use better-designed prompts as well.\n\n- What's the unit of values in the bar plots (Fig 1, 4)? From Eq (1), if $\\phi(d, x)$ refers to a log probability, then the score would be in the range of $(-\\infty, 0]$, no?\n\n- The quantitative experiment is weak. I would classify Sec 3.2-3.4 as qualitative analysis since they are not evaluated on well-recognized benchmarks. I think it would strengthen the paper if there is more quantitative experiment other than image classification. Maybe long-tail object detection on LVIS or so?\n\n- I didn't fully get the statement \"Our approach does not fit on the ImageNet training dataset\"? Do you mean that your method is not trained on ImageNet train set (equivalent to a zero-shot setting)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%