PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts
Authors: Bang An
Sicheng Zhu
Michael-Andrei Panaitescu-Liess
Chaithanya Kumar Mummadi
Furong Huang
Venue: ICLR 2024 poster (2024)
Keywords: vision-language model
CLIP
zero-shot
image classification
human perception
contexts
contextual attributes
spurious feature
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose PerceptionCLIP, a zero-shot classification method that emulates the human perception process, achieving better generatlization, group robustness and interpretability.

Abstract: Vision-language models like CLIP are widely used in zero-shot image classification due to their ability to understand various visual concepts and natural language descriptions. However, how to fully leverage CLIP's unprecedented human-like understanding capabilities to achieve better performance is still an open question. This paper draws inspiration from the human visual perception process: when classifying an object, humans first infer contextual attributes (e.g., background and orientation) which help separate the foreground object from the background, and then classify the object based on this information. Inspired by it, we observe that providing CLIP with contextual attributes improves zero-shot image classification and mitigates reliance on spurious features. We also observe that CLIP itself can reasonably infer the attributes from an image. With these observations, we propose a training-free, two-step zero-shot classification method PerceptionCLIP. Given an image, it first infers contextual attributes (e.g., background) and then performs object classification conditioning on them. Our experiments show that PerceptionCLIP achieves better generalization, group robustness, and interpretability.

Published as a conference paper at ICLR 2024
PERCEPTIONCLIP. VISUAL CLASSIFICATION BY IN- FERRING AND CONDITIONING ON CONTEXTS
Bang An
Sicheng Zhu
Michael-Andrei Panaitescu-Liess'
Chaithanya Kumar Mummadi
Furong Huang'
University of Maryland, College Park
2 Bosch Center for Artificial Intelligence
ABSTRACT
Vision-language models like CLIP are widely used zero-shot image classification due to their ability understand various visual concepts and natural language descriptions. However, how to fully leverage CLIP's unprecedented human-like un- derstanding capabilities achieve better performance is still an open question. This paper draws inspiration from the human visual perception process: when classifying object, humans first infer contextual attributes (e.g., background and orientation) which help separate the foreground object from the background, and then classify the object based on this information. Inspired by it, we observe that providing CL.IP with contextual attributes improves zero-shot image classification and mitigates reliance on spurious features. We also observe that CLIP itself can reasonably infer the attributes from an image. With these observations, two-step zero-shot classification method Percepti i onCLIP. Given an image, it first infers contextual attributes (e.g., background) and then performs object classifica- tion conditioning on them. Our experiments show that PerceptionCC cnnrpttiinCIIPP achieves better generalization, group robustness, and interpretability. Our code is available at httpps:: //git com umd huang- ab/ hetp/percee cept onCLTP
propose training- freee
Zero-shot inference
Grad-CAM visualiattin
spurious
Prompt photo of (olass)
dog. 14.33%
3.50%
CO
photo of dog, upside down.
5.19%
Perceptrowe
Prompt photo of on object, (conteat))
spurious
photo of 6005%%
0023%
Prompt photo of (class), grass.
photo of cot upside down
0.02
0442%
condition
Figure 1: (Left): CLIP co-relates natural language descriptions of contextual attributes with visual cues (orientation: upside-down). (Center): Unlike CLIP's standard zero-shot inference that uses fixed template(s) for class name retrieval, our method first infers contextual attributes (background: on the grass) using CLIP and then let CLIP predicts the class conditioned on the inferred contextual attributes. Here, background and orientation Grad-CAM visualization illustrates that our method focuses more on core features (on the dog) and is less distracted by spurious features (grass buckgroundi when performing the object classification.
both examples of contextual attributes. (Right):
INTRODUCTION
CLIP (Contrastive Language-Iman Pretraining by Radford al. (202T) is foundational Vision- I Language Model (VLM) that connects the fields of vision and natural language. By pretraining on 400 million image-caption pairs, CLIP can associate various visual concepts with their corresponding natural language descriptions, making it the foundation for numerous other vision-language models
Equul contribution

---

Published as a conference paper at ICLR 2024
2023 Liu al., 2023 Dai et al., 2023 Li et al., 2023b), diffusion models (Ramesh iet al. 2022; Rombach et al., 2022), and semantic segmentation models (Kirillov et al. 2023). This remarkable understanding capability of CLIP is significant for zero-shot classification (Larochelle et al. 2008), enabling open-ended image classification via natural language without training. This capability also addresses many challenging tasks with limited or no downstream data, such model deployment ieptlymentiii in the wild (Li et al. (2023a) medical image classification (Wang et al. 2022) and satellite object recognition (Ramaswamy et 2023).
(Zhu et al.
Although CLIP shows strong potential in zero-shot classification, current methods treat image classi- fication as a text retrieval task and lack systematic investigation into the text prompts used. This leads to sub-optimal generalization (Radford et al. 202T). reliance on spurious features Yang et al. (2233)
biased predictions (Agarwal et al. 2021 Chuang al., 2023), and lack of interpretability (Zhou et al. 2022b; Menon & Vondrick 2022). For example, Radford et al. (2021) uses a basic template "a photo of fclass name)" identify the most relevant class for an image, much less informative than the image captions used during pretraining (see examples in Table 9). Another method, prompt ensembling (Radford et al.) 202T), employs 80 crafted templates for better generalization. Neverthe- less, it remains unelear whether these templates are optimal and why they effective. By treating zero-shot classification simply as a class name retrieval problem, these methods potentially waste the capability of CLIP to understand both class-specific features and class-independen attributes such as background and orientation (referred to as contextual attributes in this paper).
Given CLIP's unprecedented human-like vision and language understanding, natural idea is to draw inspiration from human visual perception. Classic neuroscience (Kandel et al. 20133) describes human visual perception as a three-tiered, context-dependent process: first discerning basic visual attributes like color and orientation, then analyzing scene layout and distinguishing foreground from background, and finally recognizing objects (see details in Appendix C). For example, when humans classify objects in images, we unconsciously acquire contextual attributes like the background and orientation, and in the case of an upside-down image (Figure left), we first infer that the image rotated and then calibrate our classification accordingly. This hierarchical and context- dependtt process contrasts with cxisting classification methods, which overlook contextual attributes.
Building this insight, we propose zero-shot classification method called Percepti inCIIP, which emulates a crucial part of human visual perception - inferring and conditioning on the contextual attributes resulting in improved generalization, reduced reliance spurious features, better group robustness, and interpretability. Our contributions are as follows:
D (1) We prepare CLIP for perception by structuring LLIP-understandable text prompts with contextual attributes and introducing an attribute-aware CLIP score to approximate essential conditional probabiliiies for perception emulation.
> (2) Through two proof-of-concept investigations, we reveal that conditioning on ground-truth contextual attributes improves CLIP's zero-shot classification and mitigates reliance on spurious features. Moreover, CLIP has the ability to infer contextual attributes by itself.
D (3) Based on the observations. we propose PerceptionCLIP Given an image, as shown in Figure] 1 it first employs CL.IP to infer contextual attributes. Then, it uses CLIP to infer the class conditioned on the attributes by incorporating the deseriptions of the inferred attributes into the prompt. This two-step inference resembles the concept of chain-of-thoughts in language models.
D (4) We empirically demonstrate that PerceptionCLIP excels in both standard generalization and group robustness, exhibiting improved interpretability. For generalization, it consistently outperforms baselines that use simple templates and prompt ensembles on 11 datasets. For example, it provides near 5% accuracy gain the EuroSAT dataset. For group robustness, it reduces the gap between average accuracy and worst group accuracy by 19% on the Waterbirds dataset and 7% on the CelebA dataset with ViT-L/14, showing less reliance on spurious features.
2 RELATED WORK
Due to CLIP's s ability to understand finer -rer-gr visual concepts beyond classes, some work also leverages external knowledge to augment prompts. For example, [Menon
(Menon & Vondrick (2022): Pratt ct al. (022); Mao et al. (2022): et al. (203)) use large language models to generate class specifc descriptions, resulting in prompts like "a photo of hen, which has two legs". Novack et al. (003)
2

---

Published as a conference paper at ICLR 2024
use class hierarchies to generate sub-classes for each parent class and aggregate model predictions on all sub-classes to get a final prediction. [Udandarao et al. (2023) use class names to retrieve and maintain some auxiliary data to help downstream classification. In contrast, class-independent attributes (i.e., contextual attributes) such as background and orientation, whose comprehension by CLIP is not well-known. These attributes are also combinatorial, potentially covering more aspects of an image than class-specific attributes. Moreover, we can still leverage contextual attributes (e.g., gender, age) when class-specific attributes are hard to articulate, as in the hair-color classification tasks on CelebA. We defer more related work to Appendix A
method addresses
3 PRELIMINARIES
Notation. We use uppercase letters to denote random variables and lowercase letters to denote their realizations. For random variable Z, we use pz(z) to denote its probability mass or density function, and omit the subscript Z when the function's meaning can be inferred from the input notation z.
Pretraining of CLIP. CLIP is pretrained web-scale image-caption pairs, using contrastive loss to learn good image and text representations in a shared space, aiming to correctly associate images and their textual descriptions. The captions in the pretraining data (as shown in Table (9) typically describe not only the object's s class but also contextual attributes like color, style, and background.
Zero-shot classification. After pretraining. Radford et al. (2021) use universal prompt template, represented by an annotation function (B) "a photo of /class name of y/", " that takes the class index y as the input and outputs a text that only describes the class. For any image
the image
space A' and y in the class set V/ the CLIP model serves score function CLIP; ) x X' -> R via
CLIP (1;,))
(Ør(z), (T(a()))),
(I)
computing similarity score (within [-), between the image and text through inner products of their representations produced by image encoder Φi and the text encoder Ør. The subscript "I' in 'CLIP, indicates that only one textual template is used. Then, given image x, the method predicts the class ÿ E y as the one with the highest CLIP; score,
arg maxyey CLIP; (y;
al. (2021) propose prompt ensembling, which ensembles 80 manually-designee templates {a,, 2 such as 'a bad photo of a (class name of y/' and 'a sculpture of a (class name of y/' and replace CLIP; with the following cLIPsu score for inference. Prompt ensembling involves some contextual attributes in the templates, but it is ad-hoc and lacks systematic analysis.
In addition. Raadior Radford
80 Ør (a(()()))
(y & (r(())
i=l
(2)
Σ 1=
(r((a()))
80
4 PREPARING CLIP FOR PERCEPTION
4.1 STRUCTURING AND DESCRIBING CONTEXTUAL ATTRIBUTES
Contextual attributes as generative factors. We consider contextual attributes generative factors that contribute to the data generation pro- cess. Specifically,, let Y denote the underlying object class (e.g., dog) that takes values in the class set Vr. Let each Z, (1 < i < m) denote certain contextual attribute of the object (e.g.,
Attributes
Values
Descriptions
dog
"dog"
p
Y: animal type
p= 0.3
upright
"sprighe"
0.1
Z orientation
upside-down
"spsoonting"
rotated
normal bright
"brighe" "welkar"
p- 0.3
Zy, Illamination
p= 0.1
orientation) that takes values in the contextual attribute set Zi (c.g., (upright, upside-down, ro-
"sunay"
0.2
tated)) and is causally independent (Pearl (209) of the object class Y. Then, we consider an im- age X to be generated as + + )m
Figure 2: Illustration of contextual attributes, their symbolic discrete values, and the possible textual descriptions mapped by the annotation function.
Textual descriptions for contextual attributes. While CLIP requires semantic text, generative factors are often symbolized discrete values, thus creating gap. It is negligible for the objects' classes since class names descriptions with ambiguities. However, the textual descriptions
3

---

Published as a conference paper at ICLR 2024
CLIP (*)))
CLIP (y,, ArandomixX)
CLIP (y' Farrongi))
CUP score 100
28
vorcclT Fip
927 rotatin
clatiic
color insrrt
solarize
blur
grasscile
br ghr
frost
Figure 3: Evaluating CLIP scores on ImageNet with different transformations altering the contextual attributes. The attribute-aware CLIP score gives higher scores for correctly matched image-attribute pairs (green) while giving lower scores for mismatched pairs (grey) and random pairs (blue), confirm- ing CLIP's understanding of our contextual attribute descriptions. CLIP score measures the similarity between images and contextual attributes, while the original CLIP score (orange) is attribute- attribute-ggnostic.
of the contextual attributes are vague. Taking upright images an example, people may use terms like "upright," "upstanding." or no description since it is common direction. To bridge this gap and translate discrete values into re.llbreddbbe text, we introduce specific annotation function a : + (ttxt), which maps a symbolic discrete value in Z to a distribution over natural language textual descriptions. Figure illlstrates some examples. An ideal annotation function models people's preferences when captioning images. We form the final image description using the concatenation operation @ This operation results in a new description distribution (xy) (o(21) @a(22) (222)))... attributes' descriptions are concatenated together and separated by commas. For example, when y: 21, 22 represent "dog." " "upright," and "bright" respectively. the concatenation a(y) @a(2)) ) @a(2)) yields the description "a photo of a dog, upright, bright," " or "a photo of a dog, sumny," etc.
@... where
4.2 CONNECTING CONDITIONAL PROBABILITIES WITH CLIP SCORE
Attribute-aware CLIP score. Existing CLIP score is agnostic of contextual attributes and thus cannot approximate conditional probabilities that are aatribute-dependee Therefore, we define a new score function CLIP y x Z1
2m x X R:
or (a(y) @ a(a)) @ al2m))
CLIP 21,-- 2m; x)
()(x))
E or (a @ a @ @ alzm))]l
It takes contextual attributes 2,8 additional inputs, describes them internally alongside the class through the annotation function (2;)) , and calculates the similarity with the image in the embedding space. The expectation is taken over the randomness of the descriptions of contextual attributes. The defined CLIP score captures the contextual attributes and behaves like an energy function (LeCun 2006): it is high for correctly matched image-attribute pairs while low for mismatched ones. More formally, when (y" 2f..... ) are the ground fouund-truu -truth class and attributes that generate image xx whereas (y) ......y 2m are some arbitrary class and attributes.
CLIP =1 ...-.... > CLIP(Y. -m,...., Vy E V, E 2, ViSiSm
Figure 3 and 5 empirically verified this property (see Appendix E.T for details). Given the pretraining process, this observation is not surprising since it encourages high scores for correctly matched image-caption pairs where the caption describes not only the class but also the contextual attributes.
Approximating conditional probabilities. With the furegy-functioonlike CLIP score, approx- imate the conditional probabilities. Specifi- cally (in Table [ and Appendix D). we ap- proximate (1) the joint conditional probability ply, 21, 21,..., 2m |2:), which measures the likeli- hood of an object class and some contextual at- tributes occurring together given the image, requir- ing only exponentiation and normalization. Based on it, we derive the rest two using the law of to- tal probability. (2) the conditional probability pyy| = Zm, x), which measures the proba- bility of an object class given both the image and
Table 1: Conditional probabilities. x, and denote image, class, and contextual attributes. denotes (21, Fm) for simplicity.
Probubility
Approximation
*(v,)(>)
Σn (LPP(y.
(LIP(y.ea))
Σn y (LTP(s))
Σg (LIP(y.
(L:P(
Σ. Eg (LIP or (L1
4

---

Published as a conference paper at ICLR 2024
Table Classification accuracy (%) on ImageNet. We apply the left-side image transformations to alter the corresponding attributes. Different methods condition on different values of the contextual attributes. Conditioning on correct or self-inferred attribute values improves accuracy the most.
Accuracy
Contextual attribute
w/o
W/ random 2
w/ wrong
correct
w/ self-infer 2
vertical flip
51.17
52.02 (10.85)
$2.19 (11.02)
52.48 (11311)
52.54 (11)
90" rotation
57.02
58.38 (1133)
58.23 (f1.21)
58.75 (f1.73)
58.30 (11.28)
clastic-transform
48.66
(10.21)
48.75 (10.09)
48.89 (10,23)
49.00 (10.34)
color-invert
35.29
36.12 (10.83)
35.89 (10.60)
36.72 (f1.43)
36.80 (11.51)
solarize blur
49.79
49.74 (10.05)
50.20 (10.41)
50.49 (10.70)
50.54 (10.75)
38.86
39.65 (10.79)
39.21 (10.35)
39.92
39.80 (10.94)
grayscale
(10.16)
(10.03)
(10.47)
60.04 (10.53)
bright
60.81
62.04 (11.23)
60.94((0.11)
61.41 (10.60)
61.28 (10,47)
noise
14.16
14.88 (10.72)
14.75 (10.59)
15.66 (11,50)
15.68 (11.52)
snow
33.09
32.94 (10.15)
33,56 (10.47)
34,50 (1141)
34.33 (11244)
frost
31.08
31.91 (10.83)
31.76 (10.68)
32.63 (1155
32.81 (1173
fog
37,61
38.40 (10.79)
38.00 (10,39)
39.31 (11.70)
39.34 (T1.73)
jpeg
33.67
34.80 (11)3)
35.11 (1145)
35.39 (11.72)
35.47
(1180)
average
10.64
10.57
11.16
-1.17
the contextual attributes, which is our main inference objective. (3) the conditional probability ( 2m ()), measures the likelihood of some contextual attributes given the image and is used for inferring the contextual attributes. We provide two approximations, referred to as ClassAttr (left) and PureAttr (right). The textual description corresponding to CLIP z; x) in ClassAttr is "a photo of a fclass name of yl. [description of z)." while the description corresponding to CLIP(2; r) in PureAttr is aa photo of object, [description of z/" with word like "object" substituting all classes.
CONTEXTUAL ATTRIBUTES ARE HELPFUL AND INFERABLE
This section presents proof-of-conced experiments showing that emulating human perception through conditional inference contextual attributes improves zero-shot classification. Additionally, such improvement does not require ground-truth attributes, as CLIP itself can infer attributes reasonably.
5.1 CONDITIONING ON CONTEXTUAL ATTRIBUTES IS HELPFUL
We first evaluate if conditioning on the ground-truth contextual attributes improves the zero-shot classification accuracy. Given an image æ, the most likely class is ÿ arg max, p(y|æ,,", 2") with:
CLIP pr)
arg max plyle, 2*) arg max Σ, CLIP arg max CLIP(y,=";a))
where the second equality holds because Ey CLIP().2:E) a constant of y and exponential function is monotonic. Intuitively, we classify an image by considering the combinations of all possible classes with the ground-trute contextual attributes and identify the class that yields the highest CLIP score.
Conditioning on ground-truth contextual attributes improves classification accuracy. We compare the following four methods in zero-shot classification, where the last two are for ablation:
Conditioning
Calculation
Prompt example
No contextual attributes Ground-truth attribute values Wrong attrihute values Random attribute values
arg max,
CLIP: (yr x)
photo of a (class name yf.
arg max,
CLIP y 2" ==:²)
photo of (class name of yl. upside-down.
arg max,
CLIP U, Awoong x))
a photo of (class name of yl. upright.
arg max, CLIP (v, Frandomi x)
a photo of (class name of y), iaVosnODu7.
We evaluate these methods on ImageNet dataset. Similar to Figure 3 we alter easily observable and adjustable attributes such as orientation through image transformations (e.g., vertical flipping). These new attributes become part of the modified images' generation process, for which we have ground-truth annotations. Table shows that compared not using contextual attributes, conditioning
5

---

Published as a conference paper at ICLR 2024
cow feature
spa
Crignal
Care spurius sog mentation
"rest anc
between and Insers Irres"
283
50.
35.7
Crignal
Cars SpURION S08 meniatiin
"aea a house"
Ton woter"
28.5
35.1
Figure 4: Images of leopard and waterbird, core and spurious features, and Grad-CAM heatmaps using no, incorrect, and ground-trute truth contextual attributes (with text below images). The bar shows core vs. spurious ratio in the heatmap. Visualization shows that classification conditioned on correct contextual attributes enforces CLIP's focus on core features.
on ground-truth contextual attributes improves classification accuracy notably. As ablation study, conditioning on wrong or randomly generated contextual attributes does not yield similar benefits.
Conditioning ground-truth contextual attributes mitigates the reliance spurious features. Contextual attributes like background (e.g., grass) may exhibit spurious correlations with the class (e.g., dog). Classifiers relying these contextual attributes, also known spurious features, usually perform poorly. We investigate whether classification conditioned on the known spurious features can enforce CLIP's focus the object (i.e., core features). As shown in Figure we isolate the background from the core region and employ Grad-CAM (Selvaraju et al. 2017) to identify which region the model focuses during classification. Specifically, the gradients
pixels with respect to p(y" Iz, (x,=²), 2"), the likelihood of the correct class conditioned on the known background given the image, yields the saliency heatmap. Figure | and Iillustrate that CLIP may rely on spurious features, correct contextual attributes reduces such reliance and enforces the model focus on core features, resulting more interpretable and reasonable perception (see more results in Appendix E.3). Since image embedding captures both object and background, we suspect that specifying an image's background to CLIP minimizes background influence, potentially sharpening the focus object features for better image and text matching in the embedding space.
however, conditioning
5.2 CONTEXTUAL ATTRIBUTES ARE INFERABLE
The above results highlight the advantages leveraging CLIP's understanding of contextual attributes. However, manually annotating the attributes is impractical. We now investigate whether CLIP can infer contextual attributes. To infer 2, we calculate arg max, p(z[ær using one of the two approxima- tions in Table I where the ClassAttr option yields arg max, p(z[z) = arg max, Σ, (y.pa) and the PureArtr option yields arg max, p(z[2) = arg max_ CLIP
CLIP can infer contextual attributes. Different from the setting in Section 5.1 randomly apply transformations to only half of the images in ImageNet. Therefore, inferring each attribute a binary classification task with random guessing accuracy of 50%. Table 3shlws shows that the average accuracy is around 74% for both methods, indicating that CLIP can reasonably infer contextual attributes, with some attributes being easier infer than others. CLIP's understanding of contextual attributes may originate from the numerous captions during the pre-training stage. Moreover, inferring contextual attributes could be casier than determining the object class. Therefore, we may bootstrap CLIP's inference by conditioning on the contextual attributes inferred by itself which is verified in Table
Table 3: Inference accuracy (%) of two contextual attribute inference methods on ImageNet.
Attribute
vftip
rotation
clastic
invert
solarize
gray
bright
noise
snow
fog
jpeg
Avg
ClassAttr
76.30
68.65
72.03
78.67
74.67
62.91
84.67
56.98
66.00
86.56
82.39
89.11
66.66
74.28
PureAttr 77.31
66.01
60.00
80.61
88.79
39,26
74.26
58.94
67.16
86.56
78.23
93.95
68.71
73.83

---

Published as a conference paper at ICLR 2024
6 PerceptionCLIP: EMULATING HUMAN PERCEPTION
Building the above observations, we propose PerceptionCLIP, two-step zero-shot classifi- cation method for CLIP. It emulates the human perception process by first inferring the contextual attributes and then inferring the class conditioning on the contextual attributes. The pseudocode of PerceptionCLIP is outlined in Algorithm
Algorithm I: PerceptionCLIP
Require: class Y, contextual attributes {Zi,..., Zm) CLIP score (with annotation function o:),
temperature hyperparameter
Input image x
Output predicted class y
Step 1: infer the distribution of contextual attribute values
Ey CLIP (v.31.....m :2////
2m)2)))))))))))))) f Ey Σ Sm 2mmm)
or 2 Amm
Step 2: infer the class
ply 2, Z1,.... 2m)
Eg COLIP(y/ (0
i + arg max, p(ylz) = arg max, y E Ex.... sm plylar 21,...,
Zm zm) 2m 2m|z))
Step one: PerceptionCLIP estimates the distribution of contextual attributes given image. Rather than selecting the most probable attribute value, we estimate the entire distribution to accommodate introduce temperature hyperparameter
CLIP's inherent uncertainty. In addition, in the estimation. A temperature т greater than 1 smoothens CLIP's estimation, implying less trust in its predictions. The two-step nature also allows for other interventions, such truncating top predicted values (i.e., beam search), which we leave for future work.
intervene
Step two: PerceptionCLIP first approximates the class distribution conditioning on each possible contextual attributes' value. Then, it uses the estimated distribution of contextual attributes to calculate the weighted sum of these class distributions, marginalizing out the contextual attributes. Finally, it selects the most probable class y as the predicted output.
Simplifying into a single step. It can be seen from Algorithm I that setting the temperature and ignoring constant terms yields ý < arg max,, y Em,....m. CLIPTTy.1.....fen the two-step algorithm into single step. Intuitively, for each possible class, it sums the exponentiated CLIP scores calculated over each contextual attribute value, resulting in an aggregated score for the class. Then, it selects the class with the highest aggregated score.
essentially simplifying
Single-step vs. prompt ensembling. This single-step approach, as a special case of our method, coincides with the prompt ensembling method if we aggregate over some randomly selected attributes (as in 80 templates) instead of all contextual attribute combinations. This coincidence explains the effectiveness of prompt ensembling- it undergoes an implicit perception process. Nevertheless, our experiments show that constructing diverse and systematic prompts using our contextual attribute combinations superior to ad-hoe template selections in prompt ensembling.
Two-step vs. single-step. The one-step method is simpler to implement but lacks two key features. disallows human intervention when inferring contextual attributes. Our experiments indicate that CLIP does not always infer contextual attributes correctly, whereas human intervention can leverage our prior knowledge to adjust estimation. Second, the one-step method prevents us from knowing the inferred contextual attributes, which could have improved the interpretability of the results.
Constructing contextual attributes. The set of possible contextual attributes is at the core of Percepti inCII... We construct it with two approaches: 1) We manually construct essential attributes that may be generative factors in the image generation process, especially those causing spurious correlations. This is particularly effective when we know of the dataset. For instance, for the CelebA dataset, we consider gender, age, and race as the attributes. 2) We leverage the in-context learning of large language models for anttmeet construction (shown in Appendix E4
7

---

Published as a conference paper at ICLR 2024
Table Zero-shot classification accuracy five datasets using ViT-B/16. The best result in each column is highlighted in bold, while the next three highest values are underlined.
Attributes
ImageNet
ImageNetV2
ImageNet-R
ImageNet-A
ImageNet-Sketch
single template
66.72
60.85
73.99
47.80
46.16
80 templates
68.32
61.93
77.71
49.95
48.26
background
67.98
61.65
75.87
49.85
47.08
illumination
67.47
61.48
75.37
48.90
46.67
orientation
67.28
61.11
74.51
48.47
46.87
quality
68.18
76.23
quantity
67.64
61.46
75.37
50,04
46.59
single attribute
perspective
67.90
61.27
75,00
49.61
46.84
art medium
67.53
61.11
77.16
49,48
47.96
67.58
61.31
76.67
49.62
47.37
condition
68.39
61.69
75.74
49,54
47,41
color-scheme
66.89
60.70
74.47
48.14
47.03
tool
67.42
61.02
76.72
48.88
48.19
composition of top attributes
68.52
62.28
77.78
$0.88
48,46
composition of top attribures
63.80
62.22
78.14
51.15
48.92
composition of top attributes
68.71
62.32
78.38
51.39
49.10
Table 5: Classification accuracy of ViT-B/16 on different data domains with PerceptionCLIP.
CUB200
EuroSAT
Places365
Flowers102
Food101
Oxford Pets
simple template
56.07
51.44
38.93
67.73
88.24
88.25
domain template
56.32
54.99 94
38.93
70.99
88.72
89.04
2
57.08
59.23
40.92
72.86
89.19
90.38
7 EXPERIMENTS
7.1 ZERO-SHOT GENERALIZATION
Settings. We test Percepti PerceptionCLID InCLIP on ImageNet (Deng al. 2009) and its out-of-distribution datasets, including ImageNetV2 (Recht et al. 2019). ImageNet-R (Hendrycks et al.) (021a). ImageNet-A (Hendrycks
al. 2021b). and ImageNet-Sketor (Wang al.) 2019). We also test on different data domains (e.g.. satellite images), including CUB200 (Wah et al. 2011). Eu- roSAT (Helher et al. | 2019). Places365 (Zhou et al.) al.] 2017). Flowers (Nilsback & Zisserman 2008 Food101 (Bossard al. 2014), and Oxford Pets (Parkhi et al. (2012). For natural images, we compile a set of possible contextual attributes. Each attribute has multiple possible values, and each value has multiple possible descriptions with uniform possibilities to simulate the unknown distribution (details in Appendix E.4p. For the dataset in a specific domain, we use -ppaif- contextual attributes, for example, image source for EuroSAT, cuisine for Food101, species for Oxford Pets. We two-step method (ClassAl(r) with the temperature as hyperparameter (details in Appendix E.4).
Using single attribute. Table shows that compared to using the simple template "a photo of (elass name)." considering almost any single contextual attribute improves the accuracy, some even surpassing the use of 80 templates. We also observe that the most influential contextual attributes vary for different datasets, potentially attributable to different data generation processes. For example, all images in ImageNet-Sketch are sketches, making tool and crucial contextual attributes for image generation. This also indicates that perceptionCL.I works the best when the considered contextual attributes cover the generation process of the dataset.
Using multiple attributes. Talso presents the results considering multiple contextual attributes. PerceptionCLIP. using the two most effective attributes, can already outperform prompt ensem- bling using 80 templates across all datasets. As the number of attributes considered increases, the classification accuracy gradually improves. We also test our method on different domains data in 5 The domain templates provided in RRafford al. (202I) already describe the domain in text prompt (e.g., centered satellite photo of [class name?" ") where the domain is a known contextual attribute. As expected, specifying improves accuracy. PerceptionCLIP considers attributes and further improves zero-shot classification accuracy. For instance, by considering image source and condition for the Au dataset, PerceptionCLII achieves 5% gain in accuracy.
contextual
8

---

Published as a conference paper at ICLR 2024
Ablation studies in Appendix E.4 demonstrate that substituting contextual attributes with random strings markedly reduces performance, highlighting their critical role in our method's effectiveness.
Intervening in attributes inference. In Table 6 we evaluate the effectiveness of the intervention. We set temperature = 3 and consider the top four attributes. Results show that intervening in infer- ring contextual attributes achieves modest but consis- tent performance gains across datasets. In practice, we find that setting the temperature to 3 or usu- ally yields better performance, which also confirms that CLIP cannot perfectly infer contextual attributes.
Table 6: Intervening in inferring contextual attributes improves zero-shot classification.
Without intervention
With intervention
ClassAtrr
PureAttr
ImageNet
68.59%.
68.70%
68.72%
ImageNetV2
62.10%
62.31%
62.32%
ImageNet-R
78.12%
78.38%
78.27%
MetgeNett-A
51.17%
$1.39%
51.22%
ImageNet-Sketch
49.03%
49.10%
49.10%
One can also search for the best temperature with validation set when applicable.
7.2 GROUP ROBUSTNESS
Group robustness is a critical measure of model's ' bias. It measures the ability to perform consistently across different subgroups within dataset (Liu et al., 2021). We evaluate the group robustness of PerceptionCll through bird type classification on the Waterbirds dataset (Sagawa et al.) 2020) and hair color classification on the CelebA (Liu et al., 2015) dataset. In both datasets, each image has an underlying group attribute unknown to the model. These group attributes are background in Waterbirds and gender in CelebA. They both spuriously correlate with the class but do not causally determine the class. To evaluate the worst-group accuracy, we group images by their classes and attributes, then assess each group's 's accuracy following Sagawa et al. (2020). Table and 8 show that when the text prompts only describe the class and ignore contextual attributes (first row), such as "a photo of a (lundbiru/waaerbirrd]"" and "a photo of a celebrity with (dark hair/blond hair)," CLIP exhibits biased accuracy, with a significant discrepancy between average accuracy and the worst-grow group accuracy. This bias arises because CLIP overly relies on spurious features, such as associating images with a water background to the waterbird class, instead of focusing on the bird. As shown in Figure 4 conditioning on group attributes such as background helps reduce CLIP's reliance on spurious features, making the model less biased. Results in Table 7 and 8 also confirm that by considering background (with values in (on land, on water)) )) for Waterbird dataset, and gender (with values in (female, male)) ) for CelebA dataset, Percepti InCLIP reduces the accuracy gap in most cases. By incorporating more values (e.g., in forest) into the attribute background+ or considering more contextual attributes like age and race, the group robustness can be further improved.
Table 7: Average accuracy and worst group accuracy on the Waterbirds dataset.
RN50 Worst
VYT-B/32
VIT-B/16
VIT-L/14
Avg
Gap.
Avg
Worst
Gap
Worst
Gap
Avg
Worst
Gap
without 2
90.47
16.07
74.40
87.34
47.
40.06
87.34
26.79
60.56
90.55
44.64
45.91
2= background)
72.71
89.80
82.98
41.51
Z- [hackgrund
90.32
35.71
54.61
78.60
60.33
18.28
85.80
41.07
44.73
87.74
61.12
26.62
Table 8: Average accuracy and worst group accuracy on the CelebA dataset.
RN50
VIT-R/32
VIT-B/16
VIT-L/14
Avg
Worst
Gapl.
Avg
Worst
Gap
Avg
Worst
Gap
Asg
Worst
Gap
without 2
23.87
7.19
80.73
75.82
4.91
75.16
62.01
13.16
86.98
77.36
9.61
2= 2=(gender)
85.10
80.44
4.65
79.89
76.70
3.19
75.27
65.13
10.14
80.30
74.31
5.99
2- [gengeedr age)
87.71
84.98
2.74
82.82
78.06
4.76
75.81
65.52
10.29
82.26
79.06
3.21
Z=| Z=/gender age, race)
85.55
82.51
3.05
82.02
75.94
6.09
77.17
69.18
7.99
83.04
80.84
2.20
8 CONCLUSION
This paper proposes PerceptionCLPP zero-shot classification method for CLIP that emulates human visual perception. By doing classification conditioned on self-inferred contextual attributes, achieves improved generalization, less reliance spurious features, and improved group robustness. One limitation of our method its sensitivity to text descriptions. Although using a distribution of descriptions alleviates this sensitivity, it is an intrinsic problem of CLIP itself. Future work may overcome this limitation by using advanced language models. Another future direction is applying this technique pre-training and fine-tuning stages (see more in Appendix G

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
An. Zhu, Panaitescu-Liess and Huang are supported by National Science Foundation NSF-IIS- 2147276 FAI, DOD-ONR-Office of Naval Research under award number N00014-22-1-2333, DOD- AFOSR-Air Force Office of Scientific Research under award number FA9550-23-1-0048 DOD- PARPA-Defense Advanced Research Projects Agency Guaranteeing AI Robustness against Deception (GARD) HR001 12020007, Adobe, Capital One and JP Morgan faculty fellowships.
REFERENCES
Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. Evaluating clip: towards characterization of broader capabilities and downstream implications. arXiv preprint arXiv:2108.02815: 2021.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 111-1001mmmmi 101-mining discriminative compo- nents with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part VI 13, pp. 446-461. Springer, 2014.
Ching- Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.00070 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purposs, purpose lanion-language models with instruction tuning. (arXiv:2305.066) (06500), Jun 2023. URL httt:://taxii../////////// 06500 arXiv:2305.06500 [cs].
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309. 165888 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor Guilherme Turrisi da Costa, Cees G. M. Snoek, Georgios Tzimiropoulos, and Brais Martinez. Bayesian prompt learning for image-language model generalization, 2023.
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. arXiv preprint arXiv:2305. 20088, 2023.
Zhili Feng, Anna Bair, and J. Zico Kolter. Leveraging multiple descriptive features for robust few-shot image learning. ArXiv, (bbs/2307.04311 2023. URL https///c.pp..eennntccccccc...a
orc/CorpusID:225501112
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Dan Hendrycks and Thomas G Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. arXiv preprint Nw.Xiv18007.0111 2018.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1340-8349 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021b.
Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision- lasion- models. arXiv preprint arXiv:2204.03649, 2022.
10

---

Published as a conference paper at ICLR 2024
Neel Jain, Ping-yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, et al. Neftune: Noisy embeddings improve instruction finetuning. arXiv preprint arXiv:2310.05914 2023.
Eric Kandel, James H Schwartz, Thomas M Jessell, Steven Siegelbaum, A James Hudspeth, Sarah Mack, et al. Principles of neural science, Fifth Edition, volume 4. McGraw-hill New York, 2013.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo. Piotr Dollár, and Ross Girshick. Segment anything. (arXiv:2304.02643) Apr 2023. doi: (0048550/axxi.2234.000
URL
/ttD:://AX.Voorgg/bb//22000 02643 arXiv:2304.02644 [cs].
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In AAAI, volume 1, pp. 3, 2008.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Hanting Li, Hongjing Niu, Zhaoqing Zhu, and Feng Zhao. Cliper: A unified vision-language
framework for in-the-wild facial expression recognition. ArXiv, abs/2333.00199, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. (arXiv:330.125597) Jun 2023b. URL
org abs// 12597 arXiv:2301.1259) [cs].
Liunian Harold Li, Zi-Yi Dou, Nanyun Peng, and Kai-Wei Chang. Desco: Learning object recognition with rich language descriptions. (arXiv:2306. 14060). Jun 2023c. doi: 00.48550/rrXiv22306.40000
URL
23002//100b/333000000 14060 arXiv:2306. 14060 [cs].
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tun- ing. (arXiv:2304.0445) 088485), Apr 2023. URL arXiv:2304.08488 [es].
08485
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang, Xin Eric Wang, and Carl Vondrick. Doubly right object recognition: A why prompt for visual rationales. ArXiv, abs/2212.06202,, 2022.
Cristina Menghini, Andrew Delworth, and Stephen H. Bach. Enhancing clip with clip: Exploring pseudolabeling for limited-label prompt tuning. 20xXiv:2306.01669, Jun 2023. doi: 10.48550/ arXiv.2306.09 066... /ttp:
/20.e 01669 arXiv:2306.01 666 [es].
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183, 2022.
M. Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Mateusz Kozinski, Horst Possegger, Rogerio Feris, and Horst Bischof. Lafter: Label-free tuning of zero-shot classifier using language and unlabeled image collections, (arXiv:2305.1 (1E.I:22005.1887),, May 2023. URL httt::t//aaxxiv/ org. /twpg//bb///// 2305
18287 1rXiv:2305.1828" [cs].
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEFF, 2008.
Zachary Novack, S. Garg, Julian McAuley, and Zachary Chase Lipton. Chils: Zero-shot image classification with hierarchical label sets. ArXiv, abs/2302.02551, 2023.
11

---

Published as a conference paper at ICLR 2024
OpenAl. Gpt-4 technical report, 2023.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Judea Pearl. Causal inference in statistics: An overview. 2009.
Sarah Pratt, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. arXiv preprint arXiv:2209.03322 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8760 PMLR, 2021.
Vikram V. Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron B. Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. Geode: a geographically diverse evaluation dataset for object recognition. (arXiv:2301 022560), Apr 2023. doi: 10.4 10.485500arXiv.230 230.00260...... URL http:://axxivoorr//ab///303 02560 arXiv:2301.022. 1.256000006011 [cs].
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. (arXiv:2204.061255, Apr 2022. doi: 10.48550/ arXiv.2204.06125 URL http:///aaa.w....r/aaa//////// 06125 arXiv:2204.06125 [es].
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? International conference on machine learning, pp. 5389-54991 PMLR, 2019.
Robin Rombach, Andreas Blattmann. Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. pp. 10684-10695, 2022. thecvf. com/ wwtt////ttte//ecc/ /Cwpn//22c22222222 /tt.l /Rombach
URL https: org.aaaee..aaaaaaaaaaaaaaaaaaaar Modeeloo__MMM_____ High wew.oooootoooonmaeeeooooooooooee Synthesis With Laath__atett__iiiiteeee CVPR 20t__22222_ppp22_p2
Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts, 2023.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL https
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtred 400 million image-text pairs. arXiv preprint arXiv:21 11.222111..11 1.02114, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-baser local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 318-626, 2017.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in lasion-language models. arXiv preprint arXiv:2209.07511, 2022.
Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. Sus-x: Training-fres name-only transfer of vision-language models. (arXiv:22 166198)).. Jul 2023. doi: 10.4550/arXi..211111188 URL
wwwpg//b////o /211 16198 arXiv:221 1.t:2211.1661 1.16198 [cs].
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
12

---

Published as a conference paper at ICLR 2024
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems. 32, 2019.
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. ArXiv, abs/2210. 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1900, 2022.
Yu Yang, Besmira Nushi, Hamid Palangi, and Baharan Mirzasoleiman. Mitigating spurious correla- tions in multi-modal models during fine-tuning. ArXiv, abs/2304.03916, 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint urXiv:2305.10600 2023.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why rision-language models behave like bags-of-words, and what do about it? In The Eleventh International Conference Learning Representations, 2023. URL https::/
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEFE transactions pattern analysis and machine intelligence, 1452-1444,14 2017.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 168 (6816-16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-23) 2022b.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. arXiv preprint arXiv:2205.. 14865, 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- vision-language understanding with advanced large language models. (arXiv:2304.10592) Apr 2023. URL http:///axxia org/ /2344 10592 arXiv:2304.10592 [cs].
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A EXTENDED RELATED WORK
Descriptive prompts with external knowledge. Due to CLIP's ability to understand finer-grained visual concepts beyond classes (e.g., body parts and components), some work leverages external knowledge to augment prompts with additional visual concepts to improve CLIP's zero-shot classifi- cation. For example, |Meoon & Vondrick 2022); Pratt et al. (2022); Mao al. (2022); Feng al. (2023) use large language models (LLMs) like GPT-3 to generate class-specific descriptions for cach class and incorporate them into prompts, resulting in prompts like "a photo of hen, which has two legs". Novack et al. (2023) use class hierarchies (existing or by querying GPT-3) to generate sub-classes for each parent class and aggregate model predictions on all sub-classes to get a final prediction. [Udandarao et al. (2023) use class names to retrieve and maintain some auxiliary data to help downstream classification. In contrast, our method addresses caas-indeeendent attributes (i.e., contextual attributes) such as background and orientation, whose comprehension by CLIP is not well-known. These attributes are also combinatorial, potentially covering more aspects of an image than class-ppecific attributes. Moreover, we can still leverage contextual attributes (e.g., gender, age) when class-specific attributes are hard to articulate, as in the hair-color classification tasks on CelebA. We also find that specifying spurious contextual attributes reduces distractions from their spurious correlations.
Does CLIP truly understand descriptive prompts? Some work investigates a seemingly obvious question: do these descriptive prompts play a role in CLIP's prediction? Roth et al. (2023) show that replacing class-specific descriptions in prior work with random words even meaning characters can achieve similar performance, resembling the effect of noise augmentation or randomized smooth- ing. et al. (2023c) find that GLIP (a similar VLM CLIP), often disregards contextual information in the prompts and relies heavily on class names in object detection. Addressing these findings, we ablate our method and show that random attributes or meaningless characters yield approximately half the benefit compared to using correct or self-inferred attributes, indicating that our method's effectiveness stems from the proper use of contextual attributes instead of noise augmentation. Roth et al. (202)) also show that appending high-level class- dass-independent descriptions (e.g.. "food" for Food101, "place" " for Places365) to prompts helps classification, which aligns with our findings.
Prompt tuning. Another line of work that modifies prompts to improve CLIP's classification is prompt tuning. which optimizes the prefix characters of the prompts. Typical prompt tuning methods
require labeled (Zhou et al., [022b Zhu et al., 2022 Derakhshani et al., 2023) or unlabeled
downstream data (Huang et al.] 2022 Mirza et al., [203] Menghini et al. 2023), making them
fall outside our scope of zero-shot (data-free) classificatinn. They are also prone to overfitting the training dataset, whereas our method relies on general image attributes (e.g, illumination) shared by common datasets. On the other hand, [hhu al. (2022) use test-time prompt tuning that applies to zero-shot classification. Specifically, they generate multiple views for each test image and optimize the prompt to minimize the entropy of the model's prediction on these views. This method introduces several hyperparameters that require tuning on labeled proxy validation set. In contrast, our method, depending on implementation, introduces either no additional hyperparameters or only one (temperature). Furthermore, method is training-free and can work in the black-box setting.
Reasoning and chain-of-thoughts.. The inference process of our method resembles the reasoning or -hain-of-thouuhts in prompting LLMs (Wei et al. 2022 Yao et al. 2023), where the model is prompted to give some intermediate step results and then conditioning on them to give final results. However, CLIP itself cannot do step-wise reasoning out of the box, so our method manually prompts it through the reasoning process.
B IMAGE CAPTION EXAMPLES
In the pertaining stage, the human- caption for each image typically describes the visual object, encompassing its class and a few contextual attributes. We show some caption examples in Table g chosen from similar dataset LAION-400M (Schuhmann et al.] (2021), since the original pretraining dataset of CLIP is not made public. We can see that those captions not only describe class but also contextual attributes like color, style, and background.
14

---

Published as a conference paper at ICLR 2024
Table Image caption examples from LAION-400M (comparable CLIP's pretraining dataset).
Caption #1
Men's 's Classics Round Bracelets Watch in Grey
Caption #2
stock photo of gremlins -3 d cartoon cute green gremlin monster JPG
Caption #3
Medium Size of Chair: Fabulous Mid Century Modern Chair Adalyn Accent In Red:
C
HUMAN VISUAL PERCEPTION
The classic neuroscience textbook Kandel et al. (003)) offers modern view of human visual perception, presenting significant difference from current zero-shot classification methods:
"The brain analyzes visual scene at three levels: low, intermediate, and high. At the lowest level, visual attributes such as local contrast, orientation, color, and movement are discriminated. The intermediate level involves analysis of the layout of scenes and of surface properties, parsing the visual image into surfaces and global contours, and distinguishing foreground from background. The highest level involves object recognition."
".. the perceptual interpretation we make of any visual object depends not just on the properties of the stimulus but also on its context, on other features the visual field."
This perception process is hierarchical, cascaded, and context-dependent differing from current zero-shot classification methods, which overlook contextual attributes. In this paper, we propose PerceptionCLIP to mimic human perception process.
D APPROXIMATING CONDITIONAL PROBABILITIES
With the energy-function-like CLIP score
(5r(#),
or (aly) @ a(x1) o .... @ a(mm)) [ Φr (a(y) + a
CLIP 21,...,, Zm:
(6)
@ a22m))ll
we first approximate the joint conditional probability ply, 21 ...... 3m (r. It measures the likelihood of an object class and some contextual attributes occurring together given the image as
CCLIP(y,
ply, 21. Zm (2:) En E. (m:
which is essentially the normalization of the exponential of the CLIP score. Then, derive the conditional probability p(y|21, 2m: x) which measures the probability an object class given both the image and the contextual attributes
ply, 21, . 2m (r)
p(y|,, 21,..,, 2m)
(8)
pl p(21,.... am x)
p(y 21, Zm (x) E, p(y: 21 21,..., 3m (m))
(1 (x)
(10)
Σn CaI.e((.2).....-.aaaa)
using the definition of joint probability and the rules of conditional probability. Next, we approximate the conditional probability p(21,..., 2m (2), which measures the likelihood of some contextual attributes given the image
p Fm (z)
Σ
p(u, 21 .., 2ml
(11)
Σ, c1
(12)
Σ, Ey CLIP(U.E1......
15

---

Published as a conference paper at ICLR 2024
It sums up the probabilities of contextual attributes appearing in each class to give total probability of them appearing in the image. We named this method ClassArtr. Another simplified way, named PureAttr, ignores the classes and use
CLIP (mmna
p(21.-.. Zm)z) x)) 2
(13)
CLIP Pmpra)
to do the estimation. Here, we only consider the contextual attributes in the CLIP score with descriptions like "a photo of an object. /description of where we use a word like "object" instead of a particular class, making the CLIP score class-agnostic. In our experiments, the first version occasionally outperformed the second, although the performance of the two is generally similar.
E EXPERIMENTAL DETAILS
Table 10: Summary of descriptions for different attributes used in Figure 3 Table 2 and Table 3 z" denotes the correct value of the contextual attribute, and Swrong denotes the wrong value of the contextual attribute. Ideally, each attribute has distribution of text descriptions. Here, use three descriptions and use the averaged text embeddings of them to calculate the CLIP score.
Attribute
(( @ xx)=")
a( cxf Zurrnn))
a photo of a [y]-
la photoo of a (g).
vertical flip
'a photo [st], upside-down."
"a photo a [3r], upright."
photo of [y]. the photo upside upside-down..
"a photo of [y). the photo is upright."
a photo of a [y)-
la photoo of a [y).
90 rotation
a photo [s]]. rotated."
photo a (81. upright."
photo of [y], photo rotated."
"a photo of [g]. the photo is upright."
a photo of [y]-
'a photo of a [g).
clastic-transfomm
'a photo of [y]] with distortion." "
"a photo a (8). normal,"
photo of [y]]_the the photo distorted."
"a photo of [2]. the photo is normal."
"a photo of [y]-
photoo of [g]."
color-invert
"a photo of [y]. color-inverted."
"a photo a (8). normal,"
"a photo of [y]. the photo color-inverted."
"a "aphoooo of [y). the photo is normal."
'a photo of [y]-
"a photo of [v])"
solarize
"a photo [y). solarized."
"a photo a (g). normal."
photo of [y]. the photo solarized."
"a photoo of [y). the photo is normal."
ta photo of [y]-"
"a photo of [9)).
blur
a photo [s/], blurred."
photo a [9r). clear."
photo of [y], photo blurred."
"a photo of [y]. the photo is clear."
a photo of [y]."
"a photoo of a [9)).
grayscale
a photo [5/]. grayscale."
photoo a (8). colorful."
photo of [y]]_he photo in black and white."
"a photo of [y], the photo is colorful."
aa photo of [y]."
"a photo of [sr)."
bright
'a photo of [y]] bright."
"a photo a (9). dark."
photo [y], photo bright."
"a photo of [2], the photo is dark."
1a photo of [y]-"
"a photo of [g]]"
noise
"a photo [y]. with noise."
"a photo a (8). clear."
'a photo [y]. photo has noise."
"a "appooto of [2], the photo is clean.
1a photo of [y]-"
"a "photo of [gr).
snow
"a photo a [y]. in the snow."
"a phpooo a [g). clear."
'a photo of [y]. the photo the
"a "a.photo of [2). the photo is clean."
*a photo of [y]-"
"a "aphoto of [9]]"
frost
*a photo [y]. in the fross."
"a photo a [g). clear."
a photo of [y]. the photo the frost."
"a pa.hoto of a [g]- the photo is clear."
a photo of [y]."
"a photoo of a [g]."
fog
*a photo [y]] in the fog."
"a photo a [g), clear."
photo of [y], the photo the fog."
photo of [2], the photo is clear."
a photo of [y]."
photo of [g)."
ipeg
*a photo a [y]]. in jpeg format."
"a photo of a [17), high resolution."
photo of [V], the photo ipeg format."
photo [2], the photo is high resolution."
E. DETAILS THE EVALUATION IN FIGURE AND TABLE 2
We do evaluation on the ImageNet dataset. Due to the lack of annotated contextual attributes, consider some easily observable and adjustable attributes, including image orientation, illumination, etc. We first examine and confirm that most ImageNet images share the same attribute values, including upright orientation, natural illumination, and standard image quality. However, these default values too trivial, making their textual descriptions unlikely to appear in the captions of the
16

---

Published as a conference paper at ICLR 2024
Table 11: Similarity score and classification accuracy on ImageNet test set. We apply a composition of two transformation functions on images, and use the composition of attributes' descriptions for text.
Similarity (CLIP score x 100)
Attributes
w/o 2
W/ random 2
w/ wrong 2
W/ correct 2
vertical flip color-invert grayscule tusntc-transform
25.39
28.23 (12.84)
26.28 (10.88)
30.26 (14,86) 32.15 (15.49)
26.66
30.55 (13.89)
26.48 (10.19)
Accuracy (%)
Attributes
w/o
w/ random =
w/ wrong =
w/ correct
vertical flip- color-invert
19.44
20.88 (11.44
20.01 (10.57)
21.32 (f188
grayscale crastic-transfom
29.79
30.49 (10.70)
30.14 (10.35)
30.59 (10.80)
pretraining data. Therefore, we then alter these attribute values through certain image transformations (e.g., vertical flipping), thus making the new attribute values have non-trivial descriptions. These new attribute values become part of the modified images' data generation process, for which we have ground-truth annotations.
Contextual attributes and their descriptions. We separately apply thirteen image transformation functions (e.g., vertical flip) to all the ImageNet test images. Note that the last five transformations (i.e., noise, snow, frost, fog, jpeg) are tested directly on the ImageNet-C dataset (Hendrycks & Dietterich, 2018), which contains the same images as in the ImageNet test set while images are corrupted with certain transformations. We use relatively strong strengths in those transformations, ensuring nonnegligible generative factors. This is also why CLIP has degraded performance on these corrupted data. Table [0] shows the descriptions contextual attribute, we use simple template, "a photo of a (class name].". When considering the contextual attribute,
used in this evaluation. When we ignore the
test the cases using the correct attribute value (e.g., upside-down) with "a photo of a (class name). upside-downr and the wrong attribute value (e.g.. upright) with "a photo of a fclass name], upright.", ", respectively. For each contextual value, the distribution of descriptions and average their embeddings as the text embedding to calculate the CLIP score.
three descriptions simulate
Randomized descriptions. To validate the effectiveness of contextual attributes, we also compare with the cases where we use random descriptions proposed in Roth et al. (2023). We replace every word in ax²²) with random characters while keeping the word length unchanged. For example, aty) + ((zandom of vertical flip contains three deseriptions: "a photo of a fyJ.", "a photo of a fy). ia ionoSn"D.,,."" "a photo of a fyl. 8g2, Mestx, ql, 6Ud2ole941r.
Additional results on the composition of contextual attributes. We perform the same evaluation while considering more than one contextual attribute. Table TT shows the results. We draw the same conclusion that correct contextual attribute values lead to better image-text alignment and higher classification accuracy.
Ablation the increased similarities. The CLIP score measures the similarity between the image and the text prompt. In Figure 3 we observe that correctly matched image and attribute pairs provide higher CLIP scores for the ground-truth class than mismatched pairs using wrong or random contextual attributes. It indicates that incorporating correct contextual attributes yields the greatest benefit for the ground-trute class. Additionally, we delve into the impact of including these correct attributes both the correct and incorrect classes. We calculate the increase in CLIP scores for class y with Py)) A CLIP(y, = =": CLIP, (y; r), and the increase in prediction probabilities for class y with Ap(y) A p(y| 2") p(y(z).
In Figure 5 (left), we compare the ACLIP()*) and ACLIP(Jweron)) where the latter one is the average increase of the Top-K K wrong classes. As expected, incorporating ground-truth attributes into text prompts results in increased scores for both correct and incorrect classes, and the correct class benefits more from this enhancement, as the accurate description of the class and the attribute, achieves a better alignment with the corresponding image. Figure 3 3 together validate that the
17

---

Published as a conference paper at ICLR 2024
Increase CLIP score x
Increase in Prediction Probability (%)
Top- Mernno
2.5
Top Yarong
Top 10 Yerre
Top Neroup
1.5
0.5
8
colatio
iisera
Figure 5: The increase in (left) CLIP scores and the (right) prediction probabilities by incorporating the descriptions of the correct contextual attribute into the text prompts. We compare the increased CLIP scores and prediction probabilities for the ground-truth class y" the Top-5 and Top- wrong classes. (Ieft) Incorporating ground-truth attributes into text prompts results in increased CLIP scores for both correct and incorrect classes. This improvement is attributed to the enhanced alignment of the text prompts with the images, addressing previously overlooked contextual attributes. Notably, the CLIP score of the correct class benefits more from this enhancement for all the attributes considered. This is because the accurate description of the class, combined with the contextual attributes, achieves more precise alignment with the corresponding image. (right) Therefore, the model is more likely to predict the correct class after being provided with the correct context description in the prompt.
CLIP model understands the contextual attributes, and describing correct class and attributes yields higher similarity scores as described in Equation
In Figure 5 (right), we further compare the increase in the prediction probabilities for Ap(y) and Ap(ywrong) where the probability is calculated by applying softmax on CLIP scores and is used for the final classification. By incorporating correct context, the prediction probability of the correct class increased significantly, while the wrong classes got unchanged or even decreased probabilities. The predicted probability for the correct class increases by an average of 1.5%. The predicted probabilities for top-5 and t0p-10 wrong classes decrease by an average of 0.07% and 0.05%. Such findings also explain the increased accuracy in Table 2 when incorporating the correct contextual attributes.
E.2 DETAILS ON THE EVALUATION IN TABLE 3
In Table we test whether CLIP can infer underlying contextual attributes by itself. In this experiment, we only apply transformations to half of the images from the ImageNet test set and use descriptions shown in Table 10 The task is to predict the correct contextual attribute's s value, which is a binary classification task. For example, half images are upright, and half images are upside-down, and the goal is to classify the orientation of the images by CLIP. We evaluate two methods with two approximations of p(2[2) in Table I Note that we do not intervene in the inference in this experiment.
E.3 DETAILS ON THE VISUALIZATION
We consider some spatially separable spurious attributes (also known as spurious features), such as backgrounds, and annotate core regions and spurious regions with the help of Segment Anything (Krillovv et al., (023). Then, we use Grad-CAM to generate a heatmap for salient pixels. first, we a function that computes CLIP's similarity score for each class CLIP(y, 2r) and apply softmax on top of these values. Then, we only consider the scalar value corresponding to the ground-trutt class which is essentially the conditional probability p(v" |x. 2"). We compute the gradients using the layer before the final attention block of ViT-L/14 as suggested in a popular explainability library] Intuitively, the regions where the salient pixels are located are the regions the model pays attention
18

---

Published as a conference paper at ICLR 2024
Table 13: Summary of contextual attributes and their value descriptions used in ImageNet-related datasets.
Attributes
Values
orientation
upright, upside-down, rotated others, natural, urban, indoor
background
quality
normal, good. bad, low res, pixelated, jpeg corrupted, blurry, clean, dirty
ilhomination quantity perspective art
normal, bright, dark
others, many, one, large, small normal, close up, cropped, obscured non-art, others, sculpture, rendering, graffiti, tattoo, embroidery, paper art, sketch, cartoon others, video game, plastic, toy normal, cool, nice, weird normal, black and white others, pencil, pen, digital tool
medium condition
color-scheme tool
to when making predictions, and we hope that the model focuses as much as possible on regions of core features (i.e., features with causal correlation classes). Note that, adding spurious attribute's description in this evaluation won't I really make the model look at it when classifying because all the descriptions (for all classes) will contain that attribute.
To compute the ratio between the usage of core and spurious regions in prediction, we: (1) run Segment Anything (Kirillov al. 2023) and select segmentation mask for the core region of each image (e.g.. the bird or the leopard), then consider the rest (non-core) regions as spurious; (2) use the masks to identify the core and spurious pixel values from the Grad- CAMs and compute the mean pixel value for both of these regions; (3) normalize the numbers and show them as two percentages in a bar plot for each Grad-CAM.
In Table T2, we quantitatively evaluate the model reliance on core feature and spurious feature. We use ViT-B/32 as the image encoder and evaluate it on the Waterbirds test set. The dataset contains images land birds and water birds on land or the water. We use the same method introduced above to calculate the ratio of core versus spurious through Grad-CAM We use "on land" and "on water" to describe the context (e.g., background). We compare the correct context with context, wrong context, and random context where the random context is the random string that replaces the correct context while keeping the description length unchanged. Results in Table 12 also indicate that, by incorporating the correct context, the model relies more on the core feature when doing the classification.
Table 12:
The average saliency (%) of the core feature and the spurious feature evaluated on the Waterbirds test set.
Core (1)
Spurious (L)
context
62.8
37.2
wrong context
62.6
37.4
random context
62.3
37.7
correct context
66.3
33.7
E.4 DETAILS ON THE EXPERIMENTS IN SECTION 7
we test Perceptionooe inCIip on ImageNet and its OOD datasets. We first use GPT-4 to summarize the contextual attributes involved in the 80 hand-crafted templates (Radford et al. 2021 then add three contextual attributes (orientation, background, tool) to the testing bed. Table 13 shows the values of every contextual attribute. We use multiple descriptions to describe every attribute value, and use their average text embedding of the full sentence in the implementation. When considering single attribute, we use main template, "a photo of (class name]" and concatenate it with the description of each attribute value. When considering the composition of attributes, generate combinations from the values across all attributes. Such simple concatenation of descriptions works well, probably because the pre-trained CLIP model behaves like bag-of-wordh (Yuksekgonul et al. 2023). Future works could explore better ways of composing text prompts.
In Table 4
Table 14 and 15 list the contextual attributes used in Table and Attributes and their values are manually designed based on our priors of datasets. Experiments on Waterbirds and CclebA are conducted on their training set.
19

---

Published as a conference paper at ICLR 2024
Table 14: Datasets, domain templates and contextual attributes used in Table 5
Dataset
Domain Template
Attributes
CUB200
"a photo of (y). type of bird"
size, buckground, condition condition, source background, quality, condition background, illumination, quality, condition cuisines, condition
EuroSAT
"a centered satellite photo of [w]"
Places365
"a photo of {z)"
Flowers102
"a photo of a {2/) type of flower"
Food101
"a photo of a ). type of food"
Oxford Pets
"a photo of (y). type of pet"
species, background, pose, interaction
Table 15: Domain templates, contextual attributes and their descriptions used in Table and Table
Dataset
Domain Template
Attributes
Values
Waterbirds
"a photo [y}"
buckground
on land, on water
background*
+ forest, in sky, on street, on grass, on tree, with flowers, on heach, with human, on a branch
CelebA
"a photo of a celebrity with (v)"
gender age
female, male young. old
race
white skin, dark skin, asian
All the above experiments use ClassAttr version PerceptionCLie and the intervention by setting temperature in the first step (i.e., inferring contextual attributes). We found that mildly smoothing the estimation by setting to be or usually has the best performance. When we do not have a good prior of the temperature, just setting it to can also have relatively good results. The reported numbers in our experiments use temperature selected from [1,3,5100] that performs the best on the particular dataset.
Ablation studies. In Table and by incorporating contextual attributes, PerceptionCLII improves the zero-shot classífication accuracy in all cases. Adding descriptions of contextual attributes to text prompts has two effects: 1) it introduces more tokens to the text prompt, 2) and tokens describe the contextual attributes. To figure out which effect causes the improvement, we conduct ablation studies by replacing the descriptions of contextual attributes with the same-length random strings. In Table [6 we do ablation studies on the best attribute composition for all datasets. In Table 17 we keep the domain template but randomize other contextual attributes. For every dataset, we run times with random seeds and report the mean and variance. Results show that adding more tokens can improve the accuracy marginally in most cases, but also decrease the accuracy in the case of EuroSAT and Oxford Pets. The improvement brought by adding random tokens might be result of augmentation (Jain et al., (023) or register (Darcet et al. (023). More importantly, there is a significant performance gap between using the random strings and the descriptions of contextual attributes, suggesting that the improvement provided by our method primarily stems from the incorporation of contextual attributes.
Table 16: Ablation study ImageNet and related datasets.
Attributes
ImageNet
ImageNetV2
ImageNet-R
ImageNet-A
ImageNet-Sketch
CLIP
66.72
60.85
73.99
47.80
46.16
PerceptionCLIP
68.80
62.32
78.38
51.39
49.10
ablation w/ random 67.59 + 0.27
61.27 + 0.11
75.53 + 0.28
49.74 + 0.37
47.63 + 0.23
F ADDITIONAL RESULTS AND ANALYSIS
F.1 COMPUTATIONAL COMPLEXITY.
Similar to the implementation of prompt ensembling, we pre-compute the embeddings of all class and contextual attribute combinations, and then use these pre-computed embeddings in each inference process. Since the average of text embeddings when there multiple descriptions for
20

---

Published as a conference paper at ICLR 2024
Table 17: Ablation study different data domains.
CUB200
EuroSAT
Places365
Flowers102
Food101
Oxford Pets
domain template
56.32
54.94
38.93
70.99
88.72
89.14
72.86
89.19
90.38
random
56.68
0.17
53.58
2.34
39.98
0.37
71.41 + 0.45
88.89
0.08
88.35 0.45
Table 18: Performance PerceptionCLIP using two order types in the attribute concatenation.
Order
ImageNet
ImageNetV2
ImageNet-R
ImiggeNet-A
ImageNet-Skecie Skeethh
forward
68,71
62.32
78.25
51.39
48.97
49.10
one value, our method needs multiple forward passes to get the text embeddings, causing longer preparation time. Since these computations are one-time, the time complexity during inference is unaffected by the number of contextual attributes. Compared to the basic method, which stores O(D1)) embedding vectors, this implementation needs store (()) x | x.. vectors. The space complexity limits the number of contextual attributes considered in practice. We will consider using beam search to only reserve top-k attributes to reduce the space storage requirement in future work.
...... x (2ml) embedding
F.2 AN ANALYSIS OF THE ORDER IN ATTRIBUTE COMBINATION
When considering multiple contextual attributes, we concatenate their textual descriptions. An interesting question is whether their order in the text affects the performance. In Table [8] we test two order types when combining the top four attributes in the ImageNet experiments. The forward direction ordering attributes from the most powerful to the last. The backward direction does the opposite ordering. Unfortunately, we do not observe a good way of ordering consistently outperforming others. We suspect that it is due to CLIP's sensitivity the text, and the direct concatenation may not be the best way of combining attributes to approximate the distributions of captions in the pertaining stage.
F.3 MORE VISCALIZATIONS
We show more visualizations in Figure and Figure shows images from the ImageNet dataset with the ground-truth class leopard. Figure 7 shows images from the Waterbirds dataset with the ground-truth class waterbird. Grad-CAMs show that CLIP relies more on core features when conditioned on the correct contextual attributes (e.g., background) for classification. The reliance on core features also improves model interpretability.
21

---

Published as a conference paper at ICLR 2024
com feature
spurious festure
Original
Core spurious segmentation
"next car"
between leaves and trees"
58.4
40.6
40.1
50.8
75.7
24.3
Original
spurious segmentation
house"
"next sea"
Criginal
Core spurious segmentation
"new water?
"neer rocks"
783
42
57.4
Crigina
Core spurius sogmentation
Tnext to cer"
Twith trees and grass sround"
Figure 6: Leopard images from ImageNet dataset. Visualization of the original image, the regions of core and spurious features, and the Grad-CAMs obtained using no, incorrect, and ground-truth contextual attributes.
22

---

Published as a conference paper at ICLR 2024
spu apunous feature
Original
Core-spurioss segmentation
"near house"
"on the water"
59
Original
Core-spe sere-spurious segmentation
"near house"
'near ccean"
Core segmentation
Original
Core-spurious segmentation
house"
54.1
66.8
Figure 7: Waterbird images from Waterbirds dataset. Visualization the original image, the regions of core and spurious features, and the Grad-CAMs obtained using no, incorrect, and ground-truth contextual attributes.
23

---

Published as a conference paper at ICLR 2024
F.4 DISCOVERING CONTEXTUAL ATTRIBUTES BY LLMS
In this section, we provide an example of how to use GPT-4 O 2023 to generate contextual attributes and their descriptions automatically. We take EuroSAT dataset as example. There three steps:
1. Given a dataset (e.g., EuroSAT) with a specific domain, we retrieve similar images (e.g.. satellite images) from large image+text dataset LAION-400M
2. We crawl the captions and randomly sample limited number of these captions (e.g., 200).
3. We provide GPT-4 with the captions and the information of the dataset, and ask it to extract contextual attributes using Prompt 1.
Table 19 shows the contextual attributes discovered by GPT from captions. Adding those attributes to the domain template, we improve the accuracy from 51.4 11.44% to 59.20% (with intervention T 5), which is comparable to manually designed ones. However, we found that the attributes identified by GPT are not always appropriate, possibly because of the gap between the retrieved images and our dataset. Future work could involve using image-based searches to find more similar images rather than relying language-bel based searches.
Prompt I: An example prompt for discovering contextual attributes and their descriptions from description for every attribute value for
example captions for EuroSAT dataset. Here simplicity.
You are a helpful assistant who helps me summarize. my text captions. have dataset of image-caption pairs, where each caption briefly describes the image. I need to extract sone attributes from these textual descriptions that contribute to the data generation process the image.
For examplc, from the three deseriptions ["A black dog on the green grass "A red car the road in a bright "nvinnne "A white
refrigerator"), you can summarize the "background", "color", $ilumination" as three attributes, with possible values ["grass field ", "road", "], ["black", "green", "red", "white", "], ["bright", "dark", " "IJ respectively. Note that they each have an empty value because the human annotators may choose not mention them in the captions.
Note:
1. The number of potential values for each factor should not exceed 3, you should extract the most representative values.
2. You should summarize at most 5 attributes, covering the most
representative attributes in the provided captions.
3. have list of labels for these images, namely [' l'annual crop land', 'forest', 'brushland or shrubland', 'highway buildings er commercial buildings', 'pasture Land', 'permanent erop land', "residenia" bui Idingg or homes or apartments', 'river', (1ake or sea', The ittributes you summarized should not overlap with the concepts of these labels, and the values you summarized should not include any these Labels. For example, since "river" is in my label set, your summarized values should not include "river" for any attributes.
road', "industrial
4. The set of all values for all attributes summarized should not overlap.
need your summary to have the following format:
summerized zed factors seattrrrrrddfatorssssssss "background"; ""
grass",
24

---

Published as a conference paper at ICLR 2024
"road",
1,
"color": [ "black", "green", "red", "white",
"i "illumination": "bright", "dark",
Hore are the captions:
//200 captiens
Table 19: Contextual attributes and their value descriptions for EuroSAT generated by GPT-4.
Attributes
Value Descriptions
source
"Yandex satellite", "NASA", "Google Maps"
geographical feature image rype
"island", "ul.". "street" "satellite", "acrial", "map"
natural phenomenon
"hurricane", "carthquake", "deforestation"
structure type
"residential", "commercial", "fortress"
G IMPACT, LIMITATION AND FUTURE WORK
In this paper, we propose PerceptionCLIP. zero-shot classification method for CLIP that emulates the human visual perception. By doing class inference conditioned on self-inferred contextual attributes, it achieves improved generalization, less reliance on spurious features, and improved interpretability. Along the path of proposing PerceptionCLII,, we show CLIP's understanding of object attributes beyond common category features. Our work indicates that CLIP, as model capable of communicating with humans via natural language, can achieve things that traditional classifiers find challenging. Hence, it still has great potential in zero-shot classification and even broader tasks like image generation. Furthermore, this capability complements the study of neuroscience, enabling better transition of the latter's 's research findings into practical use.
Limitations. One limitation of PerceptionCLIP is its sensitivity to text description perturbations: using different synonyms describe the same attribute sometimes has non-trivial effects the results. Although using more descriptions to describe an attribute value (Figure 2 alleviates this sensitivity, this issue more intrinsic CLIP and still persists. Future work may overcome this limitation by replacing CLIP with other vision-language models or improving CLIP's sensitivity textual perturbations (e.g., through training-time limitation of perceptionCIIP is the need to design set of contextual attributes. While this process provides way to integrate human prior knowledge. also requires additional effort, especially when we aim to cover many attributes. Currently, we use caption retrieval from the LAION-400M dataset and the in-context learning ability of large language models to remi-automate the construction process. In the future, our goal is to automate this process fully. In paper, show that directly concatenating multiple attributes' descriptions is simple and effective way to generate the image's description. Future work explore more effective and efficient approaches for it.
augmentation (Fan al. 2023 Another
Ethical Statement. this paper, we use the CelebA dataset, employing descriptors of gender and race to enhance classification accuracy. We acknowledge the sensitivity of these attributes in societal and ethical contexts. Our use of gender and race strictly as example contextual attributes within analytical framework, and not endorsements of any form of racial or gender-based bias. The
25

---

Published as a conference paper at ICLR 2024
inclusion of these attributes is solely for the purpose of exploring and improving the performance of zero-shot classification, without attributing any significance beyond their technical utility. We are committed to maintaining ethical principles in our research and upholding respect and diversity in all aspects of our work.
26


Reviews:
[{"review_id": "LvPzGT6njs", "reviewer": "Reviewer_k2xX", "summary": "This paper is inspired by human visual perception, where humans first discern contextual attributes, such as background and orientation, to distinguish objects, and then classify them. Similarly, when CLIP is provided with these contextual attributes, it improves in zero-shot image classification and reduces dependence on irrelevant features. Authors found that CLIP can deduce these attributes from an image itself and based on this fact to propose PerceptionCLIP. PerceptionCLIP first determines contextual attributes from an image and then classifies the object based on these attributes. Experiments are done on CLIP's zero-shot classification settings and show clear improvements over the original CLIP.", "questions": "Post-rebuttal:\n\nI genuinely appreciate your great efforts put into this rebuttal!\n\nI read the (updated) paper one more time, the authors' responses to me thoroughly, and the responses to other reviewers.\nWhile PercentionCLIP is indeed powerful and but answer of \"why does it work\" is not fully addressed via the GradCAM visualization as raised by W2 of reviewer vx4m. \n\nI would like to keep my rating for now and may change later after discussing with other reviewers.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy-to-follow. While the concept of utilizing background information for image classification isn't groundbreaking in literature, its application to CLIP could be innovative.\n- The experiments show clear advantages of using contextual attributes over the traditional 80 templates.", "weaknesses": "- The authors assert at least twice that PerceptionCLIP mirrors human perception. However, I'm not entirely convinced. Authors gave the preliminary that: “humans first infer contextual attributes (e.g., background and orientation) which help separate the foreground object from the background, and then classify the object based on this information.” Yet, there's no evidence indicating that PerceptionCLIP actively separate foreground from background during classification, or that such separation is utilized the model. It's possible that PerceptionCLIP utilizes background attributes differently.\n- The authors refer to background information as spurious features (e.g. Figure 1). To my knowledge, it is not completely correct. Though they can sometimes overlap, they are not the same. Background information is a broader concept, while spurious features specifically refer to misleading patterns that a model might incorrectly learn as being important. In addition, the GradCAM in Figure 1 primarily emphasizes the foreground, consistent with [a], without highlighting any reduced reliance on spurious features. It's more accurate to state that it offers enhanced focus on foreground objects.\n- When I like the idea of Textual descriptions for contextual attributes Sec 4.1., I could not find how exactly you map Z using the proposed annotation function alpha to attribute text descriptions. Also, why do you say this annotation function model human preferences in captioning? Authors may also want to clarify the p value associated with the textual descriptions.  I imagine that these descriptions can also easily obtained using LLMs (e.g. ChatGPT).\n- The name of the proposed metric can be easily confused with the original CLIP score. How about naming it as Attribute-CLIP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PSX3OLIumI", "reviewer": "Reviewer_U13E", "summary": "Inspired by the human perception process that the contextual attributes are separated from the foreground objects, this paper proposes a training-free, two-step zero-shot classification method PerceptionCLIP to first infer the contextual attributes (e.g., background) and then performs object clas- sification conditioning on them. A proof-of-concept investigations reveal that conditioning on ground-truth contextual attributes improves CLIP’s zero-shot classification. The proposed PerceptionCLIP demonstrates performance gain and improved interpretability on several datasets.", "questions": "In Table 7, why lower gap between the Avg and Worst is better?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The idea of imitating human perception process to improve the generalization and group robustness of the image classification model is insightful for the community.\nThe proposed method is extensively evaluated on 11 datasets.", "weaknesses": "1. Collecting the contextual attributes requires either pre-knowledge for the test image or a large dataset containing captions, which hinders the generalization ability of the proposed method in the real world. For instance, contextual attributes for the CelebA dataset are manually defined, e.g., gender, age, etc. To collect the contextual attributes for the remote sensing dataset EuroSAT, the authors first retrieve similar images and captions from a large image+text dataset LAION-400M, then ask GPT-4 to summarize the contextual attributes. What if we do not have external datasets to provide captions?\n\n2. The qualitative results in Figure 4 indicate that introducing the contextual attributes reduces reliance on the spurious features and the model focuses more on the core features. It would be fairer to provide a quantitative evaluation, e.g., counting the percentage of model attention on the core features versus on spurious feature on all test set in ImageNet, and compare the ratio of different models.\n\n3. The performance gain seems marginal on most of the datasets. For instance, in Table 4, the performance gain is only around 2% on ImageNet, ImageNetV2, ImageNet-A and ImageNet-Sketch. Besides, since introducing a random attribute or even a wrong attribute can improve the accuracy in Table 2, it would be interesting to include the results of the wrong attribute and random attribute in Table 4 as well.\n\n4. The results in Table 7 are not consistent among different backbones. It is hard to get any conclusion on which method is better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AXlFUFtCHU", "reviewer": "Reviewer_vx4m", "summary": "The paper explores text-prompting in CLIP, to make the inference process more ``human-like”. The authors show that their two-phased prompting process improves (i) performance, and (ii) robustness against specious features (shortcuts).", "questions": "Please refer to weaknesses above", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- In this work, prompting is class-independent; which makes it easily applicable to numerous datasets. \n- Authors systematically evaluate the prowess of CLIP in inferring contextual attributes\n- The performance gain on domain-specific and out-of-distribution ImageNet datasets shows promise in the claims and approach of the authors\n- This work allows for building domain-specific (yet, class independent) augmentations with the possibility of human-in-loop intervention\n- The approach presented is elegant and interpretable", "weaknesses": "- If I understand correctly, Figure 3 gives only the score (x100) of the correct class in different scenarios. Is this *completely* informative? I think it can be easily misleading. What if the model provides relatively higher scores to some of the wrong classes as well? Can the authors analyse the score distribution of the wrong classes? Reporting the mean of CLIP_score@topK might be a good start to understanding the false positives.\n- I appreciate the visualisation study provided by GradCAM as a qualitative analysis but I am not confident of the calculation of “core” versus “spurious” features [1]. \n- Can the authors also report the random accuracy for Tables 4 and 5? It is important to have a random baseline (that is, random string in place of the inferred context having the same token (and not string) length) here to isolate the effect of “registers” versus actually using the context [1]. \n- The authors have not provided the code for reproducing the paper; implementation details are missing.\n\n**TL;DR**: The authors make strong claims about reducing the reliance on shortcuts, however, the missing baselines and analyses do not make me confident of their approach. However, some analyses seem misrepresented/miscommunicated. If the authors can answer my questions, I’d be open to changing the score.\n\nSome clarity formatting and typographical errors to rectify:\n\n- Overall, the paper is well-written and ideas well-presented\n- The paper, at some points, deviates from standard ICLR formatting:\n  - “interleaved” figures and tables (minor inconvenience)\n  - Unlabelled table: “Conditioning on ground-truth contextual attributes improves classification accuracy”\n- Minor typographical errors:\n  - Section 5.1: “Intuitively, we classify an image using both – possible classes and the ground-truth contextual attributes”\n  - Remove the asterisk in the author list of “Distributionally robust neural networks“\n\n[1] Darcet, Timothée, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. \"Vision Transformers Need Registers.\" arXiv preprint arXiv:2309.16588 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lWNIekObYx", "reviewer": "Reviewer_VSta", "summary": "The authors show that adding text that describes the context of an object in an image can improve the performance of CLIP-based zero-shot classification. In addition, they show that said context can be inferred using CLIP itself.", "questions": "Some minor issues:\n- For some of the experiments I couldn’t find if they employed ClassAttr or PureAttr.\n- In Algo 1, I assume it should be “Set of classes Y” rather than “class Y”, and that the sum is over y \\in Y.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1- The paper shows some evidence towards towards the fact that CLIP representations do capture the context as well as the foreground objects, making the alignment with text prompts better when the appropriate context is included in the text.\n2- Although the improvements in performance are not large in many of the benchmarks, they come at little cost, probably making it applicable in practice.", "weaknesses": "I haven’t found any major weakness in this work (although it is not fully within my expertise).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)