Delving into the Openness of CLIP
Authors: Shuhuai Ren
Lei Li
Xuancheng Ren
Guangxiang Zhao
Xu Sun
Venue: Submitted to ICLR 2023 (2023)
Keywords: Contrastive Language-Image Pre-training
CLIP
Openness
Vision-and-Language

Abstract: Contrastive Language-Image Pre-training (CLIP) has demonstrated great potential in realizing open-vocabulary visual recognition in a matching style, due to its holistic use of natural language supervision that covers unconstrained real-world visual concepts. However, it is, in turn, also difficult to evaluate and analyze the openness of CLIP-like models, since they are in theory open to any vocabulary but the actual accuracy varies. To address the insufficiency of conventional studies on openness, we resort to an incremental perspective and define the extensibility, which essentially approximates the model's ability to deal with new visual concepts, by evaluating openness through vocabulary expansions. Our evaluation based on extensibility shows that CLIP-like models are hardly truly open and their performances degrade as the vocabulary expands to different degrees. Further analysis reveals that the over-estimation of openness is not because CLIP-like models fail to capture the general similarity of image and text features of novel visual concepts, but because of the confusion among competing text features, that is, they are not stable with respect to the vocabulary. In light of this, we propose to improve the openness of CLIP in feature space by enforcing the distinguishability of text features. Our method retrieves relevant texts from the pre-training corpus to enhance prompts for inference, which boosts the extensibility and stability of CLIP even without fine-tuning.

Under review as a conference paper at ICLR 2023
DELVING INTO THE OPENNESS OF CLIP
Anonymous authors Paper under double-blind review
ABSTRACT
Contrastive Language-Image Pre-training (CLIP) has demonstrated great potential in realizing open- vvea-vocabulary visual recognition in matching style, due to its holistic use of natural language supervision that covers unconstrained real-world visual concepts. However, it is, in turn, also difficult to evaluate and analyze the openness of CLIP-like models, since they are in theory open to any vocabulary but the actual accuracy varies. To address the insufficiency of conventional studies on openness, we resort to an incremental perspective and define the extensibility, which essentially approximates the model's 's ability to deal with new visual concepts, by evaluating openness through vocabulary expansions. Our evaluation based on extensibility shows that CLIP-like models are hardly truly open and their performances degrade as the vocabulary expands to different degrees. Further analysis reveals that the verressimation of openness not because CLIP-like models fail to capture the general similarity of image and text features of novel visual concepts, but because of the confusion among competing text features, that is, they are not stable with respect to the vocabulary. In light of this, we propose improve the openness of CLIP in feature space by enforcing the distinguishability of text features. Our method retrieves relevant texts from the pre-training corpus enhance prompts for inference, which boosts the extensibility and stability of CLIP even without fine-tuning.
INTRODUCTION
The seek for an intrinsically open mechanism of visual recognition (Dnng et al. 22009; He ct al. 2016) has always been a shared goal in the computer vision community (Scheirer et al.) 2013 Geng et al.. 2021 Bendale & Boult 2015). It requires models to maintain flexibility to cope with the scaling of the recognition target, where both input images and the corresponding classes will dynamically expand according to actual needs. For example, in medical diagnosis (Razzak et al. 2017). new diseases emerge constantly and in e-commerce, new categories of products appear daily (Xu et al. 2019), which cannot be predefined in finite class set and remain fixed during inference.
Faced with this challenging open-world recognition problem, traditional supervised classifiers and algorithms have struggled, as they only learn to discriminate limited classes in a closed set, and cannot adapt to the scaling of target classes. However, the emergence of Contrastive Language- Pre-training (CLIP) (Radford et al. 202T) and its open-vocabulary learning paradigm has reversed this situation. CLIP models visual recognition as a task of image-text matching rather than the classic image classification. It pre-trained on web-scale collections of image-text pairs, learning unconstrained visual concepts from natural language supervision with contrastive learning. During inference, it devises a textual prompt like "a photo of a [CLASS]] ", where the class token can be replaced by any potential class name from a vocabulary. The prompt-formed class description with the highest similarity to the input image is predicted as the target class. This modeling paradigm makes CLIP operationally suitable for open tasks in the real world. When input images and the target classes change, CLIP can still conduct zero-shot inference by adaptively adjusting the class names in the vocabulary and then modifying the corresponding class descriptions for matching, sparing re- traiiiinnn the entire model on new data like the traditional classification based methods. Nevertheless, contrary the note "CLIP has wide range of capabilities due to its ability carry out arbitrary image classificatton tasks" in (Radford et al. (002I) previous evaluation of CLIP is still limited in the closed set, leaving its actual performance on open tasks in shadow.

---

Under review as a conference paper at ICLR 2023
In this work, we rethink openness, the intriguing but under-explored property of CLIP, and present protocol for evaluating the openness of CLIP-like models (Radford et al., 2021 Li et al. 2021b Mu et al. 202T; Yao et al. 2021: Zhou et al. 202T) from an incremental view. Specificall,, we define extensibility, which essentially approximates the models' ability in dealing with new visual concepts through vocabulary expansion. Our experimental results based extensibility show that CLIP and its variants have a significant drop in accuracy, c.g. 12.9% of CLIP (RN10I) on CIFARI00 as the vocabulary size expands from to 100, indicating that the limited zero-shot capability of CLIP-like models is not sufficient to support its deployment in the open world. Different from previous openness- related work, we focus on analyzing how the new class descriptions introduced with vocabulary expansion affect the stability of classification on the old input images. Our investigation reveals that the small margin between text features of different classes leads to the prediction shift. To improve the distinguishability of text features and the semantic alignment between images and their textual description, we propose a non- non-parametri method named (REPE), which retrieves relevant captions from the pre-training corpus to customize the prompt for each class during zero-shot inference.
Prompt Enginecring
To summarize, our contribution is folde-fold: (1) To our best knowledge, we the first to systemati- cally investigate the openness of CLIP, for which we design the evaluation protocol and two indicators of extensibility and stability. Through an analysis of the prediction shift during vocabulary expansion, we find that the performance of CLIP is greatly reduced by adding a small number of adversarial non-target classes, exposing the huge risk of its deployment in the open world. (2) We further dissect the feature space of CLIP from the perspectives of representation alignment and uniformity, observing that the uniformity of the textual space critical for better extensibility. (3) We propose a simple yet effective method, REPE, to improve the extensibility and stability of CLIP without fine-tuning.
2 OPENNESS, EXTENSIBILITY, AND STABILITY
In this section, we first review CLIP's visual recognition paradigm based on image-text matching, and then demonstrate how it realizes open- neee-vocabulary image classification in theory by vocabulary expansion (6 2.1. To quantify the actual performance of CLIP-like models as the vocabulary expands, we define the metric of extensibility and propose a systematical evaluation protocol (8 (.2). The exper- imental results and further analysis reveal that, as the vocabulary expands, the predictions of CLIP are unstable and prone to drift the competing class descriptions that are newly introduced, which limits its extensibility and leaves a huge security risk when deployed in real-world applications ($ 2.3
2.1 OPENNESS OF CLIP
In contrast to the classic supervised methods (He et al. 2016 Dosovitskiy et al. 2021), CLIP (Radford ct al. (021) models visual recognition an image- matching task with self supervied contrastive pre-training. Formally, let f be the CLIP model, takes an image x and a target vocabulary y(() = } of the class names w; as inputs, and outputs the predicted label á of the image as:
i = f ((T) = arg max P (y = = xx)
exp (itn(f((t).f()x)))
(1)
= arg max
(V(T) D== exp sim(ff(t))) f(()))
where 6; is the textual description of the class name we in a prompt format, e.g., "a photo of a w.", w,".sim(-" denotes cosine similarity, fr and fi is the text and image encoder in CLIP, respectively. Such a modeling paradigm can realize the open world image classification in theory by extending the target vocabulary (TT to arbitrary degrees. However, in most previous work (Radford al. 2021 Li et al. 202 Mu et al.) [021]] Yao et al., [021] Zhou et al.) (02I). CLIP is evaluated with fixed ((T) depending on the target classes of the downstream dataset DIT);
Acc (DmT) = [DIT]] (X.))
1
Σ (x))))
=
(2)
where [DT) denotes the size of the dataset, and II(-) is the indicator function. This vanilla evaluation setting with restricted input images and classes is insufficient for the open recognition tasks, as it
2

---

Under review as a conference paper at ICLR 2023
Images
dog
dog
oierr
dog
uger
wrlgghouse
Targes Voc0o
Acc-E
Ace (v)7)
Ace
(v
Acc (vT uvu"
Encoder
Text Groodir
Hour Images
proso a (2166))
Targer (T) wostb
Kar-Targen (NI))
siger
dog
Uger
bridgehouse
eBt
deg
tiger
ion
vocaa
(ooo Aece Sec(piirpu (pompee (1/ her (pmmpe u (v7 wigh
Figure 1: Left: the original accuracy of CLIP with target vocabulary (Eq 2 and the conditional accuracy of CLIP with non-target vocabulary 5). .).1. In the latter, the classes from the non-target vocabulary are involved as distractors for the input images restricted in the target vocabulary. Upper right: calculation of Acc-E (Eq. (3. It measures the extensibility of models when recognition targets including both classes and the associated input images are scaling simultaneously. Bottom right: calculation of Acc-S (Eq. (6. which is sub-problem introduced by Acc-E. It measures the prediction stability on the images from the target vocabulary as the distractors from the non- non-target vocabularies are incorporated incrementally.
does not explicitly and systematically models the dynamics of vocabulary expansion during inference, and thus cannot reflect the actual openness of CLIP in the face of real-life class scaling.
2.2 QUANTIFYING EXTENSIBILITY OF CLIP FOR THE OPEN WORLD
To quantify the model's capability in dealing with newly emerged recognition targets, we propose an evaluation protocol and define metric of extensibility based vocabulary expansion. Concretely, incrementally expand the vocabulary (TT) in Eq. 2 by introducing new classes and the associated input images, then evaluate the accuracy after each vocabulary expansion. These values of accuracy measure the dynamic performance of the model as the openness gradually increases, and their expected average is defined as the model's extensibility. In practice, we achieve this expansion by incrementally unioning N disjoint target vocabularies as shown in the upper right panel of Figure
Definition 2.1 (Extensibility). Given N disjoint target vocabularies (D(T)... .... (TT)), we denote
Sy as their full permutation and (((T) as the f(th) vocabulary in permutation Sy. When we vocabularies, we achieve a vocabulary expansion
union the ((h) vocabulary with the previous and obtain D(T)) U... U D(()) The extensibility refers to the averaged classification accuracy across N incremental expansions increases from 1 to N:
Acc-E
E #ESN
Acc (MIT) U....U!?'
(3)
Experimental settings We evaluate the extensibility of CLIP and its variants, including DeCLIP (Li al. 2021b), SLIP (Mu et al., (202T), Prompt Ensemble (Radford et al., (021). CoOp (Zhou et al. 2021). on the CIFARTOD (Krizhevsky & Hinton, 2009) and ImageNet (Deng al.] 2009) datasets. Non-matching methods (Gao et al. 2021 Zhang et al., 202T Wortsman et al. (021) like linear probing, etc., NOT included since they require training classifier with finite class vectors, and thus are not suitable for class scaling in operation. To construct the vocabulary. we leverage the underlying superclass-class, hierarchical structure of the two datasets (Krizhevsky & Hinton 2009 Santurkar et al., 2021), and group the classes belonging to the same superclass into a vocabulary. There 20 vocabularies in CIFARIOM each with classes. For ImageNet, utilize two superclass- class structures (Santurkar et al. 2021): Entity 13 and Living 17. The former has 13 vocabularies, each
'iice (TT) is bound with D(T) in Fq. target vocabulary expansion implics expanding (TT) (including input images and their labels) the same time, which omit for brevity.
3

---

Under review as a conference paper at ICLR 2023
Table I: Extensibility and stability of CLIP-like models on CIFARI00 and ImageNet datasets. A refers to the decline of
(%) compared to Ace-C (%). PE denotes Prompt Ensemble. CoOp requires access to the training data in downstream datasets, and prompt-tuned on all classes with 16 shots, which can be viewed as the upper bound of other zero-shot models.
CIFAR100
ImageNet (Entity13)
ImageNet (Living17)
Extensibility
Stubility
Extensibility
Stubility
Extensibility
Stability
Model
Acc-C
Acc-C
Acc-C
Acc-E
Acc-$
A
Acc-E
A
Acc-S
A
Acc-E
Acc-S
CLIP (RN101)
68.3
55.4
-12.9
54.9
-13,4
80.4
77.4
-3.0
77.3
3.1
77.6
74.5
-3.1
74.4
-3.2
CLIP (VIT-B/32)
78.0
69.6
-8.4
68.9
-9.1
80.8
78.0
-288
77.8
-3.0
78.0
74.4
-3,6
75.0
-3.0
CLIP (VIT-B/16)
79.7
72.6
-7.1
72.0
-7.7
83.5
81.1
-2,4
81.0
-2,5
79.5
77.9
-1.6
77.6
-1.9
SLIP (ViT-B/16)
63.9
51.1
-12.8
50.4
-13.5
65.7
62.3
-3.4
62.0
-3.7
65.7
62.6
-3.1
62.5
.3.2
DeCLIP (VIT-B/32)
78.7
70.8
-7.9
70.4
-8.3
81.9
79,2
-2.7
79.1
-2.8
82.1
80.2
-1.9
80.0
-2.1
PE (ViT-B/32)
78.3
70.3
-8.0
69.9
-8.4
81.9
79.4
-2.5
79.2
-2.7
78.7
76.0
-2.7
75.8
-2.9
PE (ViT-B/16)
79.6
72.6
72.0
-7.6
85.3
83.2
-2.1
83.1
-2.2
79.6
78.2
-1.4
78.0
CoOp (VIT-B/16)
83.6
76.9
-6.7
76.7
-6.9
87.5
85.3
-2.2
85.5
-2.0
82.7
82.6
-0.1
81.3
with 20 classes, while the latter has 17 vocabularies, each with classes. Tables in the Appendix A list all the vocabularies in the two datasets. For each dataset, calculate Acc-C, the averaged classification accuracy across all single vocabularies, based on Eq.
N E Acc (V(T)) Acc-C = N
(4)
It represents the original model performance on closed vocabularies. To calculate the expectation in Acc-E, we sample 100 N permutations for N vocabularies and take the average.
Results As shown in Table I all models exhibit clear drop in performance as the openness of tasks increases. For example, on CIFAR100, compared with the accuracy on closed vocabulary (Acc-C), the accuracy after the vocabulary expansion (Acc-E) of CLIP (RN101) sharply decreased by 12.9%. The performance on the data splits in ImageNet is relatively better, with an averaged decline of 2. 2.7% Appendix B provides results of expansion at the dataset level where the expanded vocabularies from five other datasets. The performance of CLIP-like models drops even more dramatically by an average of 15.3% on generic dataset expansion. These results demonstrate that the openness of CLIP-like models is overestimated under the vanilla evaluation mechanism. Besides, there are some interesting findings: (1) From the perspective of pre-training, introducing a stronger vision backbone (VIT (Dosovitskiy et al., 2 v.s. ResNet (He et al. (016)). widespread supervision (DeCLIP (L et 2021bb v.s. CLIP), and more pre-training data (CLIP v.s. SLIP (Mu et al. (02T)) improve the extensibility of models open tasks. (2) During inference, the performance of CLIP can be boosted by ensembling different prompts (Radford et al. 2021). (3) CoOp (Zhou et al., 2021) that conducts prompt tuning on all classes of CIFARIO0 and ImageNet yields the most extensible results. However, the prompt tuning method utilizes the predefined category information and training data in the target dataset, which cannot be transferred to real-life open tasks.
2.3 STABILITY DURING VOCABULARY EXPANSION
As the vocabulary expansion introduces new classes incrementally, some input images belonging to the previous vocabulary will be incorrectly predicted as new classes, which leads to an accuracy drop and the poor extensibility. To analyze the prediction stability of CLIPP during vocabulary expansion, we introduce the non-target classes. They do NOT correspond to any input images, only serving as distractors for the target classes. Based on it, we define conditional classification accuracy follows:
Acc (v00/rn u u (DDT)
1
Σ (X.))CD(T)
(x,, (xx DlO u u = v)
(5)
where ((TT) is the non-target vocabulary, i.e., the vocabulary of non-target classes. The left panel of Figure gives an illustration of the conditional accuracy. In Eq. [ the categories of the input images
Hence, the N in Def. 2.1 for CIFAR100, ImageNct (Fntity13) and (Living17) is 20, 13 and 17, respectively.
4

---

Under review as a conference paper at ICLR 2023
85
CUPMIT81
CLIP VIT/B22
90
DeCUP-VTT:032
Acc-S
SLIPIIT/
80
75
PE PEMIIII1
70
70
60
65
40
aqualic fish
flowers
food
fruit /egetall les
electrical devices
household
insects
carnivores
outdoor things
outdoor scenes
amnivores herbivores
mammals
invertebrates
people
repa les
trees
veliec Menneee 2
55
suus
(a) Acc-S drops as non- non-target vocabulary extends (Insects target vocabulary).
(b) Difference between Acc-C and Acc-S of CLIP (VIT-
B/32) on different groups.
Acc-C and Acc-S (%) of CLIP and its variants on CIFAR100. The horizontal axis represents the extended non-target vocabularies in order. PE refers to Prompt Ensemble.
Figure 2:
are limited to the target vocabulary ((x, y) E D( but CLIP is asked to distinguish all categories from broader vocabulary y(T) U D(N2), In other words, compared to the traditional closed-set classification, CLIP is supposed to reject all the negative categories from D(NT), The model is required to distinguish visual concepts stably and robustly, rather than making wrong predictions in the presence of other distractors. Based on Eq. we define the stability of CLIIP in the open task:
Definition 2.2 (Stability). Given a target vocabulary D(T) and M non-target vocabularies (D(NT) (((T) }. we denote SM as their full permutation and (NTT) the j(th) vocabulary in a permutation E SM- We design the local stability to measure the averaged classification accuracy of CLIP on the given target vocabulary when non-target vocabularies are extended incrementally:
Acc-S (VCT), D(NT))=
1 M
Σ Acc (ytr) pot u (MtNT) U...UU... .....U))TT))
E .ESM
(6)
As Eq. fonly reflects the local stability with respect to single target vocabulary, we further design the stabilityy as an average of local stability over a set of target vocabularies to reduce the bias from data distribution and vocabulary sampling. Specifically, given N vocabularies (Vi,'.. VN }. we regard each vocabulary V; as the target vocabulary D(T) and the rest Vis as the non-target vocabularies (NT)), and then formulate the general stability
Acc-S
Acc-S (Vi, Vei)
Experimental settings and results The models and datasets adopted for evaluation are consistent with that in 8 [.2 For the calculation of stability, take CIFAR 100 with N 20 vocabularies as an example, we treat each vocabulary as the target vocabulary and the rest are treated as the non-target vocabularies for Eq. 7 To calculate the expectation in Eq. 6 we sample 100 permutations for M 19 non-target vocabularies and report the averaged scores.
Table T demonstrates the stability of CLIP-like models. On CIFAR100, the Acc-S of CLIP (RN101) decreased by 13.4%. Figure 2a shows Acc-S on CIFAR 100 during non-target vocabulary expansion. Given a closed (TT) = Insects, CLIP (ViT-B/32) achieves an accuracy of 81. 81.2%. However, when the remaining 19 non-target vocabularies are incorporated, the accuracy sharply drops to 57.0%. The decrease of Acc-S brought by each introduction of non-target vocabulary indicates that more images from Insects are incorrectly classified into the new vocabulary. Figure2b]demonn 2bddemonst the difference between Acc-C and Acc-S for each target vocabulary. When (T) Medium- sized Mammals, CLIIP
5

---

Under review as a conference paper at ICLR 2023
is most easily interfered with by the non-target vocabularies, with 21.08% performance drop. It suggests that the unstable predictions lead to the poor extensibility of CLIP when new categories are introduced. Besides, we notice that CLIP performs stably groups like Flowers, where its Acc-S only declines by 0.53% compared to Acc-C. The different behaviors of different groups indicates that the stability is also influenced by the inherent property of the image categories.
Furthermore, In order to explore the lower bound of the stability, we define the adversarial non- target vocabulary V(ANT) as the non-target vocabulary that reduces Acc-S the most. Specifically, we maliciously introduce only three new classes and find that the performance of CLIP drops dramatically, e.g., e.g.,2..7% 52.7% accuracy drop on CIFAR I0. It reveals the vulnerability of CLIP and its poor semantic modeling on those objects with higher abstraction levels. Please refer to Appendix [ for details.
DISSECTING AND IMPROVING THE EXTENSIBILITY OF CLIP
Our experimental results in expose the unsatisfying performance of CLIP on open tasks. In this section, we dive into the representation space of CLIP to find the key to understanding and improving its extensibility. We first point out that the small margin between positive and negative class descriptions leads to the prediction shift when competing text features appear, which thus limits the stability CLIP &$ G.T). Further, we investigate the representation space of CLIP-like models via two metrics of model alignment and intra-modal uniformity. The results show that enforcing the distinguishability of text features enlarges the margin and makes models scale more stably ($ 3.2). In response, we propose non-parametric method named (REPE), which boosts the performance of CLIP even without fine-tuning ($ 3.3
Prompt Engineering
3.1 SMALL MARGIN LIMITS THE STABILITY OF CLIP
Since CLIP formalizes the visual recognition as an image-text matching task (Eq, D. each text feature of the class description corresponds to the class vector in traditional classifiers, and the image-text similarity scores are thus analogous to the logits in classification. Ideally, no matter how the vocabulary expands, for image, the similarity of the positive pair, i..., the image with the text specifying the ground-truth class, should be higher than those of the negative pairs, i.e., the image with the texts specifying other classes, ensure the correct prediction of CLIP on open tasks. In other words, the margin (Jiang et al. 2019) between positive and the largest negative similarity is a direct contributing factor the stability.
Unfortunately, the similarity and margin distribution of CLIP does not meet our expectations. Figure 3 illustrates the averaged cosine similarity of CLIP (ViT-B/32) on classes of CIFAR100. Similarity over the intact dataset is in Appendix E The diagonal elements represent the similarity of the positive image-text text pairs, while the others represent that of the negative ones. As shown in Figure the similarity histogram of positive and negative pairs has large overlap. Its margin in Figure is clustered around zero, leaving the prediction of models at risk being reversed to the new non-target classes. For example, the vocabulary extends from the red box to the green box (diagonal) the yellow box (horizontal) in Figure 3 more deceptive classes (circle) with negative margin are added, which leads the prediction shift. Particularly, the classes belonging to the same vocabulary have higher similarity and smaller margin, which is more likely to be confused with each other.
3.2 IOTER-MODAL ALIGNMENT AND INTRA- UNIFORMITY GROUND THE MARGIN
The results in 8 3.1 raise a natural question: what vision-and lnd-lang feature distribution can maintain large margin between different classes so that the model can scale stably in the open world? Here we present two properties of the ideal feature space: First, the text feature class name is supposed to stay close to the features of the images it describes, promoting the similarity of positive pairs. Second, intra-modal features, especially the textual features should be uniformly distributed to preserve maximal information and make the descriptions of competing categories more distinguishable. Accordingly, we define inter-modal alignment and intra-modal uniformity, two metrics to measure the quality of representations in contrastive learning (Wang & Isola 2020) for the sadion-and-languune domain. Inter-modal alignment calculates the expected distance between
Evvry 5 adjacent classes in Figure constitute vocabulary (superclass), see Table in Appendix

---

Under review as a conference paper at ICLR 2023
3qu fish-T
ray-1
sharict
troucCT
flatishT
beavert
dolghina
ottersT
seal-F
whale-T
bearT
leopard-1
5gecT
WA0F-T
303-
aqu. fish-I
627
323
624
0:5
230
032
311
621
638
831
033
ray-
shark-l
trout-1
224
0.2
3.24
0.23
6.8
214
521
125
6UT
218
6.5
022
Onaee
Nattish-
Figure 4: Cosine similarity his- togram of positive (pos) and neg- ative (neg) image-text pairs with large overlap.
beaver-1
630
631
dolphin-1
6:22
022
6.5
621
532
143
8:30
020
seal-1
122
622
whale-1
622
022
6:31
6.00
022
620
6:1
6:00
631
030
522
leopard-1
0.00
0.2
8.00
642
3.1
1.3
2.21
021
lion-1
6.00
0.31
8.38
6.09
3.11
2:2
531
233
031
0.3
100
tiger-
-0/0 -254
ESS C.0 AM OZE CES
wolf-1
6.0
0.33
3.00
333
125
232
2:1
6:2
031
002
Figure 5: Margin distribution of similarity scores, which cen- tered around zero with a median value of .005 (the blue vertical line). It indicates that the predic- tions can be easily inverted with competing classes appearing.
Figure 3: Averaged cosine similarity between image (-I) and text (- T) features of CLIP (ViT-B/32) on CIFARI00, The expansion from the red box to the green box (diagonal) and the yellow box (horizontal) refer to the calculation of extensibility and stability, respectively. The circle represents that more than 15 wrong pre- dictions have arised after adding this class.
features of positive image-text pairs Ppos
falign
E
[[r(x) fr (c)11,
(8)
(x.t))Ppnn
while intra-modal uniformity measures how well the image or text features are uniformly distributed:
Euniform
Lunitoomal
+ Lunitomm-
log
e -1111(x))--1xxxx111 + log
2 -112(())-2((tt))
t,,,y 12.d PimTT
where P and P denotes the image and text data distribution, respectively. Figure Table in Appendix provide quantified loss of alignment and uniformity. CLIP with only cross-modal contrastive learning results in poor intra-modas uniformity (Funiform -2.0), especially the text side. Introducing intra-modal contrastive learning like SLIP and DeCLIP in pre-training can force both image and text features separate better from classes, reducing funifiem below -4.5. As for the prompt tuning method (CoOp), achieves better falign of 1.4 compared to CLIP (1.5) and the lowest E of -.22. According the visualization via Multidimensional Scaling (MDS) (Borg) & Groenen| 1997) in Figure 7 the optimization trajectory of prompts is towards the cluster center of the corresponding image Teatures, while dispersing the position of the prompt features, which improves both text uniformity and inter-modal alignment, achieving the best extensibility.
3.3 METHODOLOGY: DETRIEVAL-ENHANCES PROMPT ENGINEERING (REPE)
In light of the previous investigations, we propose a simple but effective method named Retrieval- enhanced Prompt Engineering (REPE) to enforce the distinguishability of text features and the semantic alignment (Cao et al. 2020 Ren et al. 2021). Recall that the context for each class name is the same in vanilla CLIP- like models (c.g., photo of [CLASS] "), making it difficult to discriminate between distinct visual categories because the semantics each cannot be holistic represented/Zhou al. (2022) To remedy this, propose customize each class description with
7

---

Under review as a conference paper at ICLR 2023
lespard
CUIP (aiin))
12.5
72.4
78
0.50
leepard-
2.5
78.5
60.5
0.25
woman-
womaniT
worms
Funtorm
-3.0
70
-0.25
88
-0.30
catorpilaani
3.5
-0.75
platet
18.9
62
-1.00
plates
SUP (TT-010
58
-0.5
-1.5
10:8
54
Figure 7: Representation visualization of CLIP and CoOp (ViT-B/16). The five classes with different colors are from CIFARIO0 refers to image features (-I), while x and * refers to text features (-T) of CLIP and CoOp,
50
1.2
1.3
1.4
1.5
1.5
1.7
Zalign
Figure 6: Ealign and luniform of CLIP-like models. For both two metrics, lower numbers are better. The color of points and numbers denote the extensibility perfor- mance (Ace-E) CIFARI00 (higher is better).
respectively. The color from transparent to opaque indicates the optimization trajec- tory during the CoOp prompt-tuning process.
Table 2: Extensibility and stability of our REPE method on CIFAR100 and ImageNet datasets.
CIFARI00
ImageNet (Entity13)
ImageNet (Living17)
Model
Acc-C
Acc-E
Acc-S
Acc-C
Acc-E
Acc-S
Acc-C
Acc-E
Acc-S
CLIP (RN10L)
68.3
55.4
54.9
80.4
77.4
77.3
77.6
74.5
74.4
REPE (RNIOI)
68.4
55,5 (1.1)
55.2
81.7 (-1.5)
79.2
79.0
77.8 (-0.3)
75.3
75.2 (-0.)
CLIP (VIT-B/32)
78.0
69.6
68.9
80.8
78.0
77.8
78.0
74.4
75.0
REPE (VIT-B/32)
(v..5)
70.9 (11.3)
70.6 (..7
82.3 (11.5)
79.8 (11.5
79.6
79.0 (x10)
76.4 (-2.0)
76.2 (61.2)
CLIP (VIT-B/16)
79.7
72.6
72.0
83.5
81.1
81.0
79.5
77.9
77.6
REPE (VIT-B/1)
79.8.-0.1 (+0.1)
72.9 (+0.3)
72.6..0.0
85.4 (..9)
83.3 (-2.2)
83.2 (12.3)
79.9 (-0.4)
78.4 (-0.5)
78.2 (-0.6)
diverse captions retrieved from the pre-training corpus prompt ensemble. Specifically, for each class description based on the original prompt, we utilize CLIP to recall the most similar images from the pre- pre-training dataset via image-text similarity then obtain their corresponding captions. The retrieved captions without class name appearance are filtered out, finally resulting in K captions. Such workflow leverages both visual semantics and class name, achieving better performance. Appendix G shows the instances of the retrieved captions. They provide the context in which the class name is located and thus have richer semantics. After that, encode the retrieved captions and conduct a mean pooling operation among them. The final text representation is
REPE (t.) (1- ()f(t)) +
K Σ fr(tty))
(10)
where rt. is the j((h) retrieved caption for class i, A is a weighting factor and is adjusted on the validation set. The ensemble text representation f (t.) is then adopted as the class anchor for conducting the image classification. The text representation for the target class is thus shifted towards the representative captions in the pre-training dataset, which alleviates the semantic inconsistency between pre-training and inference.
Experiments We retrieve the images and captions from CC12M (Changpinyo et al. 2021) a subset of the pre-training dataset of CLIP. The images and captions are pre-encoded within an hour using a single RTX TITAN GPU, then we build their indices for KNN search with FAISS framework (Johnson) al. 2019), which also takes about an hour. Once the indices are built, we can efficiently search over the dataset according to the query image in less than 5 ms, which is applicable for query-intensive scenarios. Appendix HH provides the detailed process and computational overhead of the retrieval. Table shows the results of retrieval-enhanned prompt engineering. The hyper-paramete K is 100 and À is 0.25. We find that on all the three datasets, REPE consistently improves the extensibility and stability of CLIP by an average of 1.2%. We further evaluate the loss of text uniformity and inter-modal alignment for probing the quality of the enhanced representations. As shown in Figure 6 the former is effectively reduced from -0.8 to 1.0 and the latter is reduced from 1.5 to 1.4, verifying
8

---

Under review as a conference paper at ICLR 2023
our proposal can improve the class anchor for better extensibility and stability. Additionally, REPE increases the median value of the margin distribution from 0.005 to 0.01 and pushes the overall distribution towards the positive side compared to vanilla CLIP (Figure 12 in Appendix [E). It indicates that REPE widens the gap between positive and negative class features, making it more difficult to invert predictions with competing classes. In conclusion, all of these findings support REPE's efficacy in addressing the openness issue.
It is worth noting that compared the method that requires computation-intensive pre-training procedure (DeCLIP and SLIP), and the prompt-tuning approach (CoOp) demands access to the downstream target dataset, our REPE is a lightweight framework for the zero-shot inference stage without fine-tuning. Besides, since REPE is model-agnostic and orthogonal to gerameter-tuning methods, it can be also combined with fine-tuning methods like adapter-tunig (Gao al.) (021), to achieve further performance boost by an average of 0.6 on CIFARION 100 and ImageNet, which demonstrates the adaptability and superiority of our method. Please refer to Appendix for details.
4 RELATED WORK
Contrastive language-image pre-training and open-vocabulary learning CLIP (Rafford et al. 2021) enables learning transferable visual models from natural language supervision and makes visual recognition generalize in the wild (Zareian et al., 2021] Gu et al.) 2022 Ghiasi et al. 2021). It is pre-trained on web-scale collections of image-text pairs, learning tremendous visual concepts described by natural language with contrastive learning. During inference, it devises a prompt like "a photo of [CLASS]", ", where the class token a placeholder for any potential class name from a vocabulary, and the class description with the highest similarity to the input image is predicted as the target class. Another line of recent studies (Li et al., 2021a Wang al., 2022 Yu et al., 2022] Alayrac et al. 2022) adopts seq2seq generation instead of contrastive discrimination framework to achieve open-vocabulayy recognition. We leave the investigation of their extensibility for future work.
Open set and open-world visual recognition Open Set Recognition (OSR) (Scheirer et al. 2013 Geng et al., 2021) requires classifiers to identify images that have not been introduced during training as "unknown". The task can be formalized as one-vs-reset classification (Scheirer al. 2013) or multi-class classification (Jain et al. 2014 Scheirer et al. 2014). Furthermore, Open World Recognition (OWR) (Bendale Boult 2015) raises higher demands that the model must incrementally learn and extend the multi-class classifier as the unknowns are labeled for new class learning. Contrary to the above research, the CL.IP-based unsupervised. We focus on the model performance on zero-shot inference, without training on the target dataset. Appendix proovddss more detailed comparison of OSR, OWR, and OVR.
Recognition (OVR)
LIMITATIONS AND FUTURE WORK
To facilitate future research, we analyze the difficulties and possible solutions in this new area. (1) As present extensive empirical results and address the weakness of CLIP vocabulary expansion, (2) The current evaluation protocol is an approximation of the real open world. An evolving benchmark could facilitate future research. (3) For various visual categories, their degree of abstraction, the ease of describing them in natural language, and their density in the data distribution can also influence the extensibility and stability of models, which are worth studying.
its theoretical risk on open tasks is urged to be investigated.
(4) Within the CLIP & prompting framework, our REPE can be easily adopted in various downstream visual tasks like open-vocabulary object detection (Zareian et al. 2021) semantic segmentation (Ghiasi et al. 2021), etc. We leave it for future work.
6 CONCLUSION
In this paper, we evaluate the extensibility of CLIP-like models for open-vocabulary visual recognition. Our extensive investigation shows that the performance deteriorates seriously as the vocabulary expands, which results from the indistinguishable text features among competing classes. To remedy this, we propose REPE to enhance the textual representations with relevaatl captions retrieved from the pre-training corpus, which boosts the extensibility and stability even without fine-tuning.

---

Under review as a conference paper at ICLR 2023
REFERENCES
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 11.44.14999 14198, 2022.
Abhijit Bendale and Terrance E. Boult. Towards open world recognition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 1893-1902. IEEE Computer Society, 2015. doi: 10.1109/CVPR.2015572299
URL https
7/doi.org 10. www9/1110...... 7298799
Ingwer Borg and Patrick J. F. Groenen. Modern multidimensional scaling: Theory and applications.
Journal of Educational Measurement, 40:277-280, 1997.
Jize Cao. Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. Behind the scene: Revealing the secrets of pre-trained iision-and-languag models. In ECCV, 2020.
Soravit Changpinyo, Piyush Kumar Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Push- ing web-scale image-text pre-training recognize long-tail visual concepts. 202/ IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3557-3567, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A acale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEEE org/ 10
Computer Society, 2009, doi: 10. 10.11000 200.0///VV...00...2.2. 5206848. URL
1109/CVPR. 2009 5206848
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:://opessee
net forum?id
YicbFaNTTy
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested 101 object categories. CVPR Workshops, 2004.
Christiane D. Fellbaum. Wordnet an electronic lexical database. Language, 76:706, 2000.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Jiao Qiao. Clip-adapter: Better vision-language models with feature adapters. ArXiv, abs/2110.04544, 2021.
Chuanxing Geng, Sheng-Jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43:3614-3631, 2021.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Y Yi Lin. Open-vocabulary image segmentation. ArXiv, abs/2112.12143. 2021.
Xiuye Gu, Tsung-Y Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2022. htt.aatpss//ppeeeeeeeeeeeee
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1 2..1109/VPR.2001..3
URI. /oii //0g 1109/ 2016.
Lalit P. Jain, Walter J. Scheirer, and Terrance E. Boult. Multi-class open set recognition using probability of inclusion. In ECCV, 2014.
10

---

Under review as a conference paper at ICLR 2023
Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htttps://openrest ew net forum? ? d=HJ] @f
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547, 2019.
Jonathan Krause, Michael Stark, Jía Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. 2013 IEEE International Conference on Computer Vision Workshops, pp. 554 561, 2013.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705, 2021a.
Yangguang Li, Feng Liang. Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive language-image pre-training paradigm. ArXiv, abs/2110.05206 2021b.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: supervvipni meets language-image pre-training, ArXiv, 12./2112.12750, 2021.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pp. 722-729, 2008.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, pp. 1-998-3005. IEEE Computer Society, 2012. doi: https: www.l.oogg100/ 1
URL
2012. 6248092
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Reseurch, pp. 8748-87686 PMLR, 2021. URL http::h///pp/oeeeeeeee radford21a html
ml htt.::s.........ss///mmmm
Muhammad Imran Razzak, Saeeda Naz, and Ahmad Zaib. Deep learning for medical image process- ing: Overview, challenges and future. ArXiv, abs/1704.0082.. 2017.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1085-1097, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/1199 103. URL htttts://ccclantttt
org/P19-1103
Shuhuai Ren, Junyang Lin, Guangxiang Zhao, Rui Men, An Yang, Jingren Zhou, Xu Sun, and Hongxia Yang. Learning relation alignment for calibrated cross-modal retrieval. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- fional Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pp. 514-524, Online, 2021. Association for Computational Linguistics. doi: 10. 00.18653/11/222..oo 1ac.200o..al-oooo.4444
URL / www.:tthhwooooyyo.o/2202.2.2 (tlpg/2221.cc0 -onn...og... .43
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. BREEDS: benchmarks for subpopulation
shift. In 9th International Conference Learning Representattinns ICLR 2021. Virtall Event, Austria, May 3-7, 2021. OpenReview.net, 2021. wtt.../R//tpps///ppeeee..eccco
het.leewwweeeeetteee net.
H 1d=m@@BmvyAuu
11

---

Under review as a conference paper at ICLR 2023
Walter J. Scheirer, Anderson Rocha, Archana Sapkota, and Terrance E. Boult. Toward open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35:1757-1772. 2013.
Walter J. Scheirer, Lalit P. Jain, and Terrance Boult. Probability models for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36:2317-2324, 2014.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequence-to-seen tosuuunnce learning framework. arXiv preprint (rxXi:2222.030355 2022.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Con- ference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 9929-9939. PMLR, 2020. URL http: //proceedings.u ml htt.:/eeeedaaa.....r....e....e.e v http///////gg....mmm html
Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. ArXiv, abs/2109.01903, 2021.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. The Twenty-Thid IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, pp. 3485-3492. IEEE Computer Society, 2010. doi: 10.109/CVPR.2210.555900 URL https: org/10 wttp://..1119///220 1119//00PSS 2010..5399700
Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. Open-world learning and application to product classification. In Ling Liu, Ryen W. White, Amin Mantrach, Fabrizio Silvestri, Julian J. McAuley, Ricardo Baeza- Yates, and Leila Zia (eds.). The World Wide Web Conference. WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pp. 3413-3419, ACM, 2019. doi: 10.1145/3308558, 3313644. URL URL httpt:tt:///om /doii org/ 0 145/3308558 3643
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-ing immge pre-training ArXiv, abs/21 11.07883, 2021.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Alireza Zareian, Kevin Dela Rosa. Derek Hao Hu, and Shih-Fu Chang. Open vocabulcyy object detection using captions. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14388-143977 2021.
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Jiao Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better lision-language modeling, ArXiv, abs/2111.039300 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. ArXiv, abs/2109.01134, 2021.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. ArXiv, abs/2203.05557, 2022.
A SUPERCLASS-CLASS HIERARCHY FOR VOCABULARY CONSTRUCTION
To construct the vocabulary of CIFAR100 (Krizhevsky & Hinton 2009) and ImageNet (Deng al., (209)). and group the classes belonging to the same superclass into a vocabulary. Table 3 lists the vocabularies in CIFAR100, which are specified by (Krizhevsky & Hinton 2009). There are 20 vocabularies, each with 5 classes.
we leverage the underlying superclass- class hierarchical structure
12

---

Under review as a conference paper at ICLR 2023
For ImageNet, we utilize two superclass structures, Entity13 and Living17, in (Santurkar ett al, (002). Table 4 and Table show the vocabularies in ImageNet (Entity13) and ImageNet (Living17), respectively. The former has 13 vocabularies, each with 20 classes, while the latter has 17 vocabularies, each with 4 classes
B DATASET-LEVEL EXTENSIBILITY
The evaluation protocol in & 2 estimates the extensibility and stability within single task dataset, where the input images and classes during the vocabulary expansion come from the same data distribution. While the protocol is only an approximation of the real open world, current CLIP-like models have exhibited serious performance degradation. In this section, we take step further toward the real open recognition, by conducting vocabulary expansion setting on the dataset level, where the expanded vocabularies vocabularies is more uncertain and thus can be viewed as a rigorous stress test for the CLIP-like models. Specifically, we group all categories in a dataset into one vocabulary, Afterward, the inputs and classes of the entire new dataset are introduced at each expansion. Classes in the new vocabulary will be removed if they already exist in the previous vocabularies.
from different datasets. In this way, the relationship between
The experiments are conducted with datasets for generic objects, including CIFAR10 (Krizhessyy Hinton 2 CIFARIOS 100 (Krizhevsky Hinton 2009), Caltech101 (Fei-Fei al. 20044)
SUN397 (Xiao et al. 2010) and ImageNet (Deng et al. 2009). and specialized datasets focusing fine-grained categories, including Flowers 102 (Nilsback Zisserman, 2008), OxfordPets (Parkhi
et al., 2012) and StanfordCars (Krause et al. 2013). Without loss of the generality, we merge 3
datasets and evaluate the following dataset compositions:
CIFARRI0-Caatechh0--SUSS33
CIFARIO-CIFARRIOO-mmeeeh Flowers (22--0xffodottttttttnooooo
Composition (1) and (2) probe the performance when all the expanded datasets generic thus the classes in different datasets are semantics correlated, while composition (3) targets scenarios where the coming datasets have little correlation with previous ones. To eliminate the effect of vocabulary expansion order, we report the average performance of all A 6 possible trials for each composition.
Table 6 demonstrates the result of the dataset-level expansion. First, the performance of CLIP- like models on generic dataset expansion drops dramatically. For example, the accuracy (Acc-E) of CLIP (RN101) decreases by an averaged absolute point of 14.2 on the CIFAR 100-Caltechh 101- SUN397 composition during expansion, and 14.5 on the CIFARIO- CIIARIOO- respectively. Due to the existence of subclass-superlaas relationship for some classes in different generic datasets, e-g., cat in CIFAR1O and tiger cat in ImageNet, CLIP is extremely unstable on such expansion across generic datasets. For example, the Ace-S of CLIP (RN101) on the CIFAR/0 CIFAR/ 100 100-mmaeNNe composition is 28.2% lower than Acc-C, indicating the models are prone to relationship. Meanwhile, the CLIP-like models exhibit
composition,
be confused about the subclass-supent much better extensibility and stability on the dataset-level expansion across specialized datasets, e.g.. the Flowers102 OxfordPets-StanfordCCC StanfordCar composition. The vocabularies of this composition are intrinsically disjoint in semantics, so the model can be stably extended. In summary. our investigations the dataset level expansions along with the task level in the paper show the current CLIP-like models fail to meet the expectation of conducting real open vocabulary recognition, and we hope our studies can motivate future studies this direction.
C INCREMENTAL Acc-E AND ACC-S ON CIFAR100
We record the Acc-E (Eq. 3) and Acc-S (Eq. 6 after each vocabulary expansion CIFAR100 to investigate the effect of increased task openness on CLIP-like models.
Figure X shows the Acc-E for 20 trials lines indicate that the model either performing poorly on the new input images, or that some images that were correctly identified before misclassified after introducing the new classes.
vocabularies merged incrementally. The falling
13

---

Under review as a conference paper at ICLR 2023
Table Superclass-class hierarchy in CIFARI00. Each superclass corresponds to a vocabulary, and each vocabulary has 5 classes. There are 20 kinds of vocabulary in total, specified by Krizhevsky & Hinton 2009).
Vocabulary (Superclass)
Classes
aquatic
mammals beaver, dolphin, otter, seal, whale aquarium fish, flatfish, ray, shark, trout orchids, poppies, roses, sunflowers, tulips containers bottles, bowls, cans, cups, plates apples, mushrooms, oranges, pears, sweet peppers clock, computer keyboard, lamp, telephone, television furniture bed, chair, couch, table, wardrobe bee, beetle, butterfly, caterpillar, cockroach bear, leopard, lion, tiger, wolf bridge, castle, house, road, skyscraper cloud, forest, mountain, plain, sea camel, cattle, chimpanzee, elephant, kangaroo fox. porcupine, possum, raccoon, skunk crab, lobster, smail, spider, worm baby, boy, girl, man, woman crocodile, dinosaur, lizard, snake, turtle hamster, mouse, rabbit, shrew, squirrel maple, oak, palm, pine, willow bicycle, bus, motorcycle, pickup truck, train lawn-mower, rocket, streetcar, tank, tractor
fish
flowers
food
fruit and vegetables
household electrical devices household
insects
large carnivores
large man-made outdoor things large natural outdoor scenes large omnivores and herbivores medium- cidiuum-szzed mammals non con-insect invertebrates
people
reptiles
small mammals trees vehicles 1
vehicles 2
Figure shows Ace-S of CLIP-like models during non-target vocabulary expansion. Each sub-figure represents the situation when one vocabulary is selected as the target vocabulary. As the remaining 19 non-target vocabularies are incorporated and the model is required to recognize the 5 target classes from 100 potential classes, the accuracy drops sharply. The decrease of Acc-S brought by each introduction of non-target vocabulary indicates that more images from the target vocabulary are incorrectly classified into the new non-target vocabulary by models.
D ADVERSARIAL NON-TARGET VOCABULARY
In order to explore the lower bound of the stability of CLIP, we define the adversarial non-target vocabulary ((ANT) as the non-target vocabulary that reduces Acc-S the most:
DIANT)
min Ace (VeT wiT) ulNT))
(1I)
V(NT)
To build ((ANT) we refer to the method of adversarial examples generation in the Natural Language Processing field (Ren et al. 2019) to traverse the words in a large vocabulary, e.g., the vocabulary of nouns in WordNet (Fellbaum) 2000), which are regarded as non-target classes in order to calculate Acc-S, and then take the most contusing words to form the adversarial non-target vocabulary.
We constrain the size of ((NN)) to 3. Results in Figure 10 illustrate the performance with nouns in WordNet and class names in ImageNet as the candidate vocabulary, respectively. First, we observe a clear performance degradation on both datasets under adversarial attack, e.g., adding hitmap, automobile insurance and equidae leads to an absolute 52.7% accuracy drop on CIFAR 10. Besides, we find that the selected adversarial words are much less concrete than common visual concepts like Flower, indicating the potential reason behind the poor semantic modeling of CLIP on those objects with higher abstraction levels. This investigation reveals that CLIP is vulnerable when facing malicious non-target vocabulary, and we hope future work may pay more attention to the robustness of CL.P under open recognition tasks.
14

---

Under review as a conference paper at ICLR 2023
Table 4: Superclass-class hierarchy in ImageNet (Entity13) 13), Each superclass corresponds to vocabulary, and each vocabulary has 20 classes. There are 13 kinds of vocabulary total, specified by BREEDS Santurkar
Sratukara et al. (202T).
Vocabulary (Superclass)
Classes
trench coat, abaya, gown, poncho, military uniform, jersey, cloak, bikini, miniskirt, swimming trunks,
garment
lab coat, brassiere, hoopskirt, cardigan, pajama, academic gown, apron, diaper, sweatshirt, sarong
African grey, bee cater, coucal, American coot, indigo bunting, king penguin, spoonhill, limpkin, quail, kite, prairie chicken, red-breasted merganser, albatross, water ouzel, goose, oystercatcher, American egret, hen, lorikect, ruffed grouse
bird
Gila monster, agama, triceratops. African chameleon, thunder snake, Indian cobra, green snake, mud turtle, water snake, loggerhead, sidewinder, leatherback turtle, boa constrictor, garter snake, terrapin, box turtle, ringneck snake, rock python, American chameleon, green lizard
reptile
rock crab, black and gold garden spider, tiger heetle, black widow, barn spider, leathopper, ground beetle, fiddler crab, bee, walking stick, cabbage butterfly, admiral, lacewing, trilobite, sulphur butterfly. cicada, garden spider, leaf beetle, long-horned beetle, fly
arthropod
Siamese cat, ibex, tiger, hippopotamus, Norwegian elkhound, dugong, colobus, Samoyed, Persian cat, Irish wolfhound, English setter, Ilama, lesser panda, armadillo, indri, giant schnauzer, pug, Doberman, American Staffordshire terrier, beagle
mammal
bib, feather boa, stole, plastic bag, bathing cap,
cowboy boot, necklace, crash helmet, gasmask, maillot, hair slide, umbrella, pickelhaube, mit- ten, sombrero, shower cap. sock, running shoe, mortarboard, bandkerchief
accessory
calamaran, speedboat, fireboat, yawl, airliner, container ship. liner, trimaran, space shuttle, aireraft carrier, schooner, gondola, canoe, wreck, warplane, balloon, submarine, pirate, lifeboat, airship
craft
volleyhall, notebook, basketball, hand-held computer, tripod, projector, barbell, monitor, croquet ball, balance beam, cassette player, snorkel, horizontal bar, soccer ball, racket, baseball, joystick, microphone, tape player, reflex camera
equipment
wardrobe, toilet seat, file, mosquito net, four four-poster, bassinet, chiffonier, folding chair, fire screen, shoji. studio couch, throne, crib, rocking chair, dining table, park bench, chest, window screen, medicine chest, barber chair
furniture
upright, padlock, lighter, steel drum, parking meter, cleaver, syringe, abacus, scale, corkscrew. maraca, saltshaker, magnetic compass, accordion, digital clock, screw, can opener, odometer, organ, screwdriver
instrument
castle, bell cote, fountain, planetarium, traffic light, breakwater, cliff dwelling, monastery, prison, water tower, suspension bridge, worm fence, turnstile, tile roof, beacon. street sign, maze, chain-lint fence, bakery, drilling platform
man-made structure
snowplow, trailer truck, racer, shopping cart, unicycle, motor scooter, passenger car, minibus, jeep, recreational vehicle, jinrikisha, golfcart, tow truck, ambulance, bullet train, fire engine, horse cart, streetcar, tank, Model T
wheeled vehicle
broccoli, corn, orange, cucumber, spaghetti squash, butternut squash, acorn squash, cauliflower, bell pepper, fig, pomegranate, mushroom, strawberry, lemon, head cabbage. Granny Smith, hip, ear, banana, artichoke
produce
15

---

Under review as a conference paper at ICLR 2023
Table 5: Superclass-clan hierarchy in ImageNet (Living17). Each superclass corresponds to vocabulary, and each vocabulary has 4 classes. There are 17 kinds of vocabulary in total, specified by BREEDS Santurkar et al. 2021
Vocabulary (Superclass)
Classes
salamander
eft, axolotl, common newt, spotted salamander box turtle, leatherback turtle, loggerhead. mud turtle whiptail, alligator lizard, African chameleon, banded gecko night snake, garter snake, sea snake, boa constrictor tarantula, black and gold garden spider, garden spider, wolf spider ptarmigan, prairie chicken, ruffed grouse, black grouse macaw, lorikeet, African grey, telphur-cresttdd cockatoo Dungeness crab, fiddler crab, rock crab, king crab bloodhound. Pekinese, Great Pyrenees, papillon coyote, red wolf, white wolf, timber wolf grey fox, Arctic fox, red fox, kit fox tiger cat, Egyptian cat, Persian cat, Siamese cat sloth bear, American black bear, ice bear, brown bear dung beetle, rhinoceros beetle, ground beetle, long-horned beetle sulphur butterfly, admiral, cabbage butterfly, ringlet gibbon, orangutan, gorilla, chimpanzee marmoset, titi, spider monkey, howler monkey
turtle
lizard
snake
spider
grouse
parrot
crab
dog
wolf
fox
domestic cat
bear
beetle
butterfly
ape
monkey
Table 6: Extensibility and stability of CLIP and its variants during dataset-level vocabulary expansion. A refers to the decline of Acc-E/ Scc-E/Ace-S (%) compared Acc-C (%). PE denotes Prompt Ensemble.
CIFARI00 1o SUN3977
CIFARIO CIFAR 100-aRio--maaeNe
Oxforreeees StanfordCar
Model
Acc-C
Acc-E
Extensibility
A
Acc-S
Stability
Acc-C.
Acc-E
Extensibility
Acc
Stability
Acc-C.
Acc E
Extensibility
Acc-S
Stability
CLIP (RNIOT)
51.7
-14.2
52.7
-13.2
62.4
47.9
34.2
-28.2
-2.7
CLIP (NTT-B/52)
72.0
59.4
-12.6
61.2
-10,8
70.9
52.7
-18.2
41.3
-2.6
65.8
62.0
-3.8
65.8
-0.0
CLIP (/16)
74.6
-14.0
61.7
-12.
74.7
56.6
-18.0
43.3
-31.4
72.3
69.6
-2.7
72.3
SLIP (VIT-B/16)
58.6
44.4
-14,2
46.3
-12.3
55.6
36.7
-18.9
30.5
-25.1
35.0
26.0
.9.0
35.0
DeC1 IP (VIT-B/32)
74,3
60.8
-13.5
63.3
-11.0
73.0
55.4
-17.6
45.1
-27.9
70.2
63.3
-6.9
70.2
-0.0
PE: (VIT-B/16)
75.0
61.5
-13.5
62.5
-12.5
75.4
$6.7
-18.7
41.3
.34.
72.5
70.0
-2.5
72.5
16

---

Under review as a conference paper at ICLR 2023
Table 7: Inter-modal alignment (Calign). ). text uniformity (Cunifomn-T). image uniformity (fuiiforn-),, Acc-C (Eq. D. and Acc-E (Eq. 33 of CLIP-like models on CIFAR100. For the first three metrics, lower numbers are better. For the last two metrics, higher numbers are better.
Alignment & Uniformity
Accuracy
Model
Palign ())
uunifoem T (4)
funiform (+)
Ace-C (t)
Acc-E (t)
CLIP (RNIOI)
1.15
-1.16
-0.57
68.3
55.4
CLIP (VIT-B/32)
1.48
-0.96
-0.93
78.0
69.6
CLIP (VTT-B/16)
1.50
-0.97
-0.81
79.7
72.6
SLIP (ViT-B/16)
1.73
-2.86
-1.69
63.9
51.1
DeCLIP (ViT-B/32)
1.62
-2.96
-1.87
78.7
70.8
PE (VIT-B/32)
1.46
-0,96
-0.93
78.3
70.3
PE (ViT-B/16)
1.48
-0.97
-0.81
79.6
72.6
CoOp (ViT-B/16)
1.40
-3.16
-0.81
83.6
76.9
E COSINE SIMILARITY OF IMAGE-TEXT FEATURES ON INTACT CIFAR100
Figure illustrates the averaged cosine similarity between image and text features of CLIP (ViT- B/32) on all classes of CIFAR 100. The elements on the diagonal represent the similarity of the positive image-text pairs, while the others represent that of the negative ones. Since every adjacent classes in the figure constitute vocabulary (superclass), the classes belonging to the same vocabulary have higher similarity and smaller margin, which is more likely to be confused with each other.
Our Retrieval-enhanced Prompt Engineering (REPE) method alleviates this issue by enlarging the margin between positive and the largest negative similarity. As shown in Figure [1] the median value REPE's margin distribution is .01 (the blue vertical line), which is larger than that of CLIP (ViT-B/32) with .005 (the red line). It indicates that the predictions of REPE are harder to be inverted with competing classes than the original CLIP, thus yielding better performance the open tasks.
F QUANTIFIED LOSS OF INTER-MODAL ALIGNMENT AND INTRA-MODAL UNIFORMITY
Table 7 provides quantified loss of aligmment and uniformity based on Eq. 8 and Eq.
9...9.6 defined in
$3. CLIP with only cross-modal contrastive learning results poor intra-modal uniformity. On the vision side, compared with ResNet- 101 (He et al. 2016),, using more powerful visual encoder such as VIT (Dosovitskiy al. 2021) can reduce the loss of image uniformity from -0.57 to -0.93. Besides, the Cuniform of SLIP Mu et al., 2021) and DeCLIP (Li et al.) (0221) is much lower than CLIP, indicating their better intra-modal uniformity derived by intra-modal contrastive learning in pre-training, which enforces both image and text features separate better from classes. As for the prompt tuning (Zhou et al. 2021 method (CoOp), achieves better lalign compared to CLIP and the lowest Euniform 3.16.
G CASE STUDY OF RETRIEVED CAPTION IN REPE
Table shows some cases of the captions retrieved by proposed REPE CIFAR100. They share the same target of interest with the original prompt, i.e., "a photo of a [CLASS]* " but provide the context in which the class name is located and thus have richer semantics. For example, given class like bridge, the retrieved captions describe its possible properties (c.g., "golden",, "wooded"), connections other objects (c.g., "over mountain river"), etc.. yielding more expressive and distinguishable text features of the class. However, REPE also recalls some low-quality captions. For example, given the class ray, a large, flat sea fish with a long, narrow tail, CIFAR100, the caption "Sun Rays Tours: Go Pro captured the rays under water" is retrieved, where the "ray" in the caption
*See Table in Appendix
17

---

Under review as a conference paper at ICLR 2023
Table 8: Instances of the captions retrieved by our REPE on CIFARIO0.
Class
Retrieved captions
"Apple slices stacked on top of each other"
apple
"Apples growing
tree"
"Still life with apples a basket"
"Portrait of a young woman"
woman
"Woman standing at the window"
"Confident woman in red dress and gold crown"
"The golden bridge in Bangkok" "Bridge on the River Kwai ~ Video Clip"
bridge
"Wooden bridge over mountain river"
"Stingray the Grand Cayman, Cayman Islands stock photography"
ray
"Common Stingray swimming close to the sea floor."
"Sun Rays Tours: Go Pro captured the rays under water"
Table 9: Accuracy of CLIP-Adapter and our REPE method on CIFARI00 and ImageNet datasets with few-shot learning.
Method
K-shot CIFAR100 ImageNet
CLIP-Adaa CLIP-Adapter
4
66.6
63.0
-.lP-Adapeer- REPE
67.5 (40.9)
63.3 (40.3)
CLIP-Adapter
16
69.0
64.6
CLIP-Adapter+ REPE
16
69.8 (+0.8)
64.9 (+0.3)
refers to a narrow beam of light, heat, etc. We leave the retrieval with better semantics preservation for future work.
H COMPUTATIONAL OVERHEAD OF REPE
The computational overhead of REPE is three-fold (measured a single RTX TITAN GPU): (1) Computing the text and image embeddings of CC12M dataset (Changpinyo et al., 2021) we used. takes about an hour with single RTX TITAN GPU. (2) Building indices out of the embeddings for KNN search with the FAISS framework (Johnson et al. 2019). The cost depends on the hyper- parameter setting for the target recall, and the procedure can be finished in an hour for our dataset.
(3) Retrieving the relevant images and corresponding captions for given query images. Once the indices are built,
efficiently search over the dataset according to the query image in real time (less than 5 ms for retrieving the top- 100 relevant image-caption pairs), which is thus still applicable for query-intensee intensive scenarios.
I INCORPORATION OF REPE TO DOWNSTREAM FINE-TUNING
Our REPE method is model-agnostic and orthogonal to parameter- sarameter-tuning methods. We can combine with fine-tuning methods like adapter-tuning (Gao al. 2021) (denoted as CLIPPadapter), to achieve further performance boost. Concretely, CLII--ddaper adds tunable adapter network (two linear layers with ReLU) after the last layer of both image and text encoder. Based on it, we ensemble the original class description ("a photo of a [CLASS]" ") with the captions retrieved by our REPE as the final textual input. The CLIP-Adapter is trained with K-shot training samples and then evaluated full test splits. We conduct experiments K=16, respectively. As shown in Table 9 REPE further improves the performance of adapter-tuning by an average of 0.6, which demonstrates the adaptability and superiority of our method.
CIFAR 100 and ImageNet with K=4 and
18

---

Under review as a conference paper at ICLR 2023
Table 10: comparison Closed Set Recognition, Open Set Recognition (OSR), Open World
Recognition, and Open-vocabulary Recognition (OVR).
Task
Paradigm
Goal
Signal
Training
Testing
Closed Recognition
Classification
Identifying known classes
Supervised
Known classes
Known classes
Open Set Recognition
Classification
Identilyngg known classes &
rejecting unknown classes
Supervised
Known classes
Known classes & unknown classes
Identifying known classes &
Open World Recognition
Classification
detecting unknown classes &
labeling unknown data
Supervised
Incremental known classes
Known classes & unknown classes
incremenially learn and extend classiller
Open-vocahulayy Recognition
Matching
Identifying classes via natural language Unsupervised
Classes vocahulary
J COMPARISON OF RELATED WORK
Table 10 provides more detailed comparison of Closed Set Recognition, Open Set Recognition (OSR) (Scheirer et al. 2013 Geng al. (02T). Open World Recognition (OWR) (Bendale & Boult 2015). and Open-vocabulary Recognition (OVR) (Radford et al. 2021)) from 5 perspectives paradigm, goal, signal, classes type in training, and classes type in testing, respectively. Contrary to other research, the CLIP-based Open-vocabulary Recognition is unsupervised. We focus on the model performance on zero-shot inference, without training or fine-tuning on the target dataset.
19

---

Under review as a conference paper at ICLR 2023
Adde Pat
Food
ACC0 o' Large carvhroe
ACCI Large FWESE GuRdOeT SNVEg
ACC- Lavge maRAAA subdoct SCHOOE
NCC4 of Large westroeeen wwd Soaare
PI0K reanvada
N004 COANNCI R.
Figure 8: Incremental Acc-E of CLIP and its variants on CIFAR100.
20

---

Under review as a conference paper at ICLR 2023
AC00 of Lange carvirone
ACC. LArge FWASO GURDONY Ohiige
ACC-5 of Largo IAIUASE SARAOO SCOOOE
Acod of Large wwwoee andd Siaare
AOUSC reanvada
Figure 9; Incremental Acc-S of CLIP and its variants on CIFAR100.
21

---

Under review as a conference paper at ICLR 2023
AdVernen ren torprrrgee -OCE
ocean Inar
arstion came
(Gkmup sucomoble
equiar)
58
Nood
aurorus rursor)
(endellacng
NiC4
Aodd
(a) CIFARIO
(b) CIFARIO0
Figure 10: Adversarial non-target vocabulary and the corresponding Acc-S CIFAR datasets.
Adding non con-target classes into the candidates leads to severe performance deterioration, revealing the vulnerability of CLIP when faced with malicious vocabulary.
Figure 11: Averaged cosine similarity between image (-I) and prompt (-T) features of CLIP (ViT- B/32) on intact CIFAR100. The elements on the diagonal represent the similarity of the positive image-text pairs, while the others represent that of the negative ones.
22

---

Under review as a conference paper at ICLR 2023
800
CLIP (ViT-B/32)
REPE
700
600
500
400
300
200
100
0
0.050 00.00 0.000 0.025 0.050 0.075 0.100
Margin
Figure 12: Margin distribution similarity scores of our REPE (blue) and CLIP (VIT-B/32) (red). The median value of REPE's distribution is .01 (the blue vertical linc), which is larger than that of CLIP (ViT-B/32) with .005 (the red line). It indicates that the predictions of REPE are harder to be inverted with competing classes than the original CLIP.
23


Reviews:
[{"review_id": "MrfoZLHZvU", "reviewer": "Reviewer_No3A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles three major issues with CLIP-like models: openness, extensibility and stability. The paper argues that CLIP-like models are difficult to evaluate since they are unconstrained in terms of the vocabulary use. Therefore the paper proposes to use an incremental evaluation perspective, called extensibility, to test the the models’ ability to learn new concepts.  \n\nThe paper also suggests that the ambiguity in openness is not due to the failure to capture image-text similarity, but rather confusion among similar textual features. For this, the paper proposes to use a retrieval-augmented relevant texts to impose distinction between competing texts which the paper shows improves the extensibility and stability of CLIP without fine-tuning.", "summary_of_the_review": "The paper does good job on highlighting and quantifying issue of open world recognition with CLIP-like models. However, there are other methods poised to be developed since it seems to be a known issue. Therefore, REPE giving marginal gains does not seem a strong base for future improvement. The experiments are sufficient and extensive for the given proposal. \n\nGiven many insights into the analysis with extensibility and stability, I believe that the paper has good potential and value to add to the community. \n\n--------------------------------------------------------------------------------------\nAfter reviewing the responses, I concur with other reviewers that the paper has low technical novelty. REPE is not very superior to other methods.\n\nHowever, I still believe that the paper is good work and could serve as a nice direction for future work. I will keep my rating to above acceptance threshold. ", "strength_and_weaknesses": "**Strength**\n\n1. The paper is one of the firsts to investigate the openness of CLIP-like models, with 2 evaluation measures: _extensibility_ and _stability_. And the paper proposes a method called REPE (Retrieval Enhanced Prompt Engineering) to tackle the two issues.  \n    \n2. The experiments demonstrate that the CLIP-like models are indeed susceptible to fail in visual recognition, even with marginal difference in text features of different classes.    \n    \n3. The experiments highlight that such models, which are decent zero-shot learners, do not scale well for larger vocabulary size.\n    \n\n**Weakness**\n\n1. What is the significance of using REPE, when the incremental advantages do not seem visually (Fig.7) or numerically (Tab. 2) considerable?\n    \n2. In addition to retrieving the K nearest image captions and filtering out the class name to add visual semantics in REPE, is it possible to use the bottom-K captions and recompute the accuracies (C, E, S)?  The motivation of the paper is quite clear, however I wonder how much would the accuracies suffer if given the worst captions. The increase in accuracies being marginal, this thought occurred to me.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HEk4F4TnFB6", "reviewer": "Reviewer_8DmZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors perform a detailed study of the \"openness\" of the CLIP image-text model and variants. Specifically, they study the ability of CLIP to generalize to new categories of objects in the zero-shot setting. For this, they propose an evaluation protocol that analyzes CLIP's performance along two axes - extensibility and stability. Extensibility refers to the ability of the model to deal with new object classes when the vocabulary of object classes is expanded. This means, having a set of target classes which is progressively grown in size by adding new categories. The measure of stability is interesting as it refers to keeping the set of target classes fixed, but adding a number of distractor classes that the model could potentially classify, but for which no images are actually present. Models which are stable are able to continue performing well on the set of target classes despite the distractors. The authors argue that stability may more realistically account for the model's performance in the real-world zero-shot setting.\nThe authors results show that CLIP models suffer from poor extensibility and stability in the real world. They show significant performance drops using three benchmarks (CIFAR100, ImageNet (Entity13) and ImageNet (Living17).\nTo explain the performance drops, the authors study the margin between positive and negative classes and observe this is the direct contributing factor to the model's poor stability. The authors discover classes which, when added to the set of options, cause significant misclassifications based on the similarity of the features. \nThe authors also study two other properties of the feature space. Specifically, the inter-modal alignment and intra-modal uniformity.\nInter-modal alignment is essentially just the average distance between the positive image and text pairs of embeddings.\nThe intra-modal uniformity basically samples pairs of samples from the same modality and takes their distance.\nThey show that existing models perform poorly on these measures.\nFinally, authors propose a new method for addressing these issues called REPE. The basic idea is to not use a standard text prompt like \"a photo of a [class]\" but to instead learn a class-specific text-embedding. To do this, given a prompt of the class name, the authors look in the original dataset used to train CLIP to find similar images. Then, they find the captions that correspond to those images. They encode the captions and then essentially just average them with some weighting.\nThe authors show that this approach improves the extensibility and stability of CLIP by 1.2% compared to not using it.", "summary_of_the_review": "The authors' paper makes a compelling case by pointing out key weaknesses of CLIP. They demonstrate this both quantitatively and qualitatively through illustrations in their paper. They include many supplemental results as well. However, the proposed method seems ad-hoc and doesn't necessarily directly address the problems pointed out. It also is of low novelty and results in only minor performance gains with considerable technical overhead. Thus, in my view the paper should not be accepted at this time.", "strength_and_weaknesses": "[Strengths]\nThe authors do a deep-dive on an important aspect of these large-scale vision-language models, i.e. how well do they actually generalize to their use-case. To do this, they examined two measures to account for this performance drop in the real-world - i.e. extensibility and stability. They also do a deep-drive into the feature spaces of these various methods using a number of metrics they develop and demonstrate quantitatively the problems with the feature spaces that their method tries to address.\n\nBeyond just evaluating the method and exposing the problem on CLIP's feature space, the authors also propose a method to improve CLIP's extensibility and stability. The proposed method is quite straightforward but generates consistent small gains over not using it.\n\n[Weaknesses]\nAs the authors cite in their paper, much past work has explored prompt tuning for CLIP. For example, in Table 1, we see CoOP (a prompt tuning method) achieves 76.7% while authors method, shown only with vanilla CLIP in Table 2 achieves 72.6%, for example. In other words, the CoOp approach works better than authors' method. Of course, CoOp requires access to the downstream target dataset. Yet, is this really a fair comparison, since in your approach we *do* have access to a dataset used to train CLIP. Why not use your idea to retrieve relevant images / text on the CLIP training dataset, if that's fair game?\nIn sum, the authors' approach is assuming that the large-scale CLIP pre-training dataset is available for querying. But if it is, perhaps some of these other methods could work better as well. In that scenario, it is unfair to compare just to vanilla CLIP.\n\nThe authors' method is also of low technical novelty and consists mainly of finding other captions in the train dataset and taking an average. Similarly, the proposed \"measures\" like the average distance between positive classes and the uniformity are quite simple. It is also unclear to me whether the intra-modal uniformity is a good measure. For example, it adds together the average distance between image pairs and text pairs. But these are two embedding spaces learned by two different models. Shouldn't these be normalized in some way so the two distributions (image and text embedding space) is directly comparable? For example, right now one modality might dominate. Thus, there is very little technical novelty to authors' approach.\n\nSimilarly, while authors have pointed out important problems, it isn't clear to me that the proposed approach is well-motivated to solving them. For example, why does taking the average caption solve these problems? Perhaps a better training paradigm, different representation space (e.g. hyperbolic), etc. is the correct way to address the problems the authors have pointed out, rather than just taking the average caption. Thus, it seems that the authors point out a clear problem, but that the proposed method seems ad-hoc and unrelated to the action problem. Perhaps a better way to handle it would be to learn some class-agnostic projection that solves these problems or something.\n\nI'll also point out that the author's method requires this additional step of keeping the CLIP training dataset, indexing it, and doing this class-averaging step. Thus, there is this unnatural overhead to using their method for only around a 1.2% gain in performance. It's not clear that the overhead justifies its use.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Espc_criLe", "reviewer": "Reviewer_c4Um", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduced a new concept extensibility, to explore the openness of CLIP on visual recognition task. Two new evaluation metrics Acc-E and Acc-S were proposed to quantify CLIP's extensibility and stability. The paper argued the confusion between text embeddings of different classes made the CLIP unstable in extensibility, and finally a new prompt engineering module, REPE was proposed to handle this issue.", "summary_of_the_review": "The motivation of the paper is important, to evaluate the openness of CLIP in visual task. However, the experiment design is not reasonable and the proposed prompt engineering cannot address reported issue in the paper either. Overall speaking, the paper is below the bar of acceptance.", "strength_and_weaknesses": "The paper is easy to understand and the topic of evaluating the openness of CLIP in visual task is good and important. However there are some concerns for me to accept the paper.\n- For Sec 2.2, the author proposed ACC-E to validate the extensibility of the CLIP by evaluate on different permutation of the vocabulary set.  I do not fully understand the principle and advances of the proposed evaluation mechanism. A more straightforward way is to apply the CLIP directly in different datasets with different vocabulary sizes and report the average score, especially for some uncommon classes which may not been covered by the pertaining stage of CLIP, e.g., the local food dishes in different countries.  \n- For Sec 2.2 and Table 1, the openness issue is reported in the small dataset CIFAR100. With larger dataset imageNet and stronger CLIP backbone, the gap between Acc-E/Acc-S and Acc-C reduces significantly (~13% -> ~3%). In real world, the class number is far more than ImageNet and the gap may potentially become even more marginal.\n- For Sec 2.3, it's quite reasonable in visual recognition task that involving more classes, the overall performance will drop due to the class confusion.\n- For Table 2, the results fail to validate the effectiveness of REPE to address the openness issue. The gap between Acc-E/Acc-S and Acc-C remains unchanged for both vanilla prompt design and REPE, except REPE reports higher results. REPE is a prompt engineering module and thus it's reasonable the results can improve. However, the main topic of the paper is not the prompt engineering but the openness issue of CLIP, and this issue still remains. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9_KRhnB1ye", "reviewer": "Reviewer_hc5X", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The goal of this study is to measure the “oppeness” of CLIP-based models. The authors are adding class names that are not part of the target dataset domain and evaluating the change in accuracy from that. They then show how poorly models do and propose a new approach to this via two metrics: inter-modal alignment and intra-modal uniformity.The authors also perform prompt-engineering by generating a better search algorithm from the original dataset.", "summary_of_the_review": "Overall, it felt like there is too much focus on the underlying problem and not enough on their proposed approaches REPE. The reader is left with only questions.\n\nIf the paper is re-focused with more emphasis on the why and how this differs from other prompting approaches, and certain concepts clarified, this paper would have more clarity.\n", "strength_and_weaknesses": "**Strengths**\n\n- The authors provide in-depth empirical study to analyze the performance of CLIP on novel concepts.\n\n- The authors are studying an interesting area that is relevant and useful to the community.\n\n\n**Weaknesses**\n\n- The main weakness of that the paper is that it is very confusing and does not flow correctly. Unfortunately, I think it is because there is too much packed in. The first half of the paper addresses the problem the authors are trying to solve in great detail but was confusing because some terms were undefined and the purpose of this analysis was not necessarily clear. How does it differ from text-to-image retrieval? Is this all zero-shot, in which case, why are we interested if the model performs less accurately on a fixed, target dataset if we add some classes not relevant to the target dataset? How was the superclass hierarchy used? This is all a lot of experimentation but not clearly laid out. \n\n- Then the last ¼ of the paper is a proposed approach and it felt very quick and rushed. It was very difficult to get the full grasp of the change or the approach that was used.\n\n**More details:**\n\n  - Terms are not properly defined, like “vocabulary extension”. This could mean many things, for example, the actual text-encoder has an embedding space with a fixed vocabulary size, is this what the authors are extending? If so, should models using a BERT-like text encoder not be compared since it uses sub-words during tokenization?\n\n  - It is confusing how this task varies from text-to-image/image-to-text retrieval when you have an image and several text pairs and you are measuring the accuracy of that retrieval. Is this task not just taking a classification problem and making it more like a retrieval problem because the classes are no longer fixed? If the initial analysis is zero-shot, than this difference should probably be discussed as it would also evaluate the “openness” without claiming that the model performs poorly on this fixed classification datasets. Furthermore, given this is zero-shot learning, how is that not evaluating “oppenness” if it has not seen these images and text pairs during training? These concepts should be clarified and explained.\n  \n  - Using some kind of hierarchical superclass organization sounds very interesting but is not clearly explained. \n\n  - The REPE approach definition mentions KNN and FAISS in one sentence, but does not provide enough information so it is confusing how this actually works.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject