Delving into the Openness of CLIP
Year: 2023

Abstract:
Contrastive Language-Image Pre-training (CLIP) formulates image classification as an image-to-text matching task, i.e., matching images to the corresponding natural language descriptions instead of discrete category IDs. This allows for open-vocabulary visual recognition, where the model can recognize images from an open class set (also known as an open vocabulary) in a zero-shot manner. However, evaluating the openness of CLIP-like models is challenging, as the models are open to arbitrary vocabulary in theory, but their accuracy varies in practice. To address this, we resort to an incremental perspective to assess the openness through vocabulary expansions, and define extensibility to measure a model’s ability to handle novel classes. Our evaluation shows that CLIP-like models are not truly open, and their performance deteriorates as the vocabulary expands. We further dissect the feature space of CLIP from the perspectives of representation alignment and uniformity. Our investigation reveals that the overestimation of openness is due to confusion among competing text features, rather than a failure to capture the similarity between image features and text features of novel classes. We hope that our investigation and analysis will facilitate future research on the CLIP openness issue.1

heading: 1 Introduction
text: An intrinsically open mechanism for visual recognition (Deng et al., 2009; He et al., 2016) has always been a shared goal in the computer vision community (Scheirer et al., 2013; Geng et al., 2021; Bendale and Boult, 2015). This mechanism requires models to maintain flexibility to cope with the scaling of the recognition target, where both input images and the corresponding classes will dynamically expand according to actual needs. For example, in medical diagnosis (Razzak et al., 2017), new
1Our code is available at https://github.com/ lancopku/clip-openness
diseases emerge constantly, and in e-commerce, new categories of products appear daily (Xu et al., 2019), which cannot be predefined in a finite, fixed class set.
Faced with the challenging task of open-world recognition, Contrastive Language-Image Pretraining (CLIP) (Radford et al., 2021) and its openvocabulary learning paradigm demonstrate superiority over traditional supervised classifiers (He et al., 2016; Dosovitskiy et al., 2021). CLIP pretrains a vision-language model on web-scale collections of image-text pairs, learning semantic alignment between images and corresponding textual descriptions. During inference, it formulates image classification as an image-to-text matching task, where the set of class names serves as a vocabulary, and textual prompts like "a photo of a [CLASSNAME]" are curated as class descriptions for images. By varying the [CLASSNAME] placeholder and computing the similarity between class descriptions and images, CLIP can identity the most suitable class name and predict it as the target class. This approach allows CLIP to operate with arbitrary vocabularies and adapt to novel classes by expanding the vocabulary, enabling zero-shot inference for new input images and classes.
Nevertheless, previous evaluation protocols for CLIP models only assess their accuracy on static, closed vocabularies from downstream datasets, leaving their actual performance on open tasks in the shadows (Radford et al., 2021). In this work, we delve into openness, the intriguing yet underexplored property in CLIP-like models (Li et al., 2021b; Mu et al., 2021; Yao et al., 2021; Zhou et al., 2021), and present a novel protocol for evaluating the openness from an incremental view. Specifically, we define a metric of extensibility to measure a model’s ability to handle new visual concepts through vocabulary expansion. Different from previous metrics, our metric explicitly models the dynamics of the real open world, and formulates the
9587
empirical risk of CLIP when new vocabularies incrementally emerge. Additionally, we define a metric of stability to explore how stable the model’s predictions are for old classes when new classes are introduced, which provides a tool to analyze the compatibility between different classes.
Using our protocol, we conduct a systematic and comprehensive evaluation of CLIP-like models. Our experimental results based on extensibility show that CLIP and its variants have a significant drop in accuracy as the vocabulary size increases. For example, CLIP (RN101) on CIFAR100 experiences a 12.9% drop in accuracy when the vocabulary size expands from 5 to 100. This indicates that the limited zero-shot capability of CLIP-like models is inadequate for supporting their deployment in the open world. What’s worse, through an analysis of the prediction shift during vocabulary expansion, we find that the performance of CLIP can be dramatically reduced by adding only three adversarial class names into the vocabulary, exposing the model’s poor stability and security risks. Furthermore, we investigate the representation space of CLIP-like models via three metrics: margin, inter-modal alignment, and intra-modal uniformity. Our results show that the small margin between positive and negative class descriptions leads to prediction shifting when competing class features appear. Therefore, enforcing the distinguishability of class features increases the margin and improves the stability of these models.
In summary, our contribution is threefold: First, to the best of our knowledge, we are the first to systematically quantify the openness of CLIP, for which we design the evaluation protocol and two indicators of extensibility and stability. Second, we conduct extensive experiments on CLIP-like models based on our protocol and find that their openness is overestimated and their performance declines as the vocabulary expands. Finally, we analyze the feature space of CLIP from the perspectives of representation alignment and uniformity, observing that the uniformity of the textual space is critical for better extensibility.

heading: 2 Related work
text: Contrastive language-image pre-training and open-vocabulary learning. CLIP (Radford et al., 2021) introduces the paradigm of open-vocabulary learning and learns transferable visual models from natural language supervision. The CLIP model con-
sists of an image encoder and a text encoder, which are utilized to encode image-text pairs into a joint feature space for learning the semantic alignment of vision and language. The paired images and texts are pulled together in the feature space, while the others with dissimilar semantics are pushed apart via a contrastive loss. After pre-training on large-scale image-text pairs, CLIP is able to map images to their corresponding language descriptions, which makes visual recognition generalize in the wild. Recent studies further improve CLIP by using more pre-training data (Jia et al., 2021), incorporating self-supervision (Mu et al., 2021), fine-grained supervision (Yao et al., 2021), and widespread supervision (Li et al., 2021b) to pretraining. Another line of recent studies (Li et al., 2021a; Wang et al., 2022; Yu et al., 2022; Alayrac et al., 2022) adopts seq2seq generation instead of contrastive discrimination framework to achieve open-vocabulary recognition. We leave the investigation of their extensibility for future work.
Open Set and Open-World Visual Recognition. Open Set Recognition (OSR) (Scheirer et al., 2013; Geng et al., 2021) and Open World Recognition (OWR) (Bendale and Boult, 2015) are paradigms aiming to cope with input images from novel classes during inference. OSR requires classifiers to identify images that have not been introduced during training as “unknown”. While OWR raises higher demands, models are supposed to incrementally extend and retrain the multi-class classifier as the unknowns are labeled as additional training data. Contrary to the above research, CLIP-based Open-vocabulary Recognition (OVR) aims to identify novel classes in a zero-shot manner by using natural language representations of categories instead of discrete label IDs. This allows CLIP to directly synthesize textual descriptions of novel classes for matching, eliminating the need for relabeling additional training data and re-training the entire model. A more detailed comparison of OSR, OWR, and OVR can be found in Appendix A.1.

heading: 3 Openness, Extensibility, and Stability
text: In this section, we first review CLIP’s visual recognition paradigm and demonstrate how it realizes open-vocabulary image classification through vocabulary expansion (§ 3.1). To quantify the actual performance of CLIP-like models as the vocabulary expands, we define the metric of extensibility and propose a systematical evaluation protocol (§ 3.2).
The experimental results and further analysis reveal that, as the vocabulary expands, CLIP’s predictions become unstable and prone to drift towards newly introduced competing class descriptions, which limits its extensibility and poses a huge security risk when deployed in real-world applications (§ 3.3).

heading: 3.1 Openness of CLIP
text: CLIP (Radford et al., 2021) models image classification as an image-to-text matching task. Formally, let f be the CLIP model, fT and fI be the text and image encoders in CLIP, respectively. The CLIP model takes an image x and a target vocabulary V(T ) = {wi} of the class names wi as inputs, and predicts the image label as:
f ( x,V(T ) ) = argmax
i P (y = i | x)
= argmax i
esim(fT (ti),fI(x))
|V(T )|∑ j=1 esim(fT (tj),fI(x))
,
where ti is the textual description of the class name wi in a prompt format, e.g., “a photo of a wi”, and sim(·, ·) denotes cosine similarity. Such a modeling paradigm can realize open-world image classification in theory by extending the target vocabulary V(T ) to arbitrary degrees. However, in most previous work (Radford et al., 2021; Li et al., 2021b; Mu et al., 2021; Yao et al., 2021; Zhou et al., 2021), CLIP is evaluated with a fixed vocabulary
V(T ) of a downstream dataset D(T ):
Acc ( V(T ) ) = 1 |D(T )| ∑ (x,y)∈D(T ) I ( f ( x,V(T ) ) = y ) ,
(1)
where |D(T )| is the size of the dataset and I(·) is the indicator function. This vanilla evaluation setting, utilizing restricted input images and classes, falls short for open recognition tasks. It fails to consider the dynamic expansion of vocabulary during inference and, as a result, cannot accurately reflect CLIP’s openness in real-world scenarios where the number of classes may increase.

heading: 3.2 Quantifying extensibility for open world
text: To quantify the model’s capability in dealing with newly emerged recognition targets, we propose an evaluation protocol and define a metric of extensibility based on vocabulary expansion. Concretely, we incrementally expand the vocabulary V(T ) in Eq.(1) by introducing new classes and their associated input images, then evaluate the accuracy after each expansion. These accuracy values reflect the model’s dynamic performance as openness increases, and the expected average of these values is defined as the model’s extensibility. In practice, we achieve this expansion by incrementally unioning N disjoint target vocabularies2 as shown in the
2Since V(T ) is bound with D(T ) in Eq.(1), expanding the target vocabulary also implies expanding D(T ) (including input images and their labels) at the same time, which we omit for brevity.
upper right panel of Figure 1.
Definition 3.1 (Extensibility). Given N disjoint target vocabularies {V(T )1 , · · · ,V (T ) N }, we denote the set of all possible permutations of these vocabularies as SN , and V(T )si as the i(th) vocabulary in a permutation s ∈ SN . When we union the i(th) vocabulary with the previous i− 1 vocabularies, we achieve a vocabulary expansion and obtain V(T )s1 ∪ · · · ∪ V(T )si . The extensibility refers to the averaged classification accuracy across N incremental expansions as i increases from 1 to N :
Acc-E = E s∈SN
1
N
N∑
i=1
Acc ( V(T )s1 ∪ · · · ∪ V(T )si ) .
(2)
Experimental settings We evaluate the extensibility of CLIP and its variants, including DeCLIP (Li et al., 2021b), SLIP (Mu et al., 2021), Prompt Ensemble (Radford et al., 2021), CoOp (Zhou et al., 2021), on the CIFAR100 (Krizhevsky and Hinton, 2009) and ImageNet (Deng et al., 2009) datasets. Non-matching methods (Gao et al., 2021; Zhang et al., 2021; Wortsman et al., 2021), such as linear probing, are NOT included since they train a classifier with finite class vectors, and thus are not suitable for class scaling in operation. To construct the vocabulary, we leverage the underlying superclass-class hierarchical structure of the two datasets (Krizhevsky and Hinton, 2009; Santurkar et al., 2021) by grouping classes that belong to the same superclass into a vocabulary. Accordingly, CIFAR100 has 20 vocabularies, each with 5 classes. For ImageNet, we
utilize two superclass-class structures (Santurkar et al., 2021): Entity13 and Living17. The former has 13 vocabularies, each with 20 classes, while the latter has 17 vocabularies, each with 4 classes. Tables in the Appendix A.2 list all the vocabularies in the two datasets. For each dataset, we calculate Acc-C, the averaged classification accuracy across all single vocabularies, based on Eq.(1):
Acc-C = 1
N
N∑
i=1
Acc ( V(T )i ) . (3)
It represents the original model performance on closed vocabularies. To calculate the expectation in Acc-E, we sample 100×N permutations for N vocabularies and take the average.
Results As shown in Table 1, all models exhibit a clear drop in performance as the vocabulary expands. The accuracy of CLIP (RN101) after vocabulary expansion (Acc-E) sharply decreases by 12.9% compared to the accuracy on closed vocabulary (Acc-C). The performance on the data splits in ImageNet is relatively better, with an average decline of 2.7%. Appendix A.3 provides results of expansion at the dataset level, where the expanded vocabularies are from five other datasets. These results show a more dramatic decline of an average of 15.3% on generic dataset expansion. It demonstrate that the openness of CLIP-like models is overestimated under the vanilla evaluation mechanism. Besides, there are some interesting findings: (1) From the perspective of pre-training, introducing a stronger vision backbone (ViT (Dosovitskiy et al., 2021) vs. ResNet (He et al., 2016)),
widespread supervision (DeCLIP (Li et al., 2021b) vs. CLIP), and more pre-training data (CLIP vs. SLIP (Mu et al., 2021)) can improve the extensibility of models on open tasks. (2) During inference, the performance of CLIP can be boosted by ensembling different prompts. (3) The most extensible results are obtained by CoOp (Zhou et al., 2021), which performs prompt tuning on all classes of CIFAR100 and ImageNet. However, the prompt tuning method utilizes the additional category information and training data, which cannot be applied to real-world open tasks.

heading: 3.3 Stability during vocabulary expansion
text: As the vocabulary expansion introduces new classes incrementally, some images belonging to previous vocabularies may be incorrectly predicted as new classes, resulting in a drop in accuracy and poor extensibility. To analyze the prediction stability of CLIP during vocabulary expansion, we introduce the non-target classes. They do NOT correspond to any input images, and only serving as distractors for the target classes. Based on it, we define conditional classification accuracy as:
Acc ( V(T ) ∣∣∣V(T ) ∪ V(NT ) )
= 1 |D(T )| ∑ (x,y)∈D(T ) I ( f ( x,V(T ) ∪ V(NT ) ) = y ) , (4)
where V(NT ) is the non-target vocabulary, i.e., the vocabulary of non-target classes. The conditional
accuracy is depicted in the left panel of Figure 1. In Eq.(4), the categories of the input images are limited to the target vocabulary ((x, y) ∈ D(T )), but CLIP is asked to distinguish all categories from a larger vocabulary V(T ) ∪ V(NT ). In other words, compared to traditional closed-set classification, CLIP is expected to reject all the negative categories from V(NT ). The model is required to distinguish visual concepts stably and robustly, rather than making wrong predictions in the presence of other distractors. Based on Eq.(4), we define the stability of CLIP in the open task as:
Definition 3.2 (Stability). Given a target vocabulary V(T ) and M non-target vocabularies {V(NT )1 , · · · ,V (NT ) M }, we denote SM as their full permutation, and V(NT )si as the i(th) vocabulary in a permutation s ∈ SM . We design the local stability to measure the averaged classification accuracy of CLIP on the given target vocabulary when nontarget vocabularies are extended incrementally: Acc-S̃ ( V(T ),V(NT ) ) =
E s∈SM
1
M
M∑
i=1
Acc ( V(T ) ∣∣∣V(T ) ∪ ( V(NT )s1 ∪ · · · ∪ V (NT ) si )) .
(5)
As Eq.(5) only reflects the local stability with respect to a single target vocabulary, we further design the general stability as an average of local stability over a set of target vocabularies to reduce the bias from data distribution and vocab-
ulary sampling. Specifically, given N vocabularies {V1, · · · ,VN}, we regard each vocabulary Vi as the target vocabulary V(T ) and the rest V̸=i as the non-target vocabularies V(NT ), and then formulate the general stability as:
Acc-S = 1
N
N∑
i=1
Acc-S̃ (Vi, V̸=i) . (6)
Experimental settings and results The models and datasets adopted for evaluation are consistent with that in § 3.2. For the calculation of stability, take CIFAR100 with N = 20 vocabularies as an example, we treat each vocabulary as the target vocabulary and the rest are treated as the non-target vocabularies. To calculate the expectation in Eq.(5), we sample 100 permutations for M = 19 nontarget vocabularies and report the averaged scores.
Table 1 demonstrates the stability of CLIPlike models. On CIFAR100, the Acc-S of CLIP (RN101) decreased by 13.4%. Figure 2a shows Acc-S on CIFAR100 during non-target vocabulary expansion. Given a closed V(T ) = Insects, CLIP (ViT-B/32) achieves an accuracy of 81.2%. However, when the remaining 19 non-target vocabularies are incorporated, the accuracy sharply drops to 57.0%. The decrease of Acc-S brought by the introduction of each non-target vocabulary indicates that more images from Insects are incorrectly classified into the new vocabulary. Figure 2b demonstrates the difference between Acc-C and Acc-S for each target vocabulary. When V(T ) = Medium-sized Mammals, CLIP is most easily interfered with by the non-target vocabularies, with a 21.08% performance drop. It suggests that the unstable predictions lead to the poor extensibility of CLIP when new categories are introduced. Besides, we notice that CLIP performs stably on groups like Flowers, where its Acc-S only declines by 0.53% compared to Acc-C. The different behaviors of different groups indicates that the stability
is also influenced by the inherent property of the image categories and naming variation (Silberer et al., 2020; Takmaz et al., 2022).

heading: 3.3.1 Adversarial non-target vocabulary
text: In order to explore the lower bound of the stability of CLIP, we define the adversarial non-target vocabulary V(ANT ) as the non-target vocabulary that reduces Acc-S the most:
V(ANT ) = min V(NT )
Acc ( V(T ) ∣∣∣V(T ) ∪ V(NT ) ) .
(7) To build V(ANT ), we refer to the method of adversarial examples generation (Ren et al., 2019) to traverse the words in a large vocabulary, e.g., the vocabulary of nouns in WordNet (Fellbaum, 2000), which are regarded as non-target classes in order to calculate Acc-S, and then take the most confusing words to form the adversarial non-target vocabulary.
We constrain the size of V(ANT ) to 3. Results in Figure 3 illustrate the performance with nouns in WordNet and class names in ImageNet as the candidate vocabulary, respectively. First, we observe a clear performance degradation on both datasets under adversarial attack, e.g., adding bitmap, automobile insurance and equidae leads to an absolute 52.7% accuracy drop on CIFAR10. Besides, we find that the selected adversarial words are much less concrete than common visual concepts like Flower, indicating the potential reason behind is the poor semantic modeling of CLIP on those objects with higher abstraction levels. This investigation reveals that CLIP is vulnerable when facing malicious non-target vocabulary, and we hope future work may pay more attention to the robustness of CLIP under open recognition tasks.

heading: 4 Dissecting the extensibility of CLIP
text: Our experimental results in § 3 reveal the poor performance of CLIP on open tasks. In this section,
0.10 0.15 0.20 0.25 0.30 0.35 Cosine Similarity
0
100
200
300
400
500 neg pos
Figure 5: Cosine similarity histogram of positive (pos) and negative (neg) image-text pairs with large overlap.
we delve into the representation space of CLIP to understand its extensibility. We first point out that the small margin between positive and negative class descriptions leads to the prediction shifting when competing class features appear, which thus limits the stability of CLIP (§ 4.1). Further, we investigate the representation space of CLIP-like models via two metrics: inter-modal alignment and intra-modal uniformity. The results show that enforcing the distinguishability of class features increases the margin and makes the models scale more stably (§ 4.2).

heading: 4.1 Small margin limits the stability of CLIP
text: Since CLIP formalizes the visual recognition as an image-to-text matching task, each text feature of the class description corresponds to the class vector in traditional classifiers, and the image-text similarity scores are analogous to the logits in classification. Ideally, regardless of vocabulary expansion, for an image, the similarity of the positive pair (the image with the text specifying the ground-truth class) should be higher than that of the negative pairs (the image with the texts specifying other
classes) to ensure the correct prediction on open tasks. In other words, the margin (Jiang et al., 2019) between positive and the largest negative similarity is a direct contributor to stability.
Unfortunately, the similarity and margin distribution of CLIP do not meet our expectations. Figure 4 illustrates the averaged cosine similarity of CLIP (ViT-B/32) on 15 classes of CIFAR100. The diagonal elements represent the similarity of the positive image-text pairs, while the others represent that of the negative ones. In general, the cosine similarity of image-text pairs is very low, with an average of 0.20. This number is only 0.26 even for the positive pairs. Besides, the similarities of positive and negative pairs are very close, indicating the low distinguishability between different classes. As shown in Figure 5 and Figure 6, the similarity histogram of positive and negative pairs has a large overlap, and the margin is clustered around zero, leaving the predictions of models at risk of being reversed to new non-target classes. For example, as the vocabulary extends from the red box to the green box (diagonal) or the yellow box (horizontal) in Figure 4, more deceptive classes (circles)
with negative margins are added, leading to prediction shift. Particularly, the classes belonging to the same vocabulary3 have higher similarity and smaller margin, making them more likely to be confused with each other.

heading: 4.2 Inter-modal alignment and intra-modal uniformity ground the margin
text: According to the results in § 4.1, the ideal feature space for CLIP-like models should have a large margin between different classes to ensure stability in open-vocabulary recognition tasks. To achieve this, the text feature of a class name should be close to the features of the images it describes (Ren et al., 2021), and the intra-modal features, especially textual features, should be uniformly distributed to make the descriptions of competing categories more distinguishable (Wang and Isola, 2020). In order to measure the quality of representations in the vision-and-language domain, we propose two metrics, inter-modal alignment and intra-modal uniformity. Inter-modal alignment calculates the expected distance between features of positive image-text pairs ppos :
ℓalign ≜ E (x,t)∼ppos ∥fI(x)− fT (t)∥2 , (8)
while intra-modal uniformity measures how well the image or text features are uniformly distributed:
ℓuniform ≜ℓuniform-I + ℓuniform-T ≜ log E
xi,xj∼pdata-I e−2∥fI(xi)−fI(xj)∥ 2 +
log E ti,tj∼pdata-T
e−2∥fT (ti)−fT (tj)∥ 2 ,
(9)
3Every 5 adjacent classes in Figure 4 constitute a vocabulary (superclass), see Table 4 in Appendix A.2
where pdata-I and pdata-T denotes the image and text data distribution, respectively. Figure 7 and Table 2 provide quantified loss of alignment and uniformity. CLIP with only cross-modal contrastive learning results in poor intra-modal uniformity (ℓuniform > −2.0), especially on the text side. However, models like SLIP and DeCLIP that incorporate intra-modal contrastive learning in pre-training can better separate image and text features by classes, resulting in a much lower intra-modal uniformity loss (ℓuniform < −4.5). Additionally, the prompt tuning method (CoOp (Zhou et al., 2021)) achieves better inter-modal alignment and the lowest intra-modal uniformity loss on the text side. According to the visualization via Multidimensional Scaling (MDS) (Borg and Groenen, 1997) in Figure 8, the optimization trajectory of prompts in CoOp leads to the cluster center of corresponding image features while also dispersing the position of prompt features, thereby improving both text uniformity and inter-modal alignment and achieving the best extensibility.

heading: 4.3 Discussions
text: After the preliminary explorations on openness of CLIP-like models, we present potential ways to enhance the models’s extensibility and stability.
(1) For pre-training: In order to improve the quality of CLIP’s feature space and enhance alignment and uniformity, more high-quality pre-training data and effective supervision signals such as ℓalign and ℓuniform can be introduced during pre-training.
(2) For zero-shot inference: Recall that in vanilla CLIP-like models, the context (hard prompt) for
each class name is the same during inference, making it difficult to discriminate between distinct visual categories because the semantics of each cannot be holistically represented. To remedy this, we suggest customizing class descriptions with diverse captions retrieved from the pre-training corpus as a prompt ensemble. The effectiveness of this idea is verified through experiments, details can be found in Appendix A.5.

heading: 5 Conclusion
text: In this paper, we evaluate the extensibility of CLIPlike models for open-vocabulary visual recognition. Our comprehensive study reveals that as the vocabulary expands, the performance of these models deteriorates significantly due to indistinguishable text features among competing classes. We hope that our investigation and analysis will facilitate future research on the CLIP openness issue.
Limitations
To facilitate future research, we analyze the difficulties and possible solutions in this new area. (1) As we present extensive empirical results and address the weakness of CLIP on vocabulary expansion, its theoretical risk on open tasks is urged to be investigated. (2) The current evaluation protocol is an approximation of the real open world. An evolving benchmark could facilitate future research. (3) For various visual categories, their degree of abstraction, the ease of describing them in natural language, and their density in the data distribution can also influence the extensibility and stability of models, which are worth studying.

heading: Acknowledgement
text: The authors would like to thank the reviewers for their helpful comments. This work is supported by Natural Science Foundation of China (NSFC) No. 62176002. Xu Sun is the corresponding author.

heading: A Appendix
text: A.1 Comparison of related work
Table 3 provides a more detailed comparison of Closed Set Recognition (OSR) (Scheirer et al., 2013; Geng et al., 2021), Open World Recognition (OWR) (Bendale and Boult, 2015), and Openvocabulary Recognition (OVR) (Radford et al., 2021) from 5 perspectives of paradigm, goal, signal, classes type in training, and classes type in testing, respectively. Contrary to the above research, CLIP-based OVR aims to identify novel classes in a zero-shot way. Since categories of images in CLIP are represented by natural language rather than discrete label IDs, CLIP can directly synthesize textual descriptions of novel classes for matching, sparing relabeling additional training data and re-training the entire model.
A.2 Superclass-class hierarchy for vocabulary construction
To construct the vocabularies in § 3, we leverage the underlying superclass-class hierarchical structure of CIFAR100 (Krizhevsky and Hinton, 2009) and ImageNet (Deng et al., 2009), and group the classes belonging to the same superclass into a vocabulary. Table 4 lists the vocabularies in CIFAR100, which are specified by (Krizhevsky and Hinton, 2009). There are 20 vocabularies, each with 5 classes. For ImageNet, we utilize two superclass-class structures, Entity13 and Living17 (Santurkar et al., 2021), as shown in Table 5 and Table 6, respectively. Entity13 has 13 vocabularies, each with 20 classes, while Living17 has 17 vocabularies, each with 4 classes.
A.3 Dataset-level extensibility
The evaluation protocol in § 3 estimates the extensibility and stability within a single task dataset,
Task Paradigm Goal Signal Training Testing
Closed Set Recognition Classification Identifying known classes Supervised Known classes Known classes
Open Set Recognition Classification Identifying known classes & rejecting unknown classes
Supervised Known classes Known classes & unknown classes
Open World Recognition Classification
Identifying known classes & detecting unknown classes &
labeling unknown data & incrementally learn and extend classifier
Supervised Incremental known classes Known classes & unknown classes
Open-vocabulary Recognition Matching Identifying classes via natural language Unsupervised - Classes in a vocabulary
Table 3: A comparison of Closed Set Recognition, Open Set Recognition (OSR), Open World Recognition, and Open-vocabulary Recognition (OVR).
where the input images and classes during the vocabulary expansion come from the same data distribution. While the protocol is only an approximation of the real open world, current CLIP-like models have exhibited serious performance degradation. In this section, we take a step further toward real open recognition by conducting a vocabulary expansion setting at the dataset level, where the expanded vocabularies are from different datasets. In this way, the relationship between vocabularies is more uncertain and thus can be viewed as a rigorous stress test for the CLIP-like models. Specifically, we group all categories in a dataset into one vocabulary. Afterward, the inputs and classes of the entire new dataset are introduced at each expansion. Classes in the new vocabulary will be removed if they already exist in the previous vocabularies.
The experiments are conducted with datasets for generic objects, including CIFAR10 (Krizhevsky and Hinton, 2009), CIFAR100 (Krizhevsky and Hinton, 2009), Caltech101 (Fei-Fei et al., 2004), SUN397 (Xiao et al., 2010) and ImageNet (Deng et al., 2009), and specialized datasets focusing on fine-grained categories, including Flowers102 (Nilsback and Zisserman, 2008), OxfordPets (Parkhi et al., 2012) and StanfordCars (Krause et al., 2013). Without loss of the generality, we merge 3 datasets and evaluate the following dataset compositions:
(1) CIFAR100-Caltech101-SUN397
(2) CIFAR10-CIFAR100-ImageNet
(3) Flowers102-OxfordPets-StanfordCars
Composition (1) and (2) probe the performance when all the expanded datasets are generic thus the classes in different datasets are potentially semantic-correlated, while the composition (3) targets at scenarios where the coming datasets have little correlation with previous ones. To eliminate
the effect of vocabulary expansion order, we report the average performance of all A33 = 6 possible trials for each composition.
Table 7 demonstrates the result of the datasetlevel expansion. First, the performance of CLIPlike models on generic dataset expansion drops dramatically. For example, the accuracy (Acc-E) of CLIP (RN101) decreases by an averaged absolute point of 14.2 on the CIFAR100-Caltech101SUN397 composition during expansion, and 14.5 on the CIFAR10-CIFAR100-ImageNet composition. Due to the existence of subclass-superclass relationship for some classes in different generic datasets, e.g., cat in CIFAR10 and tiger cat in ImageNet, CLIP is extremely unstable on such expansion across generic datasets. For example, the AccS of CLIP (RN101) on the CIFAR10-CIFAR100ImageNet composition is 28.2% lower than Acc-C, indicating the models are prone to be confused about the subclass-superclass relationship. Meanwhile, the CLIP-like models exhibit much better extensibility and stability on the dataset-level expansion across specialized datasets, e.g., the Flowers102-OxfordPets-StanfordCar composition. The vocabularies of this composition are intrinsically disjoint in semantics, so the model can be stably extended. In summary, our investigations on the dataset level expansions along with the task level in the paper show the current CLIP-like models fail to meet the expectation of conducting real open vocabulary recognition.
A.4 Incremental Acc-E and Acc-S on CIFAR100
We record the Acc-E (Eq.(2)) and Acc-S (Eq.(5)) after each vocabulary expansion on CIFAR100 to investigate the openness of CLIP-like models.
Figure 10 shows the Acc-E for 20 trials as new vocabularies are merged incrementally. The falling
lines indicate that the model is either performing poorly on the new input images, or that some images that were correctly identified before are misclassified after introducing the new classes.
Figure 11 shows Acc-S of CLIP-like models during non-target vocabulary expansion. Each subfigure represents the situation when one vocabulary is selected as the target vocabulary. As the remaining 19 non-target vocabularies are incorporated and the model is required to recognize the 5 target classes from 100 potential classes, the accuracy drops sharply. The decrease of Acc-S brought by each introduction of non-target vocabulary indicates that more images from the target vocabulary are incorrectly classified into the new non-target vocabulary by models.
A.5 Retrieval-enhanced prompt engineering
In light of the previous investigations, we propose a simple yet effective method named Retrievalenhanced Prompt Engineering (REPE) to enforce the distinguishability of class features and the image-class semantic alignment (Cao et al., 2020; Ren et al., 2021). Recall that the context for each class name is the same in vanilla CLIP-like models (e.g., “a photo of a [CLASSNAME]”), making it difficult to discriminate between distinct visual categories because the semantics of each cannot be holistically represented (Zhou et al., 2022).
To remedy this, we propose to customize each class description with diverse captions retrieved from the pre-training corpus as a prompt ensemble. Specifically, for each class description based on the original prompt, we utilize CLIP to recall the most similar images from the pre-training dataset via image-text similarity, then obtain their corresponding captions. The retrieved captions with no appearance of the class name are filtered out, yielding K captions. Such a workflow leverages both visual semantics and class names, achieving better performance. Table 8 shows some cases of the captions retrieved by our proposed REPE on CIFAR100. They share the same target of interest with the original prompt, i.e., “a photo of a [CLASS]”, but provide the context in which the class name is located and thus have richer semantics. For example, given a class like bridge, the retrieved captions describe its possible properties (e.g., “golden”, “wooded”), connections to other objects (e.g., “over a mountain river”), etc., yielding more expressive and distinguishable text features of the class.
After retrieval, we encode the retrieved captions and conduct a mean pooling operation among them. The final text representation is:
fREPET (ti) = (1− λ)fT (ti) + λ 1
K
K∑
j
fT (rtij),
Vocabulary (Superclass) Classes
garment
trench coat, abaya, gown, poncho, military uniform, jersey, cloak, bikini, miniskirt, swimming trunks, lab coat, brassiere, hoopskirt, cardigan, pajama, academic gown, apron, diaper, sweatshirt, sarong
bird
African grey, bee eater, coucal, American coot, indigo bunting, king penguin, spoonbill, limpkin, quail, kite, prairie chicken, red-breasted merganser, albatross, water ouzel, goose, oystercatcher, American egret, hen, lorikeet, ruffed grouse
reptile
Gila monster, agama, triceratops, African chameleon, thunder snake, Indian cobra, green snake, mud turtle, water snake, loggerhead, sidewinder, leatherback turtle, boa constrictor, garter snake, terrapin, box turtle, ringneck snake, rock python, American chameleon, green lizard
arthropod
rock crab, black and gold garden spider, tiger beetle, black widow, barn spider, leafhopper, ground beetle, fiddler crab, bee, walking stick, cabbage butterfly, admiral, lacewing, trilobite, sulphur butterfly, cicada, garden spider, leaf beetle, long-horned beetle, fly
mammal
Siamese cat, ibex, tiger, hippopotamus, Norwegian elkhound, dugong, colobus, Samoyed, Persian cat, Irish wolfhound, English setter, llama, lesser panda, armadillo, indri, giant schnauzer, pug, Doberman, American Staffordshire terrier, beagle
accessory
bib, feather boa, stole, plastic bag, bathing cap, cowboy boot, necklace, crash helmet, gasmask, maillot, hair slide, umbrella, pickelhaube, mit- ten, sombrero, shower cap, sock, running shoe, mortarboard, handkerchief
craft
catamaran, speedboat, fireboat, yawl, airliner, container ship, liner, trimaran, space shuttle, aircraft carrier, schooner, gondola, canoe, wreck, warplane, balloon, submarine, pirate, lifeboat, airship
equipment
volleyball, notebook, basketball, hand-held computer, tripod, projector, barbell, monitor, croquet ball, balance beam, cassette player, snorkel, horizontal bar, soccer ball, racket, baseball, joystick, microphone, tape player, reflex camera
furniture
wardrobe, toilet seat, file, mosquito net, four-poster, bassinet, chiffonier, folding chair, fire screen, shoji, studio couch, throne, crib, rocking chair, dining table, park bench, chest, window screen, medicine chest, barber chair
instrument
upright, padlock, lighter, steel drum, parking meter, cleaver, syringe, abacus, scale, corkscrew, maraca, saltshaker, magnetic compass, accordion, digital clock, screw, can opener, odometer, organ, screwdriver
man-made structure
castle, bell cote, fountain, planetarium, traffic light, breakwater, cliff dwelling, monastery, prison, water tower, suspension bridge, worm fence, turnstile, tile roof, beacon, street sign, maze, chain-link fence, bakery, drilling platform
wheeled vehicle
snowplow, trailer truck, racer, shopping cart, unicycle, motor scooter, passenger car, minibus, jeep, recreational vehicle, jinrikisha, golfcart, tow truck, ambulance, bullet train, fire engine, horse cart, streetcar, tank, Model T
where rtij is the j(th) retrieved caption for class i and λ is a weighting factor. After that, the ensemble text representation fREPET (ti) is adopted as the class anchor for conducting the image classification. With REPE, the representation of the class description shifts towards that of the representative captions in the pre-training dataset, which alleviates the semantic inconsistency between pretraining and inference.
Experiments We retrieve the images and captions from CC12M (Changpinyo et al., 2021), a subset of the pre-training dataset of CLIP. The images and captions are pre-encoded within an hour using a single RTX TITAN GPU, then we build their indices for KNN search with the FAISS framework (Johnson et al., 2019), which also takes about an hour. Once the indices are built, we can efficiently search over the dataset according to the query image in less than 5 ms, which is applicable for query-intensive scenarios.
Table 9 shows the results of REPE. The hyperparameter K is 100 and λ is 0.25. REPE consistently improves the extensibility and stability of CLIP by an average of 1.2% across all three datasets. We further evaluate the quality of the enhanced representations by analyzing the loss of text uniformity and inter-modal alignment. As shown in Figure 7, our proposal effectively reduces ℓuniform-T from −0.8 to −1.0 and ℓalign from 1.5 to 1.4, verifying its effectiveness in improving the class anchor for better extensibility and stability. Additionally,
as shown in Figure 9, REPE increases the median value of the margin distribution from 0.005 to 0.01 and pushes the overall distribution towards the positive side compared to vanilla CLIP. It indicates that REPE widens the gap between positive and negative class features, making it more difficult to invert predictions with competing classes. These findings support REPE’s effectiveness in alleviating the openness issue.
It is worth noting that compared to the method that requires computation-intensive pre-training procedures (DeCLIP and SLIP), and the prompttuning approach (CoOp) demands access to the downstream target dataset, our REPE is a lightweight framework for the zero-shot inference stage without fine-tuning. Besides, since REPE is model-agnostic and orthogonal to parameter-tuning methods, it can also be combined with fine-tuning methods like adapter-tuning (Gao et al., 2021), to achieve a further performance boost of 0.6 on CIFAR100 and ImageNet, which demonstrates the adaptability and superiority of our method. Please refer to Table 10 for details.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
3 A2. Did you discuss any potential risks of your work? 6
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 4.3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4.3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 3.2
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We hope that our investigation and analysis will facilitate future research on the CLIP openness issue. Limitations
To facilitate future research, we analyze the difficulties and possible solutions in this new area. (1) As we present extensive empirical results and address the weakness of CLIP on vocabulary expansion, its theoretical risk on open tasks is urged to be investigated. (2) The current evaluation protocol is an approximation of the real open world. An evolving benchmark could facilitate future research. (3) For various visual categories, their degree of abstraction, the ease of describing them in natural language, and their density in the data distribution can also influence the extensibility and stability of models, which are worth studying