Vocabulary-free Image Classification
Authors: Alessandro Conti
Enrico Fini
Massimiliano Mancini
Paolo Rota
Yiming Wang
Elisa Ricci
Venue: NeurIPS 2023 poster (2023)
Keywords: language and vision
zero-shot classification
image classification
TL;DR: Propose the vocabulary-free image classification task, to enable  zero-shot classification with vision-language models without a pre-defined set of class names.

Abstract: Recent advances in large vision-language models have revolutionized the image classification paradigm. Despite showing impressive zero-shot capabilities, a pre-defined set of categories, a.k.a. the vocabulary, is assumed at test time for composing the textual prompts. However, such assumption can be impractical when the semantic context is unknown and evolving. We thus formalize a novel task, termed as Vocabulary-free Image Classification (VIC), where we aim to assign to an input image a class that resides in an unconstrained language-induced semantic space, without the prerequisite of a known vocabulary. VIC is a challenging task as the semantic space is extremely large, containing millions of concepts, with hard-to-discriminate fine-grained categories. In this work, we first empirically verify that representing this semantic space by means of an external vision-language database is the most effective way to obtain semantically relevant content for classifying the image. We then propose Category Search from External Databases (CaSED), a method that exploits a pre-trained vision-language model and an external vision-language database to address VIC in a training-free manner. CaSED first extracts a set of candidate categories from captions retrieved from the database based on their semantic similarity to the image, and then assigns to the image the best matching candidate category according to the same vision-language model. Experiments on benchmark datasets validate that CaSED outperforms other complex vision-language frameworks, while being efficient with much fewer parameters, paving the way for future research in this direction.

Vocabulary-free Image Classification
Alessandro Conti'
Enrico Fini' Massimiliano Mancini
Paolo Rota'
Yiming Wang"
Elisa Ricei 1.2
University of Trento
Fondazione Bruno Kessler (FBK)
Abstract
Recent advances in large vision-language models have revolutionized the image classification paradigm. Despite showing impressive zero-shot capabilities, a pre-defined set of categories, a.k.u. the vocabulary, is assumed at test time for composing the textual prompts. However, such assumption can be impractical when the semantic context is unknown and evolving. We thus formalize novel task, termed as Vocabulary-free Image Classification (VIC), where we aim to assign to input image class that resides in an unconstrained language-induced semantic space, without the prerequisite of known vocabulary. VIC is challenging task as the semantic space is extremely large, containing millions of concepts, with hard- d--dscriiiinte fine- ane-ggained categories. this work, we first empirically verify that representing this semantic space by means of external vision-language database is the most effective way to obtain semantically relevant content for classifying the image. We then propose Category Search from External Databases (CaSED)) method that exploits a pre-trained vision-language model and an external vision- language database to address VIC in training-free free manner. CaSED first extracts a set of candidate categories from captions retrieved from the database based on their semantic similarity the image, and then assigns the image the best matching candidate category according to the same vision- rision-language model. Experiments on benchmark datasets validate that CaSED outperforms other complex vision- language frameworks, while being efficient with much fewer parameters, paving the way for future research in this direction
Introduction
enabled astonishing progress in computer vision by aligning multimodal semantics in a shared embedding space. This paper focuses on their use for image classification, where models such as CLIP [7) demonstrated strength in zero-shot transfer. While we witnessed advances in VLM-based classification in many directions, e.g. prompt learning [53] 66), scaling up larger models and datasets (25 45 G by jointly considering
Large-scale Vision-Language Models (VLMs) 47 62 34
captioning task [22 34), they all assume a finite set of target categories, i.e. the vocabulary, to be is often violated
pre-defined and static (as shown in Fig. [b.. However, this assumption is fragile, in practical applications (e.g. robotics, autonomous driving) where semantic categories can either the deploymen//eesting or evolve dynamically over time.
differ from the developmen//rraining
In this work, we remove this assumption and study the new task of Vocabulary- Image Classifica- tion (VIC). The objective of VIC is to assign an image a class that belongs to unconstrained language-induced semantic space at test time, without a vocabulary, i.e. without a pre-defined set of categories (as shown in Fig. [p). The unconstrained nature of the semantic space makes VIC challenging problem. First, the search space is extremely large, with cardinality on the order
'Code and demo available at httttps://github... wttp://...oomaaaamaaaaaaaaaaaaa aatdrr//vdddr/
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

V
VLM
Cassowery
Cassowery
Blu Jay Cansessary Flam
Canspwonyy
VLL
Uncoratrared langugge-nn
estimation
Reacoek
Fimu
L'LK
Pallicrn
Kidi
(a) VLM-based classification
(b) Vocabulary-free Image Classification
Figure I: Vision-Language Model (VLM)- based classification (a) assumes pre-defined set of target categories, i.e. the vocabulary, while our novel task (b) lifts this assumption by directly operating on the unconstrained language-inducet semantic space, without known vocabulary. fvin and fvux denote the pre-trained vision and text models of VI.M, respectively.
of millions semantic concepts way larger than any existing image classification benchmark (e.g. ImageNet-21k [TO). Second, also includes very fine-grained concepts that might be hard to discriminate by the model. Third, as the categories encountered at test time are undefined beforehand, VIC calls for classification methods that do not rely on any vocabulary-awan supervision.
Recent web-scale Vision-Language Databases (VLDs) 51 offer unique opportunity to address VIC, as they cover wide set of semantic concepts, ranging from general to highly specific ones. We empirically show that such external databases allow identifying semantic content that more relevant to the target category than the captioning enhanced VLMs supporting visual queries (e.g. BLIP-2 33 Motivated by this observation, we propose a training- -fren method for VIC. Category Search from External Databases (CaSED), which jointly leverages the discriminative multimodal representations derived from CLIP [47 and the information provided by recent VLDs (e.g. PMD (54)). CaSED operates in two steps: it first coarsely estimates set of candidate categories for the test image, then predicts the final category
multimodal matching. Specifically, we first retrieve the captions from the database that are semantically closer to the input image, from which we extract candidate categorics via text parsing and filtering. We then estimate the similarity score between the input image and each candidate category via CLIP, using both visual and textual information, predicting as output the best matching candidate. CaSED exploits the pre-traines CLIP without further training, thus being flexible and computationally efficient.
We experiment several datasets, considering both coarse- (e.g. Caltech- 101 [14] UCF101 55) and fine fine-grained (e.g. FGVC-Aircraft [40], Flowers- 102 [43 classification tasks. To quantitatively the performance of methods addressing VIC, we also propose set of metrics measure how well the predicted class matches the semantics of the ground-truth label. Across all tasks and metrics, CaSED consistently outperforms all baselines, including VL.Ms as complex BLIP-2 (33) We believe that thanks to its simplicity and effectiveness, our training- foining-free CaSED can serve as competitive baseline for future works aiming address the challenging VIC task.
To summarize, this work provides the following contributions:
We explore the task of Vocabulary-free Image Classification where the goal is to assign a class to an image over unconstrained set of semantic concepts, overcoming the fundamental assumption of existing VLMM-bseed methods for image classification. We formalize this task and suggest specific evaluation metrics that can be used as reference for future works.
We propose CaSED, the first method to address VIC thanks to the adoption of large captioning databases. Notably, CaSED is training- free, not requiring any additional parameter finetuning the network's 's textual and visual encoders.
Our large-scale evaluation demonstrates that CaSED consistently outperforms more complex VLM such as BLIP-2 33 on VIC, while requiring much fewer parameters.
We estimate the number of concepts using BabelNet 42 which is close 4 million for English.

---

2
Related work
Vision-Language Models. Leveraging large-scale datasets with image-text pairs [51
50 54 recent
works train models by mapping the two modalities into a shared representation space [28] 18] IT
47
notable example is CLIP 47 which, using a contrastive objective to align their output representations, showed remarkable performance on zero-shot classification. Subsequent works improved CLIP by e-g. connecting the two modalities via cross-modal attention [35], multi-object representation alignment [64] learning from weak-
26
35
[5].
encoders and
supervision 600 or unaligned data [54].
Another line of works improved vision-language pre-training for complex vision-languaag tasks, such as image captioning and visual question answering (VQA) 662 [2 [4 []] In this context, BLIP [34] exploits web data and generated captions to supervise pre-training of multimodal architecture. outperforming existing VLMs on both captioning and VQA. The current state-of-the-ee method trains a module connecting the two modalities on top of a frozen visual encoder and a frozen wrggellaguagg model, enabling instructed zero-shot image-to-tex, generation.
BLIP-2 33
In this work, we challenge a fundamental assumption of zero-shot classification with VLMs: the set of target classes is known a priori. We propose new task, VIC, which sidesteps this assumption, performing classification in a language-induccdd open-ended space of semantic categories. We show that even BLIP-2 struggles in this scenario while external multimodal databases provide valuable priors for inferring the semantic category of an image. As final note, VIC differs from open- vocabulary recognition (e.g. 663 [7)) since the latter assumes that the list of target classes is known and available to the model during inference.
Retrieval augmented models. In natural language processing, multiple works showed the benefit of retrieving information from external databases, improving the performance of large language models 19 32 3 In computer vision, such paradigm has been used mostly to deal with the
imbalanced distribution of classes. Examples are [38 39), addressing long-tail recognition by learning to retrieve training samples [38] or image-text pairs from an external database [39]. Similarly, (58) retrieves images from given dataset to learn fine- tineggrinedd visual representations. More recently, has been extended to various types of sources for visual question answering 233, as well to condition the generative process in diffusion models (2), or image captioning [48).
Our work is close in spirit to [39], as we exploit an external database. However, (39] assumes a pre-defined set of classes (and data) available for training retrieval module, something we cannot have for the extremely large semantic space of VIC. In CaSED, retrieval is leveraged to first create a set of candidate classes, and to then perform the final class prediction. Moreover, we assume the database to contain only captions, and not necessarily paired image-text data, thus being less memory memory-demanding.
Finally, the performance of retrieval models is largely affected by the employed database. In the context of VLMs, researchers collected multiple open datasets to study vision-language pre- training. Two notable examples LAION-5B (51), collected by filtering Common Crawl [8] via CLIP, and the Public Multimodal Datasets (PMD) (54) collecting image-text pairs from different public datasets, such as Conceptual Captions (52 S YFCC 1000 (57). Wikipedia Image Text (56).
and Redcaps [1). In our experiments, we use a subset of PMD as database and investigate how classification performance varies based on the size and quality of the database.
3 Vocabulary-free Image Classification
Preliminaries. Given the image space X' and set of class labels C. classification model f X C a function mapping an image x E X' to its corresponding class e E C. While C' may be represented by indices that refer to semantic classes, here focus cases where C consists of a set of concept names that correspond to real entities. Note that C is a finite set whose elements belong to the semantic space S, i.e. C' C S. In standard image classification, C' is given priori, and f is learned on a dataset of image-label pairs.
Recently, the introduction of contrastive-based VLMs 47 25 which learn a mross-modal aligned feature space, revised this paradigm by defining function frNm that infers the similarity between an
3

---

100
80
Accuracy (%)
80
0
C101 DTD
ESAT
Airc.
Flwr
Food
Pets
SUN
Cars
UCF
Average
Figure 2: Results of our preliminary study, showing the top-1 accuracy when matching semantic descriptions to ground-truth class names in ten different datasets. We compare BLIP-2 (VQA) and BLIP-2 (Captioning) with Closest Caption and Captions Centroid i.e.the average represen- tation of the retrieved captions. We additionally highlight the Upper bound for zero-shot CLIP. Representing the large semantic space VLDs and retrieving captions from it produces semantically more similar outputs to ground-truth labels w.r.t querying outputs from VQA-enabled VLMs, while requiring 10 times fewer parameters compared the latter.
image and textual description E T, i.e. fvix X' x X + E, with T being the language space. Given this function, classification can be performed with:
f(z) arg fvun(a, ó(c))
(I)
where ((c) is string concatenation operation, combining fixed text template, i.e. prompt, with class name. This definition allows for zero-shot transfer, i.e. performing arbitrary classification tasks by re-defining the set C' at test time, without re-training the model. However, such zero-shot transfer setup still assumes that the set C' is provided. this work, Vocabulary- Image Classification (VIC) is formulated to surpass this assumption.
Task definition. VIC aims to assign a class c to an image æ without prior knowledge on C, thus operating the semantic class space S that contains all the possible concepts. Formally, want to produce a function f mapping an image to a semantic label in S, i.e. f : - S. Our task definition input image and large source
implies that test time, the function f has only access of semantic concepts that approximates S. VIC is a challenging classification task by definition due the extremely large cardinality of the semantic classes in S. As example, ImageNet- 21k [10), one of the largest classification benchmarks, is 200 times smaller than the semantic classes in BabelNet [42]. This large search space poses prime challenge for distinguishing fine-grained concepts across multiple domains as well as ones that naturally follow a long-tailed distribution.
Semantic space representation. As the main challenge of VIC, how represent the large semantic space plays fundamental role in the method design. We can either model the multimodal semantic space directly with a VLM equipped with autoregressive language decoder (34) via image- text retrieval from VLDs. Consequently, we can approach VIC either via VQA-enabled VLMs by querying for the candidate class given the input image, or by retrieving and processing data from an external VLD to obtain the candidate class.
To investigate the two potential strategies, we perform a preliminary experimental analysis to under- stand how well the output of a method semantically captures the image category, or in other words, to assess the alignment of class boundaries in the visual and textual representations. Specifically. compare the semantic accuracy of querying VQA VLMs and of retrieving from VLDs ground-truth class labels. We consider the output of method as correct if its closest textual embed- ding among the target classes of the dataset corresponds to the ground -trtth class of the test sample We exploit the text encoder of CLIP (ViT-L) [4]] to obtain textual embeddings.
the
NNote that this metric is not the standard accuracy in image classification as we use distances in the embedding space ground predictions from the unconstrained semantic space to the set of classes in specific dataset.

---

Regarding experimented methods, we select BLIP-2 [33 to represent VQA-enabled VLMs for its state-of-the-art performance in VQA benchmarks, while we use a subset of PMD [54] as the VLD. In particular, we compare the following methods: i) BIJP-2 VQA, which directly queries BLIP-2 for the image category; ii) BIJP-2 Captioning. which queries BL.IP-2 for the image caption; iii) Closest Caption, which is the closest caption to the image, as retrieved from the database; iv) Caption Centroid, which averages the textual embeddings of the 10 most similar captions to the input image. As we use CLIP embeddings, if visual and textual representations perfectly align, the performance would be the same as zero-shot CLIP with given target classes. We thus report zero-shot CLIP to serve the upper bound for retrieval accuracy.
We experiment on a variety of test datasets for both coarse- and fine-grained classification (see details in Sec. 5) and report the results in Fig. 2 The average textual embedding of the retrieved captions (i.e. Caption Centroid) achieves the best semantic accuracy for datasets out of 10, consistently surpassing methods based on BLIP-2. On average, the accuracy achieved by Caption Centroid is 60.47%, which + 17.36% higher than the one achieved by BLIP-2 Captioning (43.11% Moreover, Captions Centroid achieves results much closer to the CLIP upper bound (67.17% than the other approaches. Notably, such VLD-based retrieval is also computationally more efficient, faster (- second for batch size of 64 on single A6000 GPU), and requires fewer parameters (approximately 10 times less) than BLIP-2 2 (see Tab. 7 in Appendix [B).
The results of this preliminary study clearly suggest that representing the large semantic space with VLDDs can produce (via retrieval) semantically more relevant content to the input image, comparison to querying VQA-enabled VLMs, while being computationally efficient. Based on this conclusion, we develop an approach, Category Search from External Databases (CaSED), that searches for the semantic class from the large semantic space represented in the captions of VLDs.
4 CaSED: Category Search from External Databases
Our proposed method CaSED finds the best matching category within the unconstrained semantic space by multimodal data from large VLDs. Fig. provides overview of proposed method. We first retrieve the semantically most similar captions from a database, from which we extract a of candidate categories by applying text parsing and filtering techniques. We further score the candidates using the multimodal aligned representation of the large pre-trained VLM, i.e. CLIP 47 obtain the best- mate--mttnnn category. We describe in detail each process in the following.
4.1 Generating candidate categories
We first restrict the extremely large classification space to a few most probable candidate classes. Let fvun be the pre-trained VLM and D be the external database of image captions. Given input æ, we retrieve the set DN C D of K closest captions to the input image via
Dæ = top-k frux(x, d) = top-k (füx(z), (d)( deD
(2)
dED
where : 2 is the visual encoder of the VLM, fuLN T + 2 is the textual encoder, and 2 is their shared embedding space. The operation () indicates the computation of the cosine similarity. Note that our approach is agnostic to the particular form of D, and that it can accommodate a flexible database size by including captions from additional resources.
From the set Dæy we then extract a finite set of candidate classes C. by performing simple text parsing and filtering techniques, e.g. stop- words removal, POS tagging. Details on the filtering procedure can be found in Appendix A
4.2 Multimodal candidate scoring
Among the small set C., we score each candidate by accounting for both visual and textual semantic similarities using the VLM encoders, in order to select the best-matching class for the input image.
Image-to-text score, As the image is the main driver for the semantic class we aim to recognize, we use the visual information score the candidate categories. We denote se the visual score of cach
5

---

fVLM
Z
D2 Check out tii feets Knol- Ui concnnatry business
Castowar
Brdd
.om
Retrieval from text database
bocking presatonc
fVLM
D
n fre midden gemm keegtor
Concowary
CN
cassowory
Fowl
Rird Corsowary Tooo Gom
Check nou powd Sireiie
JVLM
looking blrd
Cessowary
Bird
animal
Figure 3: Overview CaSED. Given an input image, CaSED retrieves the most relevant captions from an external database filtering them to extract candidate categories. We classify image -to- text and text to- text using the retrieved captions centroid as the textual counterpart of the input image.
candidate category c and compute it as the similarity between the visual representation of the input image and the textual representation of the candidate name:
S (fx(()))
The higher value of s indicates a closer alignment between the target image and the candidate class.
Text-to-text score. While the image-to-text score s effective, there exists a well-known modality gap in the space 2, harming the performance of zero-shot models [36.. As suggested by Fig. 2 Sec. the semantic relevance of the retrieved captions, and their centroid in particular, high w.r.t. the underlying ground-truth label. We are therefore motivated exploit such attributes and introduce a unimodal text-to-text scoring to mitigate the modality gap of cross-modal scoring.
Formally, we define the centroid dz of the retrieved captions as:
de
Σ (d)(d), K
1
dED
where the number of retrieved captions. We then define text-based matching score &f as the similarity between the centroid and the candidate category:
5/ (day f(n((c))
higher value of s means higher alignment between the caption centroid and the candidate embedding. Note that the semantic relevance of the caption centroid is an inherent property of the text encoder of VLMs (i.e. CLIP). Since CLIP trained with an image-text alignment loss, its text encoder focuses on the visual elements of the caption, discarding parts that are either non-visual or non-relevant to the visual content. This improves the model's robustness to noisy candidates.
Final predicted candidate. To predict the final candidate, we merge the two scores, obtaining the final score se for each candidate c as:
8c = a ( + (1 a) (s
where o-.) is the softmax operation on the two scores of each candidate class, and a is hyperpa- rameter regulating the contribution of the two modalities. Finally we obtain the output category as f(x) = arg max Se- Notably, CaSED respects the VIC task definition, performing classification without known class priors, while being training-free with the use a pre-trained and frozen VLM. This makes the approach flexible and applicable to variety of architectures and databases.

---

5 Experiments
We evaluate CaSED in comparison to other VLM-based methods on the novel task Vocabulary-free Image Classification with extensive benchmark datasets covering both coarse-grained and fine-grained classification. We first describe the experimental protocol in terms of the datasets, the proposed evaluation metries, and the baselines applicable to this task (..). We then discuss the quantitative results regarding the comparison between our method and baselines (Sec. [52)). Finally, we present a thorough ablation to justify the design choices of CaSED (Sec. (.3) In addition, we provide cost comparison between the baseline methods and CaSED in Appendix [B] We also offer further ablation of our method in Appendix C regarding architecture and database, and showcase qualitative results of predicted categories from multiple datasets in Appendix D
5.1 Experimental protocol
Datasets. We follow existing works [53 66 and use ten datasets that feature both coarse-grained
and fine-grained classification in different domains: Caltech-101 101 (C101) [4), DTD
77].
Eu-
roSAT (ESAT) 211. FGVC-Aireraft (Airc.) 40). Flowers- (Flwr) 43 Food-101 (Food) 4
Oxford Pets (Pets). Stanford Cars (Cars) (29). SUN397 (SUN) [6], and UCF101 (UCF)
[55
Additionally, we used ImageNet [TO] for hyperparameters tuning.
Evaluation metrics. Due to the unconstrained nature of the semantic space, evaluating the effective- ness of methods for VIC is not trivial. In this paper, we propose to two main criteria, namely semantic relevance, i.e. the similarity of the predicted class w.r.t. the ground-truth label, and image grouping, i.e. the quality of the predicted classes for organizing images into clusters. For semantic, we consider i) Semantic Similarity, i.e. the similarity of predicted/groundd-e space, and ii) Semantic IoU, i.e. the overlap of words between the prediction and the true label. More formally, given an input x: with ground-truth label y and prediction è = f'(x), we compute the Semantic Similarity as (g()).g(() gyy), where g: T- y is function mapping text to an embedding space y. Since we want to model free-form text, we use Sentence- BERT [49] as For Semantic IoU, given a predicted label c, and assuming e being a set of words, we compute the Semantic IoU as |e / U y). where y is the set of words in the ground-truth label. To assess grouping, we measure the classic Cluster Accuracy by first clustering images according to their predicted label, and then assigning each cluster to a ground-trume truth label with Hungarian matching. Sometimes, this mapping is resolved with many-to -o-eett-onee match, where a predicted cluster assigned to the most present ground-truth label. This evaluation draws inspiration from the protocols used for deep visual 20
labels in a semantic
clustering 59
Baselines. We consider three main groups of baselines for our comparisons. The most straightforward baselines consist of using CLIP with large vocabularies, such as WordNet [41) (117k names) or the English Words (234k names (66). As an upper bound, we also consider CLIP with the perfect vocabulary, i.e. the ground-truth names of the target dataset (CLIP upper bound). Due to lack of space, we only report results for CLIP with ViT-L [13], while results with other architectures are reported in Appendix C The second group of baselines consists of captioning methods, as captions can well describe the semantic content of images. We consider two options: captions retrieved from a database and captions generated by a pre- image captioning model. For the former we exploit a large collection of textual descriptions, retrieving the most fitting caption for the given image. For the latter, we exploit BLIP-2 X a VLM with remarkable performance on a variety of tasks, including image captioning
to provide description for the image. The last group of baselines consists of using VQA model to directly predict the class name associated with the image. Again, we consider BLIP-2 (33, since being highly effective also in VQA. We evaluate BLIP-2 over both ViT-L and ViT-g throughout the experiments
Implementation details. Our experiments were conducted using NVIDIA A6000 GPUs with mixed- bit precision. As database, we use subset of PMD (54), containing five of its largest datasets: Conceptual Captions (CC3M) [52], Conceptual Captions 12M (CC12M) [5) Wikipedia Image Text
(WIT) [56], Redcaps
and a subset of [57] used for PMD (YFCC100M*) Further details on the
demo, for captioning. we used the prompt "Question: what's in the image? Answer:". For VQA, we used "Question: what's the name of the object in the image? Answer: a".
FFollowing the BLIP-2 33
7

---

Method
Cluster Accuracy (%)
C101
DTD
ESAT
Aire.
Flwr
Food
Pots
SUN
Cars
UCF
Avg.
CLIP
WordNet
34.0
20.1
16.7
16.7
58.3
40.9
52.0
29.4
18.6
39.5
32.6
English Words
29.1
19.6
22.1
15.9
64.0
30.9
44.4
24.2
19.3
34.5
30.4
Closest Caption
12.8
8.9
16.7
13.3
28.5
13.1
15.0
8.6
20.0
17.8
15.5
Caption
BLIP-2 (ViT-L)
26.5
11.7
23.3
5.4
23.6
12.4
11.6
19.5
14.8
25.7
17.4
BLIP-2 (ViT-g)
37.4
13.0
25.2
10.0
29.5
19.9
15.5
21.5
27.9
32.7
23.3
BLIP-2 (VTT-L)
60.4
20.4
21.4
8.1
36.7
21.3
14.0
32.6
28.8
44.3
28.8
VQA
BLIP-2 (ViT-g)
62.2
23.8
22.0
15.9
57.8
33.4
23.4
36.4
57.2
55.4
38.7
CaSED
51.5
29.1
23.8
22.8
68.7
58.8
60.4
37.4
31.3
47.7
43.1
CLIP upper bound
87.6
52.9
47.4
31.8
89.9
65.3
76.5
72.5
69.0
Table
Cluster Accuracy on the ten datasets.
Green our method, gray shows the upper bound.
Method
Semantic Similarity (x100) t
C101
DTD
ESAT
Aire.
Flwr
Food
Pets
SUN
Cars
UCF
Avg.
CLIP
WordNet English Words
48.6
32.7
24.4
18.9
55,9
49.6
53.7
44.9
28.8
44,2
40.2
39.3
31.6
19.1
18.6
43.4
38.0
44.2
36.0
19.9
34.7
32.5
Closest Caption
42.1
23,9
23.4
29.2
40.0
46.9
40.2
39.8
49.2
40.3
37.5
Caption
BIIP-2 (VIT-L.)
57.8
31.4
39.9
24.4
36.1
44.6
29.0
45.3
46.4
38.0
39.3
BLIP-2 (ViT-g)
63.0
33.1
36.2
24.3
45.2
51.6
31.6
48.3
61.0
44.6
43.9
BLIP-2 (VIT-L)
70.5
34,9
29.7
29.1
48.8
42.0
40.0
50.6
52.4
48.6
44.7
VQA
BI.IP-2 (ViT-g)
73.5
36.5
31.4
30.8
59.9
52.1
43.9
53.3
65.1
55.1
50.1
CaSED
65.7
40.0
32.0
30.3
55.5
64.5
62.5
52.5
47.4
54.1
50.4
CLIP upper bound
90.8
69.8
67.7
66.7
83.4
93.7
91.8
80.5
92.3
83.3
82.0
Table 2: Semantic Similarity on the ten datasets. Values are multiplied by x100 for readability.
Green highlights our method and gray indicates the upper bound.
We speed up the retrieval process by embedding the database via the text encoder fyx and using fast indexing technique, iee. FAISS [27] We tuned the hyperparameter of Eq. (6) and the number of retrieved captions K of our method on the ImageNet dataset, finding that a 0.7 and K = 10 led to the best results. We use these values across all experiments.
selection are left for Appendix C
5.2 Quantitative results
The results of CaSED and the baselines are presented in Table [ Table and Table for Cluster Accuracy (%), Semantic Similarity, and Semantic IoU (%). respectively. CaSED consistently outperforms all baselines in all metrics on average and in most of the datasets. Notably, CaSED surpasses BLIP-2 (VQA) over ViT-g by +4.4% in Cluster Accuracy and +1.7% on Semantic IoU, while using much fewer parameters (i.e. 102M vs IB). The gap even larger over the same visual backbone, with CaSED outperforming BLIP-2 on VIT-L (VQA) by 14.3% on Cluster Accuracy, +5.7 in Semantic Similarity, and +6.8% on Semantic IoU. These results highlight the effectiveness of CaSED, achieving the best performance both in terms of semantic relevance and image grouping.
An interesting observation from the tables is that simply applying CLIP on top of a pre-defined, large vocabulary, is not effective in VIC. This is due to the challenge of classifying over huge search space, where class boundaries are hard to model. This is confirmed by the results of CLIP with English Words having larger search space but performing consistently worse than CLIP with WordNet across all metrics (e.g- - 7.7 on Semantic Similarity, -3.2% on Semantic loU).
final observation relates to the captioning models. Despite their ability capture the image semantic even in challenging settings (e.g. 39.3 Semantic Similarity of BLIP-2 ViT-L on ESAT), captions exhibit high variability across images of the same category. This causes the worst performance on Clustering and Semantic loU across all approaches (e.g. almost 20% less than CaSED on average in terms of Cluster Accuracy), thus demonstrating that while captions can effectively describe the content of an image, they do not necessarily imply better categorization for VIC.
8

---

Method
Semantic ToU (%)
C101
DTD
ESAT
Aire.
Flwr
Food
Pets
SUN
Cars
UCF
Avg.
WordNet
15.0
1.3
0.5
31.3
7.8
14.7
9.0
4.8
3.8
9.1
CLIP
English Words
8.0
2.0
0.0
1.1
16.4
2.0
17.2
8.1
2.7
1.8
5.9
Closest Caption
1.3
1.9
3.1
2.3
11.4
1.0
3.5
Caption
BLIP-2 (VIT-L)
13.4
0.0
4.7
4.7
11.6
5.1
BLIP-2 (ViT-g)
16.8
0.1
13.9
7.9
5.7
24.7
1.9
8.0
BLIP-2 (VTT-L)
36.1
1.8
7.0
0.1
21.5
3.7
5.7
11.5
18.9
2,5
10.9
VQA
BLIP-2 (ViT-g)
41.5
2.4
7.5
2.0
38.0
8.6
10.2
13.8
33.2
2.8
16.0
CaSED
35.4
5.1
2.3
4.8
33.1
19.4
35.1
17.2
16.2
8.4
17.7
CLIP upper bound
51.5
28.6
89.9
66.6
84.5
69.4
Table 3; Semantic IoU on the ten datasets.
Green is our method, gray shows the upper bound.
Database
Size
CA
S-Sim.
S-IoU
Candidates
Scoring Lang.
CA
S-Sim.
S-IoU
CC3M
2.8M
34.2
47.9
13.1
Generation
Vis.
WIT
4.8M
34.6
42.9
12.1
Generative 33
/
23.3
47.1
11.9
Redcaps
7.9M
42.0
49.5
17.2
41.7
49.3
17.0
CC12M
10.3M
44.0
51.3
18.3
Retrieval
42.7
50.3
17.0
YFCCI00M
29.9M
40.7
48.8
17.1
43.1
50.4
17.7
All
54.8M
43.1
50.4
17.7
(a) Ablation on candidate generation and scoring.
(b) Ablation on the database.
Table 4: Ablation studies. Metrics are averaged across the ten datasets. Green highlights our setting. Bold represents best, underline indicates second best.
5.3 Ablation studies
In this section, we present the ablation study associated with different components of our approach. We first analyze the impact of our retrieval-based candidate selection and multimodal scoring. We then show the results of CaSED for different databases, and how the number of retrieved captions impacts the performance.
Candidates generation. We consider two options to generate the set of candidate classes. The first uses BLIP-2 (ViT-g), asking for multiple candidate labels for the input image. The second is our caption retrieval and filtering strategy. Table 4a shows the results where, for fairness, we score both sets with the same VLM (i.e. CLIP ViT-L), Our approach consistently outperforms BLIP-2 across all metrics (e.g. 41.7 vs 23.3 for Cluster Accuracy). This confirms the preliminary results of our study in Fig. with retrieved captions providing better semantic priors than directly using a powerful VLM.
Multimodal scoring. The second ablation studies the effect of our proposed multimodal scoring vs its unimodal counterparts. As Table shows, multimodal candidate scoring provides the best results across all metrics, with clear improvements Accuracy). Notably, scoring via language only partially fills the gap between visual and multimodal scoring (e.g. +1% on Cluster Accuracy and +1 Semantic Similarity). confirming that caption centroids contain discriminative semantic information.
the visual modality alone (i.e. +1.4% of Cluster
Retrieval database. We analyze the impact of retrieving captions from different databases, using five public ones, i.e. CC3M, WIT, Redcaps, CC12M, and subset of YFCC100M. The databases have different sizes (e.g. from 2.8M captions of CC3M to 29.9M captions of the YFCC100M subset), and different levels of noise. As shown in Table 4b the results tend to improve the size of the database increases (e.g. +8.9% on Cluster Accuracy over CC3M). However, the quality of the captions influences the performance, with CC12M and Redcaps alone achieving either comparable or slightly better results than the full database. These results suggest that while performance improves with the size of the database, the quality of the captions has higher impact than the mere quantity.

---

Number of captions. Finally. we check how performance varies w.r.t. the number of retrieved captions. As shown in Fig. 3 all metrics con- sistently improve as the number of captions in-
-50
Metric value
40
creases from 1 to 10. After that, performance tends to saturate, with a slight decrease in terms of Semantic Similarity for K 20. These re- sults suggest that while a minimum number of captions is needed to fully capture the semantics of the image, the possible interference of less- related (or even noisy) captions may impact the final performance. Future research may further improve the performance VIC by focusing on how to deal with noisy retrieval results.
30
20
10
10 Number of retrieved captions
Figure 4:
Ablation on the number of re- trieved captions. We report Cluster accuracy (%) Semantic similarity and Semantic loU (%).
Discussion and conclusions
In this work, we proposed a new task, VIC, which operates on an unconstrained semantic space, without assuming pre-defined set of classes, brittle assumption of VLM-based classification. We experimentally verified that multimodal databases provide good semantic priors to restrict the large search space, and developed CaSED, an efficient training-free ffree approach that retrieves the closest captions to the input image to extract candidate categories and scores them in a multimodal fashion. On different benchmarks, CaSED consistently achieved better results than more complex VLMs.
Limitations and future works. The performance of CaSED strongly depends the choice of the retrieval database, with potential issues in retrieving concepts that are not well represented in the latter. Moreover, if the domain of application contains fine-grained concepts, generic database might not be suitable. Without any prior information on the test labels, it is hard to predict the performance of database priori. On the other hand, CaSED can flexibly mitigate this issue by incrementally including new concepts in the database (even domain speciiniipcffic ones) from textual corpora, without retraining. Future research may explore strategies to automatically select/extend a database based on test samples and/or pre-computing the best database to use in the absence of test label information.
Additionally, as CaSED lacks control over the classes contained in the database, its predictions might reflect potential biases. Improvements in mitigating biases and data quality control would reduce this issue. Another limitation is that CaSED does not keep track of its output history. This may lead to inconsistent predictions, i.e. assigning slightly different labels to images of the same semantic concept (e.g. cassowary vs Casmarius). Equipping CaSED with memory storing the predicted labels may address this issue. Finally, CaSED does not deal with different class granularities: e.g. an image of a cassowary can be as well predicted as bird. Future works may disambiguate such cases by explicitly integrating the user needs within VIC models.
Broader impact. CaSED addresses VIC in scalable and training-fres manner. We believe that our new problem formulation, metrics, and the effectiveness of CaSED will encourage future research in this topic, overcoming the limiting assumption VLM-based classification and allowing the power of VLMs to benefit dynamic and unconstrained scenarios.
Acknowledgements
This work was supported by the MUR PNRR project FAIR Future AI Research (PE00000013) and ICSC National Research Centre for High Performance Computing. Big Data and Quantum Computing (CN00000013) funded by the NextGenerationEU. E.R. is partially supported by the PRECRISIS, funded by the EU Internal Security Fund project SPRING (No. 871245), and by the PRIN project LEGO-AI (Prot. 2020TA3K9N), The work was carried out in the Vision and Learning joint laboratory of FBK and UNITN. We also thank the Deep Learning Lab of the ProM Facility for the GPU time.
101 100539), the EU
10

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch. Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurlPS, 2022.
[2] Andreas Blattmann. Robin Rombach, Kaan Oktay, Jonas Müller, and Bjöm Ommer. aerrentl- diffusion models. NeurlPS, 2022.
[3] Schastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Haan-Bapiiste Lespiau, Bogdan Damoe, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In ICMI,, 2022.
[4] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. with random forests. In ECCV 2014, 2014.
discriminative components
[5] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail tail visual concepts. In CVPR, 2021.
[6] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023.
[7] Mircea Cimpoi, Subhransu Maji. Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
[8]
Common Crawl. Common crawl, 2023. Accessed May 17, 2023. httts::/coomonecravl. org/
[9] Yufeng Cui, Lichen Zhao, Feng Liang. Yangguang Li, and Jing Shao. Democratizing contrastive language- image pre-training: A clip benchmark of data, model, and supervision. arXiv, 2022.
[10] Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hicrarchical
image datahase. In CVPR, 2009.
[11] Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In
CVPR, 2021.
[12] Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image-text data created by the people, for the people. arXiv, 2021.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani. Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth
16x16 words: Transformers for image recognition scale. In ICLR, 2021.
[14] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshop, 2004.
[15] Enrico Fini, Pietro Astolfi, Adriana Cameer-SSorino, Jakob Verbeek, and Michal Drozdzal. Improved baselines for Aision-languaa pre- tre-training. 2023.
[16] FreeBSD. Web2 dictionary (revision 326913), 2023. Accessed May 17, 2023,
https://swwweb. freebsd
org/base/head/aee wwwaad//saeeeeeeeeeee waaeee/ddittt/eeeeeeeeeeeeeeeeee www.:rvvee.aaraaappptatttaeeeeee
[17] Golnaz Ghiasi Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open- vocabbocay image segmentation with image-level labels. In ECCV, 2022.
[18] Lluis Gomez, Yash Patel. Marçal Rusinol, Dimosthenis Karatzas, and CV Jawahar. Self-supervised learning of visual features through embodding images into text topic spaces. In CVPR, 2017. [19] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In ICMI., 2020.
[20] Kai Han, Sylvestre-Alvies Rebuffi, Sebastien Ehrbardt, Andrea Vedaldi, and Andrew Zisserman. Automat- ically discovering and learning new visual categories with ranking statistics. In ICLR, 2019.
[21] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Selected Topics in Applied Earth Observations and Remote Sensing. 2019.
[22] Xiaowei Hu, Zhe Gan, Jianfeng Wang. Zhengyuan Yang. Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre- ve-training for image captioning. In CVPR, 2022.
[23] Ziniu Hu, Ahmet Iscen, Chen Sun. Zirui Wang, Kai-Wat Chang, Yizhou Sun, Cordelia Schmid, David Ross, and Alireza Fathi. Reveal: Retrieval- augmented visual-language pre-training with multi-source multimodal knowledge memory. In CVPR, 2023.
[24] Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image
classification and segmentation. In ICCV, 2019.
[25] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML. 2021.
[26] Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Hun-Hsuan Sung. Zhen Li. and Tom Duerig. Scaling up visual and vision languaee representation learning with noisy text supervision. In ICML, 2021.
[27] Jeff Johnson. Matthijs Douze, and Hervé Jégou. on Big Data, 2019.
similarity search with GPUs. Transactions
11

---

[28]
Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In ECCV, 2016.
[29]
Jonathan Krause, Michael Stark, Jia Deng. and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshops, 2013.
[30] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJC,, 2017.
[31]
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 2020.
[32] Patrick Lewis, Ethan Perez, Alcksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. aetrieval-augmennn generation for knowledge- intensive nlp tasks. Neur/PS, 2020.
[33] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training
with frozen image encoders and large language models. arXiv, 2023.
[34] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training
for unified vision-language understanding and generation. In ICML, 2022.
[35] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong
Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurlPS, 2021.
[36] Victor Weixin Liang. Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning, NeurlPS, 2022.
[37] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014.
[38] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long- loigetailed recognition in open world. In CVPR, 2019.
[39] Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. In CVPR, 2022.
[40] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual
classification of aircraft. arXiv, 2013.
[41] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 1995.
[42] Roberto Navigli and Simone Paolo Ponzetto. Babelnet: Building a very large multilingual semantic
network. In ACL, 2010.
[43] Maria-Elens Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
[44] Vicente Ordonez, Girish Kulkami, and Tamara Berg, Im2text: Describing images using 1 million captioned photographs. NeurIPS, 2011.
[45] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer learning. arXiv, 2021.
[46] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari, Connecting vision and language with localized narratives. In ECCV, 2020.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CMI., 2021.
[48] Rita Ramos, Desmond Elliott, and Bruno Martins. Retrieval-angmented image captioning. In EACL, 2023. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence emboddings using siamese bert-networks. In EMNLP-IJCNLP. 2019.
[49]
[50] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Sbi:n-5b: An open large-scale dataset for training next generation mmage-text models. In NewrIPS, 2022.
[51] Christoph Schuhmann, Robert Kaezmarezyk, Aran Komatsuzaki, Aarush Katta, Richard Veneu, Romain Open dataset of clip-filtered
Beaumont, Jenia Jitsev, Theo Coombes, and Clayton Mullis.
400 million image-text pairs. In NeurlPS Workshop Datacentric AI, 2021.
[52] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Annual Meeting of the Association for Computational Linguistics, 2018
[53] Manli Shu, Weili Nie, De-An Huang. Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao.
Test-time prompt tuning for zero-shot generalization vision- language models. arXiv, 2022. [54] Amanpreet Singh, Ronghang Hu, Vedanuj Goswani, Guillaume Couairon, Wojciech Galuba. Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, 2022.
12

---

[55]
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv, 2012.
[56]
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: Wikipedia- based image text dataset for multimodal multilingual machine learning. In Research and Development in Information Retrieval, 2021.
[57] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications of the ACM, 2016.
[58] Hugo Touvron, Alexandre Sablayrolles, Matthijs Douze, Matthieu Cord, and Hervé Jégou. Grafit: Learning sue-grained image representations with coarse labels. In ICCV, 2021.
[59]
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In ECCV, 2020.
[60] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. ICLR, 2022.
[61]
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-
scale recognition from abbey 200. In CVPR. 2010.
[62]
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung. Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-texi foundation models. arXiv, 2022.
[63] Alireza Zareian. Kevin Dela Rosa, Derek Hao Hu, Shih-Fu Chang. Open-vocabulary object detection
using captions. In CVPR, 2021.
[64] Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. ICML, 2022.
[65] Bolei Zhou, Hang 7hao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralha. Scene parsing through ade20k dataset. In CVPR, 2017.
[66] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJC,, 2022.
13

---

Appendix
A Candidates filtering
With the closest captions retrieved from the external database given the input image (please refer to Sec. of the main manuscript for further details), we post-process them to filter out a set of candidate category names. We first create a set of all words that are contained in the captions. Then we apply sequentially three different groups of operations the set to (i) remove noisy candidates, (ii) standardize their format, and (iii) filter them.
With the first group of operations, we remove all the irrelevant textual contents, such as tokens (i.e. "<PERSON>"), URLs, or file extensions. Note that, for the file extensions, we remove the extension but retain the file name as it might contain candidate class names. We also remove all the words that are shorter than three characters and split compound words by underscores or dashes. Finally, we remove all those terms containing symbols or numbers and meta words that are irrelevant to the classification task, such as "image". "photo", or "thumbnail". As shown in Table 5 when compared having no operation for candidate filtering (first row), this set of operations removes inappropriate content and increases the accuracy of clusters by +6.4% and improves the semantic IoU by +0.3. However, we can observe a drop in semantic similarity by -1.5. This might be due to the removal of unnatural words that could still describe well the content of the image, i.e. underline- or depa-sep words, or URLs since they are longer w.r.t. natural words.
The second group of operations standardize the candidate names by aligning words that refer the same semantic class to a standard format, reducing class redundancy. For example, "cassowary" and "Cassowary" will be considered conversion and singular form conversion. With such standardizing conversions, we observe sizeable boost in terms of performances when compared to the results obtained by applying only removal-related operations. As shown in Table we achieve higher results across all three metrics, leading relative improvement of +7.2%, +0.7, and +1.2 in terms of cluster accuracy, semantic similarity, and semantic IoU, respectively.
single class instead of two. To this end, we perform
two operations-
The last group of operations considers two forms of filtering, where the first aims to filter out entire categories of words via Part-Of-Sta Of---Of-Seecch (POS) tag and the second aims to filter out rare and noisy contents based on the word occurrences. We select these two operations since common dataset class names do not contain terms that carry no semantics e.g. articles and pronouns, and since [9] showed that CLIP performs better when exposed to a smaller amount of unique tokens. The POS tagging 5 categorizes words into groups, such adjectives, articles, nouns, or verbs, enabling us to filter all the terms that are not semantically meaningful for a classification task. Regarding the occurrence filtering. we first count how often word appears in the retrieved captions and then we remove words that appear only once to make the candidate list less noisy. We can see from Table the inclusion of this final set of operations scores the best among all three metrics when compared to the results obtained when only the previous two groups of operations are applied.
Number of captions vs number of selected candidates.
To complement the previous analysis, in Table we report the number of unique candidates extracted by the candidate filtering procedure, averaged over the ten datasets, and with an increasing number of retrieved captions, i.e., 1, 2, 5, 10, and 20. In the table, we show both the number of candidates extracted and the number of selected words. As the number of retrieved captions increases, the unique number of candidate words also increases, i.e., from 3849 with I caption to 28781 with 20. However, the number of selected words stabilizes around 800 as soon as we retrieve more than I caption. Having more captions reduces the noises in the selected words, something that might be present when relying on a single caption.
B Computational cost
We analyze the computational efficiency of CaSED versus BLIP-2 performing VQA and captioning, and report their respective number of parameters and inference time in Table Notably, the methods
We use the NLP library flair hattps://githuu..con/fttt/
14

---

Operations
CA
S-Sim.
S-IoU
Remove
Standardize
Filter
27.7
48.8
15.0
34.1
47.3
15.3
41.3
48.0
16.5
43.1
50.4
17.7
Table Ablation on the candidate filtering operations. Metrics are averaged across the ten datasets.
Num. captions
CA
S-Sim.
S-IoU
Candidates
Selected
30,9
43.2
12.4
3849
1047
35.0
46.4
14.6
6867
854
42.3
50.6
17.5
14548
775
10
43.1
50.4
17.7
22475
794
42.9
49.3
17.1
28781
802
Table 6: Extended ablation the number retrieved captions. Green represents selected configuration. We expand the results Fig. 4 in the main manuscript to show the number unique words extracted from captions (i.e., "candidates")) and the number of words selected by CaSED for classification (i.e., "selected"). Results are averaged across the ten datasets.
using external databases are consistently faster than BLIP-2. For instance, CaSED achieves a speed- up of 1.5x with respect to both the largest captioning model and the largest VQA model, while also achieving better performance. Overall, the fastest method is Closest Caption, which exploits the external database to retrieve single caption and does not consider any candidate extraction pipeline. Conversely, our method retrieves the ten most similar captions and post-processes them extract class name, resulting in increase in inference time of approximately 3 times. Compared with the CLIP upper bound, our method considers multiple additional steps, each adding extra inference time. First, our method retrieves candidates from the external database to then extract the class names. Second, we have to forward the class names through the text encoder for each sample, while CLIP forward them once and cache the features for later reuse.
When using an external database, note that increase in database size implies minimal variation in retrieval time. This is demonstrated by the computational cost required by retrieving from the large LAION-400M [51 database. the results show, inference time comparable between CaSED and CaSED (LAION despite the latter being approximately 8 times larger.
C Additional ablation study
In this section, we show the performance of our model with different backbones, and other commonly used databases for retrieval.
Backbone architecture. To answer this natural question about whether the outcome of our model depends on the backbone architecture, we further extend our main results with a CLIP model with a ResNet50 architecture. We report this additional ablation in Table g Table [9] and Table I0 for the cluster accuracy, the semantic IoU, and the semantic similarity, respectively. We can see that performance with the CLIP ResNet50 is lower across all the metrics compared to CLIP ViT- 1/14. This is expected since ResNet50 is smaller architecture, thus with reduced capacity for semantic representation learning as compared to ViT-L/14. Nevertheless, our method with ResNet50 is still competitive against BLIP-2 models while using 40x fewer parameters (note that VIT-L implementation uses 10x fewer parameters).
Retrieval database. We analyze the impact of retrieving captions from databases of different scales, expanding our main results with the four unused databases PMD, including COCO (37] SBU
Captions 44, Localized Narratives [46]. and Visual Genome 30 Moreover, we evaluate our method on other three databases, namely Ade20K (65), OpenImages (31, and LATON-400M (5])
These databases further extend the range of database sizes, with Ade20k containing only 0.07M
captions and LAION-400M having 413M. We report the results in Table TT] ordering the rows by the database size. Differently from the tables reported in the main results, the results in Table []]
15

---

Method
Num. Params.
Inference time (ms) 4
Closest Caption
0.43B
1390 10
Caption
BLIP-2 (ViT-L)
3,46B
5710 153
BLIP-2 (ViT-g)
4.37B
6870 +: 177
BLIP-2 (VIT-L)
3,46B
5670 + 135
VQA
BLIP-2 (ViT-g)
4.37B
6650 + 117
CaSED CaSED (LAION-400M)
0,43B
4370 = 13
0,43B
4350 + 16
CLIP upper bound
0.43B
645 + 77
Table 7: Computational cost of different methods. Green is our method, gray shows the upper bound, Inference time is reported on batches of size 64, as the average over multiple runs.
Method
Cluster Accuracy ( Food
C101
DTD
ESAT
Airc.
Flwr
Pets
SUN
Cars
UCF
Avg.
WordNet
30.3
18.3
22.5
13.2
47.8
31.4
45.2
26.0
14.2
31.2
28.0
CLIP
English Words
24.8
17.5
18.5
13.4
49.5
23.1
36.6
22.2
15.5
27.1
24.8
CLPP
Caption
Closest Caption
9.7
13.3
8.4
21.2
6.2
8.7
6.5
12.8
14.9
10.9
CaSED
44.6
23.9
12.5
15.3
58.8
48.7
50.1
32.8
24.6
33.9
34.5
CLIP upper bound
82.1
41.5
33.5
19.6
63.1
74.6
78.9
55.6
54.9
58.4
56.2
WordNel
34.0
20.1
16.7
16.7
58.3
40.9
52.0
29.4
18.6
39.5
32.6
JPIP WIT-114
CLIP
English Words
29.1
19.6
22.1
15.9
64.0
30.9
44.4
24.2
19.3
34.5
30.4
Caption
Closest Caption
12.8
8.9
16.7
13.3
28.5
13.1
15.0
8.6
20.0
17.8
15.5
CaSED
51.5
29.1
23.8
22.8
68.7
58.8
60),4
37.4
31.3
47.7
43.1
CLIP upper bound
87.6
52.9
31.8
78.0
89.9
88.0
65.3
76.5
72.5
69.0
Caption
BLIP-2 (VTT-L)
26.5
11.7
23.3
5.4
23.6
12.4
11.6
19.5
14.8
25.7
17.4
BLIP-2 (ViT-g)
37.4
13.0
25.2
10.0
29.5
19.9
15.5
21.5
27.9
32.7
23.3
BLIP-2 (VIT-L)
60.4
20.4
21.4
8.1
36.7
21.3
14.0
32.6
28.8
44.3
28.8
VQA
BLIP-2 (ViT-g)
62.2
23.8
22.0
15.9
57.8
33.4
23.4
36.4
57.2
55.4
38.7
Table 8: Cluster Accuracy on the ten datasets. Green is our method, gray shows the upper bound. Bold represents best, underline indicates best considering also image captioning and VQA models.
are obtained on ImageNet. We first discuss the databases belonging to the PMD superset, and then analyze the results obtained with Ade20K, Openlmages, and LAION-400M. In the table, also show the POS tag distribution (e.g. nouns, adjectives, verbs) of each dataset. Moreover, we report a metric, the semantic similarity to the closest caption in the dataset of each image (C.C. S-Sim) to check how close a database is to the target one (i.e. ImageNet).
By comparing the nine databases of PMD, we can perceive a non non-uniform variation in performance across databases, with a remarkable gap between the top-5 (highlighted in blue ) and the rest. The performance of the fifth-performing (i.e. WIT) and the sixth database (i.e. SBU Captions) differs by 12.8% on cluster accuracy, while the difference between the berf-performing (i.e. CC12M) and the fifth is 10.9%. This observation motivates us to the top-5 databases from PMD instead of the full superset to be efficient. We expand the best database with the other top-5 to ensure better coverage of class names in the textual descriptions.
With the additional databases, i.e. Ade20K, OpenImages, and LAION-400M, we further notice that there is a correlation between the size of a database and its performance. This was also noted in our results reported in the main manuscript. Ade20K, the smallest dataset, comprises only about 0.07M captions and obtains the lowest performance among all metrics. As the size of databases increase, the performance generally improves. However, it important to note that dataset size alone is not the only requirement for good results. LAION-4000 is of much larger size than our split of PMD, yet the performance with LATON-400M is worse, which we attributes to the impact of much noisier retrieval from database of such scale. Instead, datasets such as CC12M and Redcaps perform better than LAION-400M despite being approximately 40x smaller. This finding suggests that the quality of the databases could be of higher importance than its size.
16

---

Method
Semantic IoU
(%)
C101
DTD
ESAT
Aire.
Flwr
Food
Pets
SUN
Cars
UCF
Avg.
WordNet
9.8
1.9
0.9
0.0
21.2
5,5
14.1
7.3
3.5
2.6
6.7
CLIP
CL RN50
English Words
5.1
0.8
0.0
0.1
12.4
1.8
13.2
7.8
2.5
1.3
4.5
Caption
Closest Caption
3.4
0.5
0.1
1.5
6.6
2.2
2.2
2.1
9.1
0.7
2.8
CaSED
31.1
2.9
0.08
2.8
29.7
15.1
27.6
15.0
13.4
5.2
14.3
CLIP upper bound
81.5
41.4
33.9
16.5
60.2
74.6
78.9
56.9
66.5
57.1
56.8
WordNet
15.0
3.0
0.5
31.3
7.8
14.7
9.0
4.8
3.8
9.1
CL.P VIT--114
CLIP
English Words
8.0
2.0
0.0
1.1
16.4
2.0
17.2
8.1
2.7
1.8
5.9
Caption
Closest Caption
4.5
5.9
3.1
3.0
2.3
11.4
1.0
3.5
CaSED
35.4
5.1
2.3
4.8
33.1
19.4
35.1
17.2
16.2
8.4
17.7
CLIP upper bound
86.0
52.2
51.5
28.6
75.7
89.9
88.0
66.6
84.5
71.3
69.4
Caption
BLIP-2 (VIT-L)
13.4
1.4
4.8
0.0
7.5
4.7
1.7
4.7
11.6
1.1
5.1
BLIP-2 (ViT-g)
16.8
1.8
4.1
0.1
13.9
7.9
2.9
5.7
24.7
1.9
8.0
BLIP-2 (ViT-L)
36,1
1.8
7.0
0.1
21.5
3.7
5.7
11,5
18.9
2.5
10.9
VQA
BLIP-2 (ViT-g)
41.5
2.4
7.5
2.0
38.0
8.6
10.2
13.8
33.2
2.8
16.0
Table 9: Semantic IoU on the ten datasets. Green is our method, gray shows the upper bound. Bold represents best, underline indicates best considering also image captioning and VQA models.
Method
Semantic Similarity (x100)
f
C101
DTTD
ESAT
Airc.
Flwr
Food
Pets
SUN
Cars
UCF
Avg.
WordNet
43.2
29.0
18.5
21.6
46.7
44.6
50.3
42.8
26.4
40.0
36.3
CLIP
CLIPRRN0
English Words
36.0
29.5
14.9
20.0
38.1
34.2
40.7
35.4
18.3
32.4
29.9
Caption
Closest Caption
37.2
22.8
14.2
26.8
38.9
41.2
32.6
37.4
44.3
32.4
32.8
CaSED
62.3
36.4
22.6
28.7
52.8
59.0
57.0
50.2
42.9
46.2
45.8
CLIP upper bound
88.1
62.4
52.8
53.9
72.0
83.7
85.8
73.9
81.0
73.9
72.7
WordNet
48.6
32.7
24.4
18.9
55.9
49.6
53.7
44.9
28.8
44.2
40.2
CLIP VIT4
CLIP
English Words
39.3
31.6
19.1
18.6
43.4
38.0
44.2
36.0
19.9
34.7
32.5
Caption
Closest Caption
42.1
23.9
23.4
29.2
40.0
46.9
40.2
39.8
49.2
40.3
37.5
CaSED
65.7
40.0
32.0
30.3
55.5
64.5
62.5
52.5
47.4
54.1
50.4
CLIP upper bound
90.8
69.8
67.7
66.7
83.4
93.7
91.8
80.5
92.3
83.3
82.0
Caption
BLIP-2 (WTT-L)
57.8
31.4
39.9
24.4
36.1
44.6
29.0
45.3
46.4
38.0
39.3
BLIP-2 (ViT-g)
63.0
33.1
36.2
24.3
45.2
51.6
31.6
48.3
61.0
44.6
43.9
BLIP-2 (VIT-L)
70.5
34.9
29.7
29.1
48.8
42.0
40.0
50.6
52.4
48.6
44.7
VQA
BLIP-2 (ViT-g)
73.5
36.5
31.4
30.8
59.9
52.1
43.9
53.3
65.1
55.1
50.1
Table 10: Semantie similarity the ten datasets. Green is ours, gray shows the upper bound.
Bold represents best, underline indicates best considering also image captioning and VQA models.
Another observation pertains the differences between captioning and visual question answer- ing (VQA) datasets (e.g. COCO, Localized Narratives, and Visual Genome) w.r.t. image-text datasets collected from the web (e.g. CC12M, Redcaps, and (AION-400MM) In general, it appears that textual descriptions from captioning and VQA datasets perform worse than datasets scraped from the web. This difference in performance may be due to the type of description provided, as the former often describes specific regions of the image, while the latter provides a general description of the image as a whole. Since our prime objective to understand the class name of the subject in the image (whether it be a location, pet, car model), captions describing secondary objects can lead to sub-optimal results.
Interestingly, we found that the average performance on the 10 datasets of a chosen textual database generally correlates with the accuracy on ImageNet and with the closest caption semantic similarity (C.C. S-Sim in Table IT]. Examples are the subset of PMD and CC12M, achieving the best average results on the 10 datasets according to all metrics (Tab. 4b of the main paper) while being the best on the ImageNet validation set w.r.t. both the closest caption semantic similarity and the ViC metrics. The ImageNet validation set can thus be used as a proxy to pick the textual database in case of a lack of priors on the test set.
17

---

Database
Size
CA
S-Sim.
S-IoU
Nouns (%)
Adjs (%)
Verbs (%)
Concepts
C.C. S-Sim.
Ade20K
0.07M
11.0
31.8
1.7
82.7
10.6
6.7
3.1K
19.0
COCO
0.9M
13.8
35.6
2.6
86.1
9.5
4.4
14.6K
22.6
SBU Captions
1.0M
20.1
5.7
98.0
1.0
0.9
29.3K
26.9
OpenImages
1.M
16.8
37.8
4.2
85.4
10.0
4.5
21.4K
24.8
Loc. Narr.
1.9
15.7
37.1
4.1
86.9
9.8
3.2
30.9K
24.9
CC3M
2.8M
37.9
53.9
16.2
57.9
23.6
18.5
28.4K
35.2
WIT
4.8M
32.9
47.9
14.5
99.8
0.05
0.07
163.7K
30.0
Visual Genome
14.6
3.8
75.1
15.6
26.7
Redcaps
7.9M
41.1
54.7
19.6
98.6
0.6
50.5K
37.5
CC12M
10.3M
43.8
57.4
21.2
96.6
1,8
1.5
36.7K
38.7
YFCC100M
29.9M
39.1
53.5
18.9
99.7
0.2
0.1
179.
37.8
Ours
54.8M
41.5
55.7
20.4
99.8
0.05
0.05
334.1K
38.7
LAION-400M
413.8M
37.7
52.7
18.7
99.4
1.68M
37.3
Table 11: Ablation the databases on ImageNet. Green is our database, Blue shows the top-5 databases of PMD, which we used to create Ours. We also evaluate the semantic similarity of the closest caption to each image in the dataset (i.e., C.C. S-Sim.). Bold represents best, while underline second best. YFCC100M* indicates we use the subset the dataset used in PMD.
For what concerns the POS tag distribution, it is hard to extract any pattern that justifies the perfor- mance of different databases. For instance. CC3M is the only database without an extreme imbalance for nouns (i.c., 57.9% compared to >96% for the other in the top-5) while still achieving good performance. Moreover, while the number of concepts depends on the size of the database, there is no clear correlation between this value and the achieved scores. As example, Localized Narratives has comparable amount concepts w.r.t. CC3M but gap of >20% in cluster accuracy, and of 10 points in semantic similarity and IoU. Finally, it is worth noting that most of the datasets have an average of 12.6 words per caption, while Visual Genome has an average of only 5. words per caption. This characteristic may explain why Visual Genome behaves differently in terms of the database scaling rule of increasing performance with the database size.
D Qualitative results
Last, we report some qualitative results our method applied on three different datasets, namely Caltech-100 101 (Fig. [), Food101 (Fig. 6). and SUN397 7). where the first is coarse, and the last two are fine-grained, focusing on food plates and places respectively. For each, we present batch of five images, where the first three represent success cases and the last two show interesting failure cases. Each sample shows the image we input to our method with the top-5 candidate classes.
From the results, we can see that for many success cases, our method only generates the correct class name and selects it as the best matching label, but also provides valid alternatives for classification. For example, the third image in Fig 5] the second image in Fig 6 where CaSED provides the names "dessert" for the cheesecake and the label "bird" for the ibis. This phenomenon also happens in failure cases, where e.g. the last sample Fig 6 provides both the name "pizza" and the name "margherita" for the dish, despite selecting the wrong name from the set.
Another interesting observation is that our method provides names for different objects in the same scene. For instance, the third and fourth samples in Fontaci labels for both "guacamole" and "tortillas" for the first, and for "mozzarella", "insalata", and "balsamic" for the second. A further detail on the latter is the ability of CLIP to reason in multiple languages since "insalata" translates to "salad" from Italian to English.
Regarding failure cases, is interesting to note that the candidate names and the predicted label often describe well the input image despite being wrong w.r.t the dataset label. For instance, the two failure cases in Fig. select "stadium" and "dumpsite" when the ground-trus -truth class names are "football" and "garbage site". In addition, for the first case, the exact name "football" is still available among the best candidate names, but our method considers "stadium" as a better fit. Another example the last failure case in Fig 5 where the model assigns the name "nokia" to a Nokia 3310 cellphone, while ground-truth class name is "cellphone". Also in this case, the ground-truth label is present the candidate list but method considers "nokia" a more fitting class.
18

---

chandelier
dalmatien
ibis
gramophone
cellphone
chandelier inouugtt antique baraque
phanograph gramophone
noka smatiphhone
cruella
bird doco species
victrola hmv audio
bandana
ce chole mobile
Figure 5: Qualitative results on Caltech-101. The first three samples represent success cases, the last
two shows failure cases.
bibimbap
cheesecake
guacamole
caprese salad
pizza
bibimbsp
deeeseakee
foodspatting
seoul korean lunch kimchi
dessert cake
inselela mozarclla l belsernic mozzarella
molcajete tortila mexicen
mangherita pizzer
served
couls
tal len
Figure 6: Qualitative results on Food101, The first three samples represent success cases, the last two shows failure cases.
galley
football
garbage dump
golley kitchen onboard catararan cabin
shed playhouse gambrel bem gerage
Testaurant
stadium
fumpsite landfill
lodge
fied
terrace
trash
dining
football seating
garbage
Figure 7: Qualitative results SUN397. The first three samples represent success cases, the last two shows failure cases.
Finally, we notice the discovery correlations between terms in the reasoning of our model. In the provided examples, it happens multiple times that the candidate class names do not describe objects in the scene but rather a correlated concept to the image. For instance, the third example in Fig 5 shows Dalmatian, and among the candidate names there is "cruella", which is the name of the villain the movie "The Hundred and One Dalmatians". Another instance of this appears the first example of Fig where the model correctly associates the "bibimbap" dish its place of origin, Korea, with the candidate name "korean".
19


Reviews:
[{"review_id": "EB42NcNObK", "reviewer": "Reviewer_panm", "summary": "The paper presents a method to perform image classification by using the similarity text and image embeddings of a large vision-language model (VLM). The method is called \"vocabulary-free\" because the large vision-language model doesn't have a particular set of classes (vocabulary) in mind, and (contrary to zero-shot approaches), the user does not need to provide a list of prompts. In particular, a set of k closest captions to the image embedding is retrieved from the VLM, and this candidate set is later scored to predict the image class. The underlying VLM is a ViT-L CLIP model.", "questions": "- Have you tried with other strategies to compute the final candidate scores? For instance, why not a log-linear average between the text-to-text and image-to-text scores?", "limitations": "The authors have adequately adressed the limitations and broader impact of the paper in section 6.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper shows very good results when compared with different reasonable baselines (see Table 1, in particular). The experiments use several popular image classification datasets, and use different metrics, and the proposed method is significantly better than the baselines in virtually all cases, and certainly on average.\n- The figures in the paper help understanding the proposed approach (Figure 3) in detail and the difference between \"vocabulary-free\" and zero-shot image classification.\n- The paper contains results from ablation studies used to justify certain design choices (e.g. number of candidate captions retrieved from the database), the importance of the visual and language scores, etc (see section 5.3). I found these experiments very interesting (modulo some questions that were unanswered, see later comments).", "weaknesses": "- The term \"vocabulary-free\" is a bit misleading, since the VLM does have a vocabulary built-in, i.e. the so-called \"semantic space\". It's impossible to classify an image representing an object outside this semantic space. However, it's true that the semantice space is far larger than the class space of any of the datasets used for evaluation (although not necessarily a superset). What the authors mean by \"vocabulary-free\" is precisely defined in the paper (section 3), so this weakness is not critical, but as mentioned before, I find the term not accurate. \n-  My main concern with the paper is regarding the use of the datasets used to train the VLM model used by the proposed method. Section 5.1 mentions that the nearest-neighbours are searched over a collection of 5 large datasets, and the authors ablate the impact of each dataset in section 5.3 (table 4b). However, it's unclear the impact of the dataset used to train the VLM when compared with the baselines. To be more specific: is the proposed CaSED method than BLIP-2 VQA because the dataset used to perform knn is bigger, because the embeddings are better, or because knn+scoring is better than VQA (regardless of the dataset)?\n- Lines 164-165 refer to the number of parameters as a reason for better/worse speed. This is technically wrong. The number of parameters can be (extremely highly) correlated with the speed, but it's not a direct cause of it necessarily. Thus, if arguments about speed are to be made, real runtime/query (on a given hardware) or -at the very least- FLOP/query (amortized over a given batch size) should be given instead.\n\n---------\nUpdate after rebuttal: The authors have (quite successfully) addressed my comments in their rebuttal. Thus, I'm slightly increasing my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1: VLM built-in vocabulary]** Strictly speaking, VLMs, such as CLIP, process sequence of tokens. Thus, they have a possibly infinite vocabulary induced by all the possible combinations of tokens, i.e., the tokenizer can process any string/class name. While there is no performance guarantee for unseen classes, CLIP shows decent extrapolation performance [a]. Therefore, we exploit these properties to avoid using a fixed pre-defined vocabulary. We propose to use large internet-sourced databases, that we assume contain all the concepts available at a certain time with the possibility of further inclusion of new concepts as it evolves. This is better than existing approaches as they cannot recognize classes outside the fixed human-annotated vocabulary, and require additional human effort to update when new concepts appear.\n\nRegarding the use of ”-free” in the name of the task, we reckon the connotation is within the common naming practice in computer vision. One clear example is the accepted term “task-free continual learning” [b]. “-free” in this context does not mean the non-existence of tasks, but rather that task boundaries are not given to the model during training and inference, and the model should infer them on the fly. This is consistent with our definition of “vocabulary-free”, as the vocabulary for each classification task is not given apriori, it is rather inferred (or generated) through our candidate proposal routine. However, we understand that confusion might arise from the term, so we are open to improving the naming in case of suggestions.\n\n[a] Radford, et al. \"Learning transferable visual models from natural language supervision\" ICML 2021.\\\n[b] Aljundi, Rahaf, Klaas Kelchtermans, and Tinne Tuytelaars. \"Task-free continual learning\" CVPR 2019.\n\n\n**[W2: Why is CaSED better than VQA?]** We understand the concerns of the reviewer. However, we believe that our comparisons are fair in terms of data and that the improvement is due to the novel methodology we present. Here are the reasons for this statement:\n- We use the original CLIP model trained on 400M image-text pairs and in addition we retrieve from a 55M text-only samples database (PMD). In comparison, BLIP-2 uses the same CLIP model trained on 400M samples, an additional 129M dataset (roughly a superset of PMD) for training the Q-former and a text-only decoder (OPT or T5) that are trained on trillions of tokens. Therefore it seems clear that our data is roughly a subset of the data used by the strongest baseline (fairness of simpler baselines are described below).\n- All the baselines (including BLIP-2) use the same visual embeddings: CLIP embeddings. Therefore the improvement cannot come from different embeddings.\n- To further investigate if the data is the driver of the improved performance, for this rebuttal we also drafted a new baseline, PMD Words, where we extract a vocabulary composed of all the semantic terms contained in the database that we consider (PMD 55M). This elucidates whether the improvement comes from the lack of concepts in other baselines or from the novel methodology. We report the result in Tab. A1 in the rebuttal PDF. We discover that this baseline produces much worse results than our method. This experiment proves that the increased amount of concepts in larger databases is not the main cause of our performance improvements. In fact, it seems that such information cannot be trivially harnessed, and our method is needed to enable information extraction from these large databases. As demonstrated by our ablations (Tab. 4a), it is our specific design choices (e.g., the multimodal candidate scoring) that contribute to the performance improvements.\n- In the preliminary study (see Fig. 2 and subsection “Semantic space representation”), we showed that captions retrieved from databases are better than the generated captions for classification tasks. This serves as the basis for our method design. It is also the main contributor to the improvement in performance w.r.t. BLIP-2 VQA. In fact, by having a decoder, BLIP-2 is not guaranteed to always output meaningful information, thus hindering the classification performance. For instance, in some cases we found that the list of candidates generated by BLIP-2 contained repeated classes (e.g., [\"dog\", \"dog\", \"dog\"]).\n\n\n**[W3: Speed vs parameters]** We agree with the reviewer that only mentioning the number of parameters in L164-165 is not precise enough and does not directly quantify speed. We will revise the statement accordingly. Actually, we reported the inference time on an NVIDIA A6000 GPU over a batch of size 64 in Tab. 2 of the Supplementary where we observe that retrieval from databases of millions of samples can be extremely efficient. The inference time of our method takes approximately 4370 ms to predict a batch of 64, while the smallest BLIP-2 model (i.e., ViT-L) takes around 5670 ms. Last, we also show that CaSED with our subset of PMD (around 55M captions) or with LAION-400M (around 414M) takes approximately the same inference time, a promising sign for efficient database scaling.\n\n\n**[Q1: Other scoring strategies]** We experimented with the suggested strategy for computing the final candidate scores, by averaging the cosine similarities from the two modalities before applying the softmax transformation. We test this aggregation strategy on the ten datasets used in the main manuscript and report the average, with the relative gain highlighted w.r.t. our approach (averaging probabilities). For cluster accuracy, we achieve 43.0% (-0.1%), for semantic IOU 17.7% (+0.0%), for semantic similarity 50.8 (+0.4). The experiment shows that there is no significant difference between the two approaches.  Instead, our candidate generation and multimodal scoring, as ablated in the main manuscript, are the key components for our advantageous performance. We report the complete results of the suggested ablation in Tab. A1 in the rebuttal PDF."}, {"review_id": "grtGPcfSnv", "reviewer": "Reviewer_m2jc", "summary": "Current approaches to zero shot classification are limited by the assumption of an existing set of candidate classes. This paper presents a novel task, Vocabulary-free Image Classification, where semantic categories need to be automatically mined. This is important when generalising to a new domain, or the domain evolves over time. To address this task, the paper proposes to search for categories from an external database. While conceptually simple, this proves to be an effective approach, and performs better than using captioning models or large lexical databases like  WordNet. The paper also proposes metrics to evaluate on this new problem.", "questions": "1. Cluster Accuracy. When measuring cluster accuracy, one can imagine that in the vocabulary free case, there might be many more predicted labels that in the ground truth (up to the number of samples of the dataset). How is that handled?\n\n+ Please see weaknesses", "limitations": "Then limitations look good. A limitation not discussed is Weaknesses 6. Generalization", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses an important limitation of prior zero-shot classification approaches. \n- The importance of the proposed task, the motivation behind the method, and the method itself, are clearly laid out. \n- The components of the method are well ablated\n- Multiple baselines are proposed and analysed\n- It presents an avenue for future research on how to discover semantic concepts in the V+L space", "weaknesses": "1. Text-to-text score. Using CLIP as a scoring function for image-to-text makes sense. However, it is not obvious that a CLIP text encoder is optimal for the text-to-text score. L207-208 say that the CLIP text encoder ”focuses on the visual elements of the caption”. This could be backed by showing inferior performance when using a text-only model for the text-to-text score. Furthermore, an addition to the semantic space representation experiments (L136-170) would be a model that uses CLIP to retrieve captions, and then a text-only encoder to match the captions to a  semantic class. \n\n2. Reproducibility. The paper does not mention how BLIP was prompted to generate class names\n\n3. Table 4a. The numbers on the last row of table 4a (retrieval with vis + lang scoring) correspond to the avg. values on Tables 1, 2 and 3. However, the first row (BLIP2) does not correspond to any row on these tables. (Related to 2. Reproducibility -- it not clear how the different settings differ)\n\n4. Multimodal scoring results. Table 4 a) shows that scoring with the text-to-text score performs better than image-to-text score, and adding the image-to-text contributes a small amount. This means that the average caption is the most informative signal. That makes the statement on L192-193 \"The visual information is the most reliable source for scoring the candidate categories\" invalid. It is indeed an interesting finding and some intuition behind that would be useful. It would be interesting to see the performance of an ever simpler baseline that only uses text - select the class that has the highest number of occurrences in the retrieved captions. \n\n5. Candidate classes. What is the distribution of the number of extracted candidate classes? And how does it differ as you change the number of retrieved captions?\n\n6. Generalization: One of the motivations behind this work is the applicability of ZS classifiers to domains that might not have a closed set of classes available. However, the current approach relies entirely on datasets of image-text pairs, from which captions are mined. For a domain that is not sufficiently represented in such datasets (e.g. medical images), the proposed method will fail. A discussion of how text-only databases can be used to handle that would be nice to see.\n\n7. Database. In Table 4b you show that simply using the CC12M set of captions performs better than the contracted one (Ours). Why not use that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1: Text-to-text score]** When performing text-to-text scoring with a text-only model, Sentence-BERT [a], and considering only text scores (i.e., alpha=0), we get 39.8% cluster accuracy (-2.9%), 51.9 semantic similarity (+1.6), and 16.6% semantic IoU (-0.4%) w.r.t. CLIP. Thus, CLIP is better at sample grouping and slightly better at the overlap between predicted and ground-truth labels. The semantic similarity is instead better captured by Sentence-BERT: this is not surprising as the latter is also used to compute this metric (L238). The two models are on par when performing multimodal scoring (e.g., +0.5% of CLIP on cluster accuracy while -0.2% on semantic IoU). These results support the use of CLIP to perform text-to-text matching. Note that, nevertheless, we must use a VLM text encoder to retrieve captions in the first place. Thus, it is not possible to fully replace the CLIP text encoder with a text-only model. Thus, using the latter only for the multimodal scoring will lead to an increase in the total number of parameters.\n\n[a] Reimers and Gurevych. \"Sentence-bert: Sentence embeddings using siamese bert-networks\" EMNLP-IJCNLP 2019.\n\n\n**[W2: Details on BLIP-2 prompts]** We followed the BLIP-2 demo for prompting. Specifically, for captioning, we used the prompt “Question: what's in the image? Answer:”. For VQA, we used “Question: what's the name of the object in the image? Answer: a”. We will clarify the prompting in the final manuscript.\n\n\n**[W3: BLIP-2 in Tab. 4]** Tab. 4a ablates how different candidate generation strategies, i.e. generative v.s. retrieval, impact our method performance. In this ablation, we exploit BLIP-2 as the generative module for candidate generation by prompting “Question: list the objects in the image. Answer:”. This generates a list of comma-separated objects that we then parse. This is different from Tab. 1-3, where BLIP-2 is either a captioner or a VQA system. For better clarity, we will replace the name “BLIP-2“ with “Generative” (in contrast to “Retrieval” on the second row) in Tab. 4a to highlight how we generate candidates.\n\n\n**[W4.1: Reliability of image vs text]** We will remove the confusing comment and clarify what we mean. In Tab. 4a we reported three configurations of CaSED, where the best uses both the image-to-text and the text-to-text scores. However, as mentioned in the implementation details, the best value of alpha (i.e., the weight to average the multimodal scores) is 0.7, i.e., 0.7 * image-to-text-score + 0.3 * text-to-text-score, effectively using more visual than textual information. However, from the direct comparison of alpha=0.0 (text-to-text only) and alpha=1.0 (image-to-text only) it is evident that, when using a single modality, it is best to use the caption centroid. This counterintuitive behavior might be due to the modality gap, which makes it harder to linearly interpolate between the two modalities. Finally, note that the captions are retrieved in the aligned feature space using image representations as a query, and therefore they directly depend on the input image.\n\n\n**[W4.2: Occurrences baseline]** As suggested, we modified CaSED to predict semantic names as the most occurring word (CaSED top-1) instead of using multimodal candidate scoring. In case we encounter words with the same occurrences, the method uses our multimodal scoring for prediction. We test this baseline on the ten datasets and report the results in Tab. A1 of the rebuttal PDF. W.r.t our method, CaSED top-1 achieves 36.9% (-6.2%) for cluster accuracy, 14.4% (-3.2%) for semantic IoU and 47.4 (-3.0) for semantic similarity. Thus, simply selecting the most occurring word in the retrieved captions performs worse than scoring the words with our multimodal procedure. Such downgrade may be explained by the lack of any mechanism that considers the semantic similarity between modalities. Indeed, the cosine similarity used by CLIP and our multimodal scoring can effectively filter out terms that are less indicative of the subject in the image.\n\n\n**[W5: Distribution of candidates]** In Tab. A2 of the rebuttal PDF, we report the number of unique candidates extracted by the candidate filtering procedure, averaged over the ten datasets, and with an increasing number of retrieved captions, i.e., 1, 2, 5, 10, and 20. In the table, we show both the number of candidates extracted and the number of selected words. As the number of retrieved captions increases, the unique number of candidate words also increases, i.e., from 3849 with 1 caption to 28781 with 20. However, the number of selected words stabilizes around 800 as soon as we retrieve more than 1 caption. Having more captions reduces the noises in the selected words that might be present in a single caption. \n\n\n**[W6: Generalization]** The databases in our method are as relevant as the training dataset is for trained models. Consequently, it is challenging to retrieve concepts that are not well represented in the database, as it is hard to classify such concepts for trained models. CaSED, as a training-free method, can flexibly mitigate this issue by including new concepts in the database, without any retraining. Additionally, as the reviewer pointed out, our method is not restricted to the use of image-text databases and it is fully compatible with text-only datasets. We will discuss this limitation in the final manuscript.\n\n\n**[W7: Use CC12M]** We opted to use a mixture of web-scale databases (a subset of PMD), instead of cherry-picking the best performing database on ImageNet (CC12M) because we wanted to foster source diversity among databases. This is motivated by the idea that a user could simply download captions from the web without filtering or prior knowledge on the performance to solve the VIC task.\n\n\n**[Q1: Cluster accuracy]** Please, see global response."}, {"review_id": "EEg3WkiFYo", "reviewer": "Reviewer_xx1r", "summary": "This work introduces Vocabulary-free Image Classification (VIC), a task that aims to assign an image to a class in a large, evolving semantic space without a known vocabulary. The proposed method, Category Search from External Databases (CaSED), utilizes a vision-language model and an external database to extract candidate categories and assign the best matching category to the image. Experimental results demonstrate the superiority of CaSED over other frameworks, offering efficiency and promising future research opportunities.", "questions": "Please address my comments in the weakness section. ", "limitations": "N/A", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The strengths of this work can be summarized as follows:\n1.\tExploration of Vocabulary-free Image Classification (VIC) as a task, which overcomes the limitations of existing Vision-Language Models (VLM) for image classification.\n2.\tProposal of CaSED, a training-free method for VIC that utilizes large captioning databases. CaSED does not require additional parameter tuning or fine-tuning of textual and visual encoders.\n3.\tConsistent outperformance of CaSED over a more complex VLM, BLIP-2, on VIC. CaSED achieves superior performance while utilizing significantly fewer parameters.\n4.\tIntroduction of specific evaluation metrics for VIC, providing a valuable reference for future research and benchmarking in this domain.\n", "weaknesses": "1.\tThe most interesting part of this paper is the new task: vocabulary-free image classification. However, it is already studied in the existing work: https://openreview.net/forum?id=sQ0TzsZTUn [1]. The major differences between this paper and the existing work are as follows:\na)\tThe authors of the current paper extend the evaluation by using a larger vocabulary set -- BabelNet, compared to the existing work.\nb)\tThe authors adapt the text-to-text score to find the best matching class.\nFrom the perspective of the reviewer, these two differences may not be considered major advancements. Further analysis and comparison with the existing work are required to fully understand the extent of novelty and contributions of the current paper.\n2.\tTo enhance the clarity and completeness of the paper, it would be beneficial for the authors to provide additional information on how to calculate the classical Cluster Accuracy, particularly considering the open vocabulary nature of the predicted labels. Given that the vocabulary is unconstrained, it is important to explain how the accuracy metric accounts for potential variations in the predicted labels.\n3.\tIt is worth considering that most of the class names in vocabulary-free image classification may consist solely of nouns. Consequently, retrieving information from image captions, which typically contain more diverse linguistic elements, could be inefficient and computationally expensive. This raises a valid concern about the practicality of such an approach, as it may not yield meaningful results and could potentially waste computational resources. The authors should address this issue by discussing alternative strategies or potential optimizations to mitigate the computational burden associated with retrieving information from image captions.\n\n[1] Han, Kai, et al. \"What's in a Name? Beyond Class Indices for Image Recognition.\" arXiv preprint arXiv:2304.02364 (2023).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1: Relationship with [1]]** The cited paper was released on arXiv concurrently (April 5th) with the deadline of NeurIPS (May 17th), making it “contemporaneous” according to the NeurIPS call for papers. The provided OpenReview link leads us to a withdrawn paper from ICLR 2023, with unanimously “below acceptance threshold” scores. Nevertheless, our paper differs on two macro points.\n \nFirst, while the arXiv paper proposes a classification approach based on a fixed vocabulary such as WordNet, our paper utilizes a different methodology by automatically generating the vocabulary using an external database of textual descriptions. In this regard, we would like to stress that we are not using BabelNet, which we mentioned in the manuscript only for statistical purposes. This distinction allows our approach to handle the task in a unique manner that is different from [1] which requires a pre-defined well-curated vocabulary as WordNet.\n\nAdditionally, the arXiv paper relies on accessing all the images beforehand, performing classification and refinement offline. On the other hand, our approach generates the vocabulary independently among samples, working in an online fashion. Our database can be updated online without human supervision, while [1] needs human annotations to introduce new concepts.\n\nTaking these points into consideration, we firmly believe that our work tackles a more general setting than [1]. We are confident that these differences make our contribution valuable and provide significant insights to the field. While we cannot provide a fair comparison with [1], as there is no public implementation available, we are open to including this discussion on the differences in Sec. 2.\n\n\n**[W2: Details on cluster accuracy.]** Please, see the global response.\n\n\n**[W3.1: Captions may not yield meaningful results]** Indeed, captions contain more than just mere class names. However, this is their strength in the context of this task and enables superior performance of our pipeline. CLIP is trained to match images with their respective captions, therefore captions are the form of input that better reflect its training distribution. Using only class names is well known to be suboptimal [a], hence the flourishing field of prompt engineering/tuning for CLIP [b,c]. This is also confirmed by our results in Tab. A1 of the rebuttal PDF. Here, we test CLIP with a vocabulary created from all the words defined in our text database, showing that using the complete list of words is not an effective strategy, achieving the worst results among all baselines.\n\n\n**[W3.2: Computational cost]** From a computational standpoint, captions are only encoded once for creating the faiss index [d], therefore the overhead is negligible. Moreover, storing the embedding of a class name or of a caption has the same memory cost. Finally, as we show in Tab. 2 of the Supplementary, the inference time of our method is still low compared to, e.g., caption generation or VQA. Indeed, our method takes about 4370 ms to classify 64 images on average, while BLIP-2 takes 5670 ms to generate the response. Last, the dimension of the knowledge base does not imply any time overhead, as we achieve the same inference times when we use our database of 55M captions or LAION-400 with its 414M captions.\n\n[a] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision\" ICML 2021.\\\n[b] Zhou, Kaiyang, et al. \"Learning to prompt for vision-language models\" IJCV 2022.\\\n[c] Zhou, Kaiyang, et al. \"Conditional prompt learning for vision-language models\" CVPR. 2022.\\\n[d] Johnson, Jeff, Matthijs Douze, and Hervé Jégou. \"Billion-scale similarity search with gpus\" Transactions on Big Data 2019."}, {"review_id": "mEgmEdYs0D", "reviewer": "Reviewer_3fhc", "summary": "This paper proposes the task of vocabulary-free image classification -  image classification without a predefined set of categories (which is required in zero-shot classification). The paper further proposes the first method for this task - Category Search from External Databases (CaSED). For each given image, it first retrieves captions from an external text corpus, e.g. a large-scale caption dataset. The category names for the classification task are extracted from the retrieved captions via text parsing and filtering. For the classification, a scores fusion is conducted between 1) image-to-text matching score between the input image and the category names, and 2) the text-to-text matching score between the average embedding of retrieved captions and the embeddings of category names. \nThe technical contributions are\n1) The paper proposes the task of vocabulary-free image (VIC), to overcome the constraint of requirement of a predefined set of class names\n2) The paper proposes an approach for VIC task via extracting category names from an external large-scale text corpus", "questions": "1) In Sec 5.2 line 275, the paper mentions that “applying CLIP on top of a pre-defined large vocabulary is not effective” based on the inferior performance on WordNet (117k words) and the English Words (234k words). However, as the first step, CaSED uses CLIP to retrieve K closest captions to an input image from a large-scale text corpus (54.8M captions). Why is it effective here? \n2) For the cluster accuracy (line 242), is it necessary to make sure that the number of predicted labels (number of clusters) should be equal to the number of ground truth labels? How to make sure? What happens if these two are not equal? \n3) For the baselines of BLIP-2 captioning (line 253), each image has one description. How to compute the semantic relevance and the cluster accuracy? How to determine the predicted label from this description? Is the description directly treated as a predicted label?", "limitations": "One missing limitation is that the method requires selection of a text database that is semantically relevant to the contents of the query image dataset. The choice of text corpus might have a big impact on the performance.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper is clearly written and easy to follow \n2) Vocabulary-free image classification is an interesting task as it requires no prior knowledge of semantic class space of visual inputs.", "weaknesses": "1) Some details of baselines and evaluation metric are missing. See questions. \n2) The choice of the caption database (as the text corpus) might have a big impact on the performance. Guidances / criteria of selecting a good text database are missing. \n3) A baseline is missing: WordNet and English words contain different semantic words than words covered by captions in the subset of PMD. It would be interesting to extract the words from the captions in the subset PMD, and apply CLIP for image-text matching. \n4) In Table 4(b) using YFCC100M (29.9M) as the text corpus leads to worse results than the case of CC12M (10.3M) and Redcaps (7.9M). Using CC12M alone leads to better results than using all the five datasets (54.8M). Further explanations are required. \nIt would be interesting to report, besides the number of caption,   a) the number of objects / nouns covered in each database, and b) the relevance of captions in a database to the image dataset, e.g. by reporting the semantic similarities between captions and semantic labels of the image dataset,  semantic similarities between captions and the visual contents of the image dataset.\n\nMinor:\n1) Table 4(b):  asterisk meaning missing (I assume it means a subset of YFCC100M)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W2: How to select databases]** The choice of the database is indeed crucial for CaSED, as shown in Tab. 6 of the Supplementary. We also perform additional experiments and calculate the average semantic similarity between the closest caption retrieved and the ground-truth for all the images in the dataset, reporting the results in Tab. A3 of the rebuttal PDF. This metric correlates well with the database performance across the three metrics. From the tables, we can see that:\n- The quality is more important than the dimension of the database (e.g., YFCC100M subset with 179.6K concepts performs worse than CC12M with 36.7K).\n- Datasets scraped from the web (e.g. CC12M) perform better than specifically annotated ones for captioning/VQA (e.g. Visual Genome). This might be due to the type of the descriptions, more general in the first case.\nDespite these observations, it is hard to find metrics to pre-compute the best database to use in the absence of test labels information. We will discuss this limitation in Sec. 6.\n\n**[W3: PMD Words baseline]** We extracted all the unique words from our PMD subset (around 2.25M words) and used them for classification with CLIP. We tested this new baseline on the ten datasets considered in the paper and we report the results on Tab. A1 of the rebuttal . Compared to CaSED, We achieve 14.2% cluster accuracy (-28.9%), 1.4% semantic IOU (-16.3%), and 27.4 semantic similarity (-23.0).\nThis new baseline achieves the worst performance due to the extreme noise contained in web-scale databases, e.g., typos, neologisms, etc., and the usage of words instead of captions. Our analysis confirms that CLIP achieves suboptimal performance when applied to a list of class names rather than captions, a phenomenon accentuated when the list is uncurated.\n\n**[W4.1: Objects per database]** We extracted all the unique words from the databases and performed POS tagging. We report in Tab. A3 of the rebuttal PDF the statistics for nouns, adjectives and verbs, removing all the other tags. From the analysis of the POS tag distribution, it is hard to extract any pattern that justifies the performance of different databases. For instance, CC3M is the only database without an extreme imbalance for nouns (i.e., 57.9% compared to >96% for the other in the top-5) while still achieving good performance. Moreover, while the number of concepts depends on the size of the database, there is no clear correlation between this value and the achieved scores. As an example, Localized Narratives has a comparable amount of concepts w.r.t. CC3M but a gap of >20% in cluster accuracy, and of ~10 points in semantic similarity and IoU.\n\n**[W4.2: Relevance of captions]** We additionally evaluated the relevance of the captions w.r.t. the class names of ImageNet, the dataset we used for hyperparameter tuning. We report the results in Tab. A3 of the rebuttal PDF. For each image, we retrieve the closest caption in the database and use it to evaluate the semantic similarity with the ground-truth label. We call this metric Closest Caption Semantic Similarity (C.C. S-Sim). Interestingly, the best performing database achieves the highest C.C S-Sim, suggesting that a higher similarity score may correlate with the effectiveness of a database. However, although our custom database achieves the same C.C. S-Sim score as CC12M (i.e., the best database from our analyses), our split still falls behind on the three metrics, suggesting that this score may not be sufficient to assess the quality/relevance of a textual database.\n\n**[W5: YFCC100M asterisk]** Thanks for spotting this. Indeed, we use the subset of YFCC100M, used for PMD (L261 and L305). We will state it clearly in the final manuscript.\n\n**[Q1: Effectiveness of retrieval]** CLIP was trained to match images with their respective captions, therefore captions better reflect the training input distribution used for CLIP. Using only class names is suboptimal [a], hence the flourishing field of prompt engineering/tuning for CLIP [b].\nMoreover, from our preliminary study (Fig. 2 of the main paper), we discovered that captions hold a strong classification signal. When comparing the captions generated by a captioning model (i.e., BLIP-2) with those retrieved from an external source of information, the latter provide a stronger signal, thus justifying our rationale of using retrieved captions for candidate generation.\n[a] Radford, et al. \"Learning transferable visual models from natural language supervision.\" ICML, 2021.\n[b] Zhou, et al. \"Learning to prompt for vision-language models.\" IJCV 2022.\n\n**[Q2: Details on cluster accuracy]** There is no requirement for a one-to-one match between the number of predicted clusters and the number of ground-truth clusters. Please, see the global response for further details.\n\n**[Q3: BLIP-2 captioning]** We ask BLIP-2 to generate a caption given the input image and we directly treat such caption as the predicted label. Interestingly, we noticed that the captions are often repeated for similar input images. Indeed, on average across the ten datasets, BLIP-2 Caption generates 3000 different captions, clustering samples into a finite number of groups. Since we have a number of clusters smaller than the number of images, we can evaluate the cluster accuracy using the captions and the ground-truth labels. For the semantic similarity and IoU, we follow the same procedure we used for the other approaches, evaluating them between the predicted label (i.e., the caption) and the ground-truth class.\n\n**[L1: Database relevance to query]** We expect that testing CaSED using a generic database on a specific dataset (e.g., medical data) might result in bad performance. However, CaSED can be enriched with domain-specific textual information (e.g, medical corpora) without training, making it much more flexible and applicable w.r.t. a fine-tuned model. We will add these considerations in the final version."}]

Author rebuttal:
We thank the reviewers for their valuable feedback. We are especially glad the reviewers appreciated i) the novel task, with a well-thought evaluation protocol and baselines (m2jc, xx1r, 3fhc), ii) the proposed training-free method, which consistently outperforms multiple baselines on several datasets (panm, xx1r), iii) the clear explanations and motivations, with good and interesting ablations to justify our design choices (panm, m2jc, 3fhc).

Here we provide details on the clustering accuracy metric, a shared question among reviewers. Below, we answer specific concerns with comments on the respective reviews. We also attach a PDF file with additional experiments requested by reviewers to further demonstrate the design choices and validate the strengths of our method. In Tab. A1, we compare with new baselines (e.g., larger vocabulary, occurrence-based scoring). In Tab. A2, we report statistics on the distribution of retrieved and selected candidates. Last, in Tab. A3 we expand the database analysis with their relevance and statistics on their composition. From now on, we will refer to the PDF attached to this comment as the "rebuttal PDF".

Regardless of the outcome, we plan to release the complete codebase (metrics, baselines, databases, etc.) to foster further research on the topic.


**[Global response on cluster accuracy]** To evaluate cluster accuracy, we take inspiration from the evaluation protocols used for deep visual clustering [a,b,c]. After grouping the samples by their predicted semantic label, we solve a linear mapping problem to link each predicted cluster to a single ground-truth cluster. This mapping is resolved with a many-to-one match, where a predicted cluster is assigned to the most present ground-truth label. The many-to-one match is justified by the vocabulary-free paradigm. It could happen that a vocabulary-free method generates different clusters for the same ground-truth label simply due to, e.g., synonyms. For instance, a vocabulary-free method could assign labels “insect” and “bug” for images labeled as “insects”. 

Of course, this is prone to degenerate cases. For instance, if all samples are assigned to a unique predicted cluster, the cluster accuracy equals the ratio of the most present class over the dataset size. On the other hand, if we have a number of clusters equal to the number of samples, we would achieve 100% accuracy. However, these phenomena never happen in the methods we test, whether we generate captions (i.e., with BLIP-2), we retrieve from text databases (i.e., Knowledge Base), or we use fixed vocabularies (simple baselines). More precisely, here are the average numbers of predicted clusters:
- CaSED: ~800 clusters
- BLIP-2 (VQA): ~900 clusters
- BLIP-2 (caption): ~3000 clusters
- CLIP + WordNet: ~1300 clusters
- CLIP + English Words: ~1600 clusters
- CLIP + PMD Words: ~3900 clusters

All the methods produce a comparable number of clusters. While having more clusters might improve performance according to how the metric is computed, this does not happen in practice. We believe that selecting good candidates is ultimately more important than having many candidates, as it enables better grouping, increasing clustering performance.

Finally, we would like to highlight that clustering accuracy is only one of the metrics that we exploit. Together with the other metrics, our evaluation paints a clear picture:
- Cluster accuracy evaluates how well predictions are grouped together and consistent
- Semantic similarity assesses how semantically close the predicted and ground truth class are, with quasi-invariance to synonyms and orthographic differences
- Semantic IoU measures how frequently methods output the exact class name or a subset of it

Despite these differences, the ranking of the methods in our experiments is roughly the same for all the metrics, and ours consistently outperforms the others.

[a] Van Gansbeke, et al. "Scan: Learning to classify images without labels" ECCV 2020.\
[b] Ji, Henriques, and Vedaldi. "Invariant information clustering for unsupervised image classification and segmentation" ICCV 2019.\
[c] Han, et al. "Automatically Discovering and Learning New Visual Categories with Ranking Statistics" ICLR 2020.

Meta-review: After the rebuttal and discussion, the majority of reviewers recommend acceptance. There is a remaining concern of reviewer xx1r about contribution and novelty. The AC read the paper, reviews and discussion and also all related work mentioned in the discussion and finds no strong concerns about the contribution or novelty of this submission, especially when disregarding contemporaneous work. The paper proposes lifting the restriction of zero-shot classification to a pre-defined set of classes. As noted by the reviews this might not be fully well-defined as depending on the granularity of the target task the overall vocabulary might still need to change. Additionally, images can be classified into multiple classes which is difficult to model. That said, the task is interesting and the method, although somewhat simple improves over the necessary baselines by a good margin. The field of open-vocabulary classification will benefit from this paper and it should be accepted. Please make sure to update the paper with all the promised changes, discussions, and references for the final version.
Decision: Accept (poster)