Title: Vocabulary-free Image Classification
Venue: NeurIPS 2023
Accepted: yes
Keywords: language and vision

zero-shot classification

image classification
Abstract: Recent advances in large vision-language models have revolutionized the image classification paradigm. Despite showing impressive zero-shot capabilities, a pre-defined set of categories, a.k.a. the vocabulary, is assumed at test time for composing the textual prompts. However, such assumption can be impractical when the semantic context is unknown and evolving. We thus formalize a novel task, termed as Vocabulary-free Image Classification (VIC), where we aim to assign to an input image a class that resides in an unconstrained language-induced semantic space, without the prerequisite of a known vocabulary. VIC is a challenging task as the semantic space is extremely large, containing millions of concepts, with hard-to-discriminate fine-grained categories. In this work, we first empirically verify that representing this semantic space by means of an external vision-language database is the most effective way to obtain semantically relevant content for classifying the image. We then propose Category Search from External Databases (CaSED), a method that exploits a pre-trained vision-language model and an external vision-language database to address VIC in a training-free manner. CaSED first extracts a set of candidate categories from captions retrieved from the database based on their semantic similarity to the image, and then assigns to the image the best matching candidate category according to the same vision-language model. Experiments on benchmark datasets validate that CaSED outperforms other complex vision-language frameworks, while being efficient with much fewer parameters, paving the way for future research in this direction.

Vocabulary-free Image Classification

Abstract

Recent advances in large vision-language models have revolutionized the image classification paradigm. Despite showing impressive zero-shot capabilities, a pre-defined set of categories, a . k . a . the vocabulary, is assumed at test time for composing the textual prompts. However, such assumption can be impractical when the semantic context is unknown and evolving. We thus formalize a novel task, termed as Vocabulary-free Image Classification (VIC), where we aim to assign to an input image a class that resides in an unconstrained language-induced semantic space, without the prerequisite of a known vocabulary. VIC is a challenging task as the semantic space is extremely large, containing millions of concepts, with hardto-discriminate fine-grained categories. In this work, we first empirically verify that representing this semantic space by means of an external vision-language database is the most effective way to obtain semantically relevant content for classifying the image. We then propose Category Search from External Databases (CaSED), a method that exploits a pre-trained vision-language model and an external visionlanguage database to address VIC in a training-free manner. CaSED first extracts a set of candidate categories from captions retrieved from the database based on their semantic similarity to the image, and then assigns to the image the best matching candidate category according to the same vision-language model. Experiments on benchmark datasets validate that CaSED outperforms other complex visionlanguage frameworks, while being efficient with much fewer parameters, paving the way for future research in this direction 1 .

1 Introduction

Large-scale Vision-Language Models (VLMs) [47, 62, 34] enabled astonishing progress in computer vision by aligning multimodal semantics in a shared embedding space. This paper focuses on their use for image classification, where models such as CLIP [47] demonstrated strength in zero-shot transfer. While we witnessed advances in VLM-based classification in many directions, e . g . prompt learning [53, 66], scaling up to larger models and datasets [25, 45, 6], or by jointly considering captioning task [62, 34], they all assume a finite set of target categories, i . e . the vocabulary , to be pre-defined and static (as shown in Fig. 1a). However, this assumption is fragile, as it is often violated in practical applications ( e . g . robotics, autonomous driving) where semantic categories can either differ from the development/training to the deployment/testing or evolve dynamically over time.

In this work, we remove this assumption and study the new task of Vocabulary-free Image Classification (VIC). The objective of VIC is to assign an image to a class that belongs to an unconstrained language-induced semantic space at test time, without a vocabulary , i . e . without a pre-defined set of categories (as shown in Fig. 1b). The unconstrained nature of the semantic space makes VIC a challenging problem. First, the search space is extremely large, with a cardinality on the order

1

Cassowary

Flamingo

Kiwi

Pelican

Cassowary

candidate

estimation

Cassowary

Emu

Peacock

Rhea

Kiwi

Cassowary

Unconstrained

language-induced

semantic space

fU

ft

af

(a) VLM-based classification

(b) Vocabulary-free Image Classification

of millions of semantic concepts 2 , way larger than any existing image classification benchmark ( e . g . ImageNet-21k [10]). Second, it also includes very fine-grained concepts that might be hard to discriminate by the model. Third, as the categories encountered at test time are undefined beforehand, VIC calls for classification methods that do not rely on any vocabulary-aware supervision.

Recent web-scale Vision-Language Databases (VLDs) [51, 54], offer a unique opportunity to address VIC, as they cover a wide set of semantic concepts, ranging from general to highly specific ones. We empirically show that such external databases allow identifying semantic content that is more relevant to the target category than the captioning-enhanced VLMs supporting visual queries ( e . g . BLIP-2 [33]). Motivated by this observation, we propose a training-free method for VIC, Category Search from External Databases (CaSED), which jointly leverages the discriminative multimodal representations derived from CLIP [47] and the information provided by recent VLDs ( e . g . PMD [54]). CaSED operates in two steps: it first coarsely estimates a set of candidate categories for the test image, then it predicts the final category via multimodal matching. Specifically, we first retrieve the captions from the database that are semantically closer to the input image, from which we extract candidate categories via text parsing and filtering. We then estimate the similarity score between the input image and each candidate category via CLIP, using both visual and textual information, predicting as output the best matching candidate. CaSED exploits the pre-trained CLIP without further training, thus being flexible and computationally efficient.

We experiment on several datasets, considering both coarse- ( e . g . Caltech-101 [14], UCF101 [55]) and fine-grained ( e . g . FGVC-Aircraft [40], Flowers-102 [43]) classification tasks. To quantitatively assess the performance of methods addressing VIC, we also propose a set of metrics to measure how well the predicted class matches the semantics of the ground-truth label. Across all tasks and metrics, CaSED consistently outperforms all baselines, including VLMs as complex as BLIP-2 [33]. We believe that thanks to its simplicity and effectiveness, our training-free CaSED can serve as a competitive baseline for future works aiming to address the challenging VIC task.

To summarize, this work provides the following contributions:

We explore the task of Vocabulary-free Image Classification where the goal is to assign a class to an image over an unconstrained set of semantic concepts, overcoming the fundamental assumption of existing VLM-based methods for image classification. We formalize this task and suggest specific evaluation metrics that can be used as a reference for future works.

We propose CaSED, the first method to address VIC thanks to the adoption of large captioning databases. Notably, CaSED is training-free, not requiring any additional parameter nor finetuning of the network's textual and visual encoders.

Our large-scale evaluation demonstrates that CaSED consistently outperforms a more complex VLM such as BLIP-2 [33] on VIC, while requiring much fewer parameters.

2 Related work

Vision-Language Models. Leveraging large-scale datasets with image-text pairs [51, 50, 54], recent works train models by mapping the two modalities into a shared representation space [28, 18, 11, 47, 26, 35, 15]. A notable example is CLIP [47] which, using modality-specific encoders and a contrastive objective to align their output representations, showed remarkable performance on zero-shot classification. Subsequent works improved CLIP by e . g . connecting the two modalities via cross-modal attention [35], multi-object representation alignment [64], learning from weaksupervision [60] or unaligned data [54].

Another line of works improved vision-language pre-training for complex vision-language tasks, such as image captioning and visual question answering (VQA) [62, 22, 34, 1]. In this context, BLIP [34] exploits web data and generated captions to supervise pre-training of a multimodal architecture, outperforming existing VLMs on both captioning and VQA. The current state-of-the-art method BLIP-2 [33] trains a module connecting the two modalities on top of a frozen visual encoder and a frozen large-language model, enabling instructed zero-shot image-to-text generation.

In this work, we challenge a fundamental assumption of zero-shot classification with VLMs: the set of target classes is known a priori. We propose a new task, VIC, which sidesteps this assumption, performing classification in a language-induced open-ended space of semantic categories. We show that even BLIP-2 struggles in this scenario while external multimodal databases provide valuable priors for inferring the semantic category of an image. As a final note, VIC differs from openvocabulary recognition (e.g. [63, 17]) since the latter assumes that the list of target classes is known and available to the model during inference.

Retrieval augmented models. In natural language processing, multiple works showed the benefit of retrieving information from external databases, improving the performance of large language models [19, 32, 3]. In computer vision, such a paradigm has been used mostly to deal with the imbalanced distribution of classes. Examples are [38, 39], addressing long-tail recognition by learning to retrieve training samples [38] or image-text pairs from an external database [39]. Similarly, [58] retrieves images from a given dataset to learn fine-grained visual representations. More recently, retrieval-augmentation has been extended to various types of sources for visual question answering [23], as well as to condition the generative process in diffusion models [2], or image captioning [48]. Our work is close in spirit to [39], as we exploit an external database. However, [39] assumes a pre-defined set of classes (and data) available for training a retrieval module, something we cannot have for the extremely large semantic space of VIC. In CaSED, retrieval is leveraged to first create a set of candidate classes, and to then perform the final class prediction. Moreover, we assume the database to contain only captions, and not necessarily paired image-text data, thus being less memory-demanding.

Finally, the performance of retrieval models is largely affected by the employed database. In the context of VLMs, researchers collected multiple open datasets to study vision-language pretraining. Two notable examples are LAION-5B [51], collected by filtering Common Crawl [8] via CLIP, and the Public Multimodal Datasets (PMD) [54], collecting image-text pairs from different public datasets, such as Conceptual Captions [52, 5], YFCC100M [57], Wikipedia Image Text [56], and Redcaps [12]. In our experiments, we use a subset of PMD as database and investigate how classification performance varies based on the size and quality of the database.

3 Vocabulary-free Image Classification

Preliminaries. Given the image space X and a set of class labels C , a classification model f : X → C is a function mapping an image x ∈ X to its corresponding class c ∈ C . While C may be represented by indices that refer to semantic classes, here we focus on cases where C consists of a set of concept names that correspond to real entities. Note that C is a finite set whose elements belong to the semantic space S , i . e . C ⊂ S . In standard image classification, C is given a priori, and f is learned on a dataset of image-label pairs.

Recently, the introduction of contrastive-based VLMs [47, 25], which learn a cross-modal aligned feature space, revised this paradigm by defining a function f VLM that infers the similarity between an

C101

DTD

ESAT

Airc.

Food

Pets

SUN

Cars

UCF

Average

0

20

40

60

80

100

Accuracy (%)

image and a textual description t ∈ T , i . e . f VLM : X × T → R , with T being the language space. Given this function, classification can be performed with:

where ϕ ( c ) is a string concatenation operation, combining a fixed text template, i . e . a prompt , with a class name. This definition allows for zero-shot transfer, i . e . performing arbitrary classification tasks by re-defining the set C at test time, without re-training the model. However, such zero-shot transfer setup still assumes that the set C is provided. In this work, V ocabulary-free Image Classification (VIC) is formulated to surpass this assumption.

Task definition. VIC aims to assign a class c to an image x without prior knowledge on C , thus operating on the semantic class space S that contains all the possible concepts. Formally, we want to produce a function f mapping an image to a semantic label in S , i . e . f : X → S . Our task definition implies that at test time, the function f has only access to an input image x and a large source of semantic concepts that approximates S . VIC is a challenging classification task by definition due to the extremely large cardinality of the semantic classes in S . As an example, ImageNet21k [10], one of the largest classification benchmarks, is 200 times smaller than the semantic classes in BabelNet [42]. This large search space poses a prime challenge for distinguishing fine-grained concepts across multiple domains as well as ones that naturally follow a long-tailed distribution.

Semantic space representation. As the main challenge of VIC, how to represent the large semantic space plays a fundamental role in the method design. We can either model the multimodal semantic space directly with a VLM equipped with an autoregressive language decoder [34] or via imagetext retrieval from VLDs. Consequently, we can approach VIC either via VQA-enabled VLMs by querying for the candidate class given the input image, or by retrieving and processing data from an external VLD to obtain the candidate class.

To investigate the two potential strategies, we perform a preliminary experimental analysis to understand how well the output of a method semantically captures the image category, or in other words, to assess the alignment of class boundaries in the visual and textual representations. Specifically, we compare the semantic accuracy of querying VQA VLMs and of retrieving from VLDs w.r.t. the ground-truth class labels. We consider the output of a method as correct if its closest textual embedding among the target classes of the dataset corresponds to the ground-truth class of the test sample 3 . We exploit the text encoder of CLIP (ViT-L) [47] to obtain textual embeddings.

Regarding experimented methods, we select BLIP-2 [33] to represent VQA-enabled VLMs for its state-of-the-art performance in VQA benchmarks, while we use a subset of PMD [54] as the VLD. In particular, we compare the following methods: i) BLIP-2 VQA , which directly queries BLIP-2 for the image category; ii) BLIP-2 Captioning , which queries BLIP-2 for the image caption; iii) Closest Caption , which is the closest caption to the image, as retrieved from the database; iv) Caption Centroid , which averages the textual embeddings of the 10 most similar captions to the input image. As we use CLIP embeddings, if visual and textual representations perfectly align, the performance would be the same as zero-shot CLIP with given target classes. We thus report zero-shot CLIP to serve as the upper bound for retrieval accuracy.

We experiment on a variety of test datasets for both coarse- and fine-grained classification (see details in Sec. 5), and report the results in Fig. 2. The average textual embedding of the retrieved captions ( i . e . Caption Centroid) achieves the best semantic accuracy for 9 datasets out of 10, consistently surpassing methods based on BLIP-2. On average, the accuracy achieved by Caption Centroid is 60 . 47% , which is +17 . 36% higher than the one achieved by BLIP-2 Captioning ( 43 . 11% ). Moreover, Captions Centroid achieves results much closer to the CLIP upper bound ( 67 . 17% ) than the other approaches. Notably, such VLD-based retrieval is also computationally more efficient, faster (~ 4 second for a batch size of 64 on a single A6000 GPU), and requires fewer parameters (approximately 10 times less) than BLIP-2 (see Tab. 7 in Appendix B).

The results of this preliminary study clearly suggest that representing the large semantic space with VLDs can produce (via retrieval) semantically more relevant content to the input image, in comparison to querying VQA-enabled VLMs, while being computationally efficient. Based on this conclusion, we develop an approach, Category Search from External Databases (CaSED), that searches for the semantic class from the large semantic space represented in the captions of VLDs.

4 CaSED: Category Search from External Databases

Our proposed method CaSED finds the best matching category within the unconstrained semantic space by multimodal data from large VLDs. Fig. 3 provides an overview of our proposed method. We first retrieve the semantically most similar captions from a database, from which we extract a set of candidate categories by applying text parsing and filtering techniques. We further score the candidates using the multimodal aligned representation of the large pre-trained VLM, i . e . CLIP [47], to obtain the best-matching category. We describe in detail each process in the following.

4.1 Generating candidate categories

We first restrict the extremely large classification space to a few most probable candidate classes. Let f VLM be the pre-trained VLM and D be the external database of image captions. Given an input x , we retrieve the set D x ⊂ D of K closest captions to the input image via

where f v VLM : X → Z is the visual encoder of the VLM, f t VLM : T → Z is the textual encoder, and Z is their shared embedding space. The operation ⟨· , ·⟩ indicates the computation of the cosine similarity. Note that our approach is agnostic to the particular form of D , and that it can accommodate a flexible database size by including captions from additional resources.

From the set D x , we then extract a finite set of candidate classes C x by performing simple text parsing and filtering techniques, e . g . stop-words removal, POS tagging. Details on the filtering procedure can be found in Appendix A.

4.2 Multimodal candidate scoring

Among the small set C x , we score each candidate by accounting for both visual and textual semantic similarities using the VLM encoders, in order to select the best-matching class for the input image.

Image-to-text score. As the image is the main driver for the semantic class we aim to recognize, we use the visual information to score the candidate categories. We denote s v c as the visual score of each

a

v

g

Check out this fierce fowl -

the cassowary means

business!

Check out this prehistoric-

looking bird!

This colorful cassowary is

a true hidden gem of the

animal kingdom.

Animal

Gem

Bird

Cassowary

Fowl

Animal

Bird

Cassowary

Fowl

Gem

Retrieval

from

text

database

Check out this fierce

fowl

-

the

cassowary

means

business!

Check out this prehistoric-

looking

bird

!

This colorful

cassowary

is

a true hidden

gem

of the

animal

kingdom.

…

Cassowary

Gem

Cassowary

…

Cassowary

Gem

Bird

Bird

D

fVLM

▼

Z

candidate category c and compute it as the similarity between the visual representation of the input image and the textual representation of the candidate name:

The higher value of s v c indicates a closer alignment between the target image and the candidate class.

Text-to-text score. While the image-to-text score s v c is effective, there exists a well-known modality gap in the space Z , harming the performance of zero-shot models [36]. As suggested by Fig. 2 in Sec. 3, the semantic relevance of the retrieved captions, and their centroid in particular, is high w.r.t. the underlying ground-truth label. We are therefore motivated to exploit such attributes and introduce a unimodal text-to-text scoring to mitigate the modality gap of cross-modal scoring.

Formally, we define the centroid ¯ d x of the retrieved captions as:

where K is the number of retrieved captions. We then define the text-based matching score s t c as the similarity between the centroid and the candidate category:

A higher value of s t c means a higher alignment between the caption centroid and the candidate embedding. Note that the semantic relevance of the caption centroid is an inherent property of the text encoder of VLMs ( i . e . CLIP). Since CLIP is trained with an image-text alignment loss, its text encoder focuses on the visual elements of the caption, discarding parts that are either non-visual or non-relevant to the visual content. This improves the model's robustness to noisy candidates.

Final predicted candidate. To predict the final candidate, we merge the two scores, obtaining the final score s c for each candidate c as:

where σ ( · ) is the softmax operation on the two scores of each candidate class, and α is a hyperparameter regulating the contribution of the two modalities. Finally we obtain the output category as f ( x ) = arg max c ∈ C x s c . Notably, CaSED respects the VIC task definition, performing classification without known class priors, while being training-free with the use of a pre-trained and frozen VLM. This makes the approach flexible and applicable to a variety of architectures and databases.

5 Experiments

We evaluate CaSED in comparison to other VLM-based methods on the novel task Vocabulary-free Image Classification with extensive benchmark datasets covering both coarse-grained and fine-grained classification. We first describe the experimental protocol in terms of the datasets, the proposed evaluation metrics, and the baselines applicable to this task (Sec. 5.1). We then discuss the quantitative results regarding the comparison between our method and baselines (Sec. 5.2). Finally, we present a thorough ablation to justify the design choices of CaSED (Sec. 5.3). In addition, we provide a cost comparison between the baseline methods and CaSED in Appendix B. We also offer further ablation of our method in Appendix C regarding architecture and database, and showcase qualitative results of predicted categories from multiple datasets in Appendix D.

5.1 Experimental protocol

Datasets. We follow existing works [53, 66] and use ten datasets that feature both coarse-grained and fine-grained classification in different domains: Caltech-101 (C101) [14], DTD [7], EuroSAT (ESAT) [21], FGVC-Aircraft (Airc.) [40], Flowers-102 (Flwr) [43], Food-101 (Food) [4], Oxford Pets (Pets), Stanford Cars (Cars) [29], SUN397 (SUN) [61], and UCF101 (UCF) [55]. Additionally, we used ImageNet [10] for hyperparameters tuning.

Evaluation metrics. Due to the unconstrained nature of the semantic space, evaluating the effectiveness of methods for VIC is not trivial. In this paper, we propose to use two main criteria, namely semantic relevance , i . e . the similarity of the predicted class w.r.t. the ground-truth label, and image grouping , i . e . the quality of the predicted classes for organizing images into clusters. For semantic, we consider i) Semantic Similarity , i . e . the similarity of predicted/ground-truth labels in a semantic space, and ii) Semantic IoU , i . e . the overlap of words between the prediction and the true label. More formally, given an input x with ground-truth label y and prediction ˆ c = f ( x ) , we compute the Semantic Similarity as ⟨ g (ˆ c ) , g ( y ) ⟩ , where g : T → Y is a function mapping text to an embedding space Y . Since we want to model free-form text, we use Sentence-BERT [49] as g . For Semantic IoU , given a predicted label c , and assuming c being a set of words, we compute the Semantic IoU as | c ∩ y | / | c ∪ y | , where y is the set of words in the ground-truth label. To assess grouping, we measure the classic Cluster Accuracy by first clustering images according to their predicted label, and then assigning each cluster to a ground-truth label with Hungarian matching. Sometimes, this mapping is resolved with a many-to-one match, where a predicted cluster is assigned to the most present ground-truth label. This evaluation draws inspiration from the protocols used for deep visual clustering [59, 24, 20].

Baselines. We consider three main groups of baselines for our comparisons. The most straightforward baselines consist of using CLIP with large vocabularies, such as WordNet [41] (117k names) or the English Words (234k names [16]). As an upper bound, we also consider CLIP with the perfect vocabulary, i . e . the ground-truth names of the target dataset (CLIP upper bound). Due to lack of space, we only report results for CLIP with ViT-L [13], while results with other architectures are reported in Appendix C. The second group of baselines consists of captioning methods, as captions can well describe the semantic content of images. We consider two options: captions retrieved from a database and captions generated by a pre-trained image captioning model. For the former we exploit a large collection of textual descriptions, retrieving the most fitting caption for the given image. For the latter, we exploit BLIP-2 [33] - a VLM with remarkable performance on a variety of tasks, including image captioning - to provide a description for the image. The last group of baselines consists of using a VQA model to directly predict the class name associated with the image. Again, we consider BLIP-2 [33], since being highly effective also in VQA. We evaluate BLIP-2 over both ViT-L and ViT-g throughout the experiments 4 .

Implementation details. Our experiments were conducted using NVIDIA A6000 GPUs with mixedbit precision. As database, we use a subset of PMD [54], containing five of its largest datasets: Conceptual Captions (CC3M) [52], Conceptual Captions 12M (CC12M) [5], Wikipedia Image Text (WIT) [56], Redcaps [12], and a subset of [57] used for PMD (YFCC100M*). Further details on the

selection are left for Appendix C. We speed up the retrieval process by embedding the database via the text encoder f t VLM and using fast indexing technique, i . e . FAISS [27]. We tuned the α hyperparameter of Eq. (6) and the number of retrieved captions K of our method on the ImageNet dataset, finding that α = 0 . 7 and K = 10 led to the best results. We use these values across all experiments.

5.2 Quantitative results

The results of CaSED and the baselines are presented in Table 1, Table 2, and Table 3 for Cluster Accuracy ( % ), Semantic Similarity, and Semantic IoU ( % ), respectively. CaSED consistently outperforms all baselines in all metrics on average and in most of the datasets. Notably, CaSED surpasses BLIP-2 (VQA) over ViT-g by +4 . 4% in Cluster Accuracy and +1 . 7% on Semantic IoU, while using much fewer parameters ( i . e . 102M vs 4.1B). The gap is even larger over the same visual backbone, with CaSED outperforming BLIP-2 on ViT-L (VQA) by +14 . 3% on Cluster Accuracy, +5 . 7 in Semantic Similarity, and +6 . 8% on Semantic IoU. These results highlight the effectiveness of CaSED, achieving the best performance both in terms of semantic relevance and image grouping.

An interesting observation from the tables is that simply applying CLIP on top of a pre-defined, large vocabulary, is not effective in VIC. This is due to the challenge of classifying over a huge search space, where class boundaries are hard to model. This is confirmed by the results of CLIP with English Words having a larger search space but performing consistently worse than CLIP with WordNet across all metrics ( e . g . -7 . 7 on Semantic Similarity, -3.2% on Semantic IoU).

Afinal observation relates to the captioning models. Despite their ability to capture the image semantic even in challenging settings ( e . g . 39.3 Semantic Similarity of BLIP-2 ViT-L on ESAT), captions exhibit high variability across images of the same category. This causes the worst performance on Clustering and Semantic IoU across all approaches ( e . g . almost -20% less than CaSED on average in terms of Cluster Accuracy), thus demonstrating that while captions can effectively describe the content of an image, they do not necessarily imply better categorization for VIC.

(a) Ablation on candidate generation and scoring.

(b) Ablation on the database.

5.3 Ablation studies

In this section, we present the ablation study associated with different components of our approach. We first analyze the impact of our retrieval-based candidate selection and multimodal scoring. We then show the results of CaSED for different databases, and how the number of retrieved captions impacts the performance.

Candidates generation. We consider two options to generate the set of candidate classes. The first uses BLIP-2 (ViT-g), asking for multiple candidate labels for the input image. The second is our caption retrieval and filtering strategy. Table 4a shows the results where, for fairness, we score both sets with the same VLM ( i . e . CLIP ViT-L). Our approach consistently outperforms BLIP-2 across all metrics ( e . g . 41.7 vs 23.3 for Cluster Accuracy). This confirms the preliminary results of our study in Fig. 2, with retrieved captions providing better semantic priors than directly using a powerful VLM.

Multimodal scoring. The second ablation studies the effect of our proposed multimodal scoring vs its unimodal counterparts. As Table 4a shows, multimodal candidate scoring provides the best results across all metrics, with clear improvements over the visual modality alone ( i . e . +1.4% of Cluster Accuracy). Notably, scoring via language only partially fills the gap between visual and multimodal scoring ( e . g . +1% on Cluster Accuracy and +1 on Semantic Similarity), confirming that caption centroids contain discriminative semantic information.

Retrieval database. We analyze the impact of retrieving captions from different databases, using five public ones, i . e . CC3M, WIT, Redcaps, CC12M, and a subset of YFCC100M. The databases have different sizes ( e . g . from 2.8M captions of CC3M to 29.9M captions of the YFCC100M subset), and different levels of noise. As shown in Table 4b, the results tend to improve as the size of the database increases ( e . g . +8.9% on Cluster Accuracy over CC3M). However, the quality of the captions influences the performance, with CC12M and Redcaps alone achieving either comparable or slightly better results than the full database. These results suggest that while performance improves with the size of the database, the quality of the captions has a higher impact than the mere quantity.

Number of captions. Finally, we check how performance varies w.r.t. the number of retrieved captions. As shown in Fig. 4, all metrics consistently improve as the number of captions increases from 1 to 10. After that, performance tends to saturate, with a slight decrease in terms of Semantic Similarity for K = 20 . These results suggest that while a minimum number of captions is needed to fully capture the semantics of the image, the possible interference of lessrelated (or even noisy) captions may impact the final performance. Future research may further improve the performance on VIC by focusing on how to deal with noisy retrieval results.

6 Discussion and conclusions

In this work, we proposed a new task, VIC, which operates on an unconstrained semantic space, without assuming a pre-defined set of classes, a brittle assumption of VLM-based classification. We experimentally verified that multimodal databases provide good semantic priors to restrict the large search space, and developed CaSED, an efficient training-free approach that retrieves the closest captions to the input image to extract candidate categories and scores them in a multimodal fashion. On different benchmarks, CaSED consistently achieved better results than more complex VLMs.

Limitations and future works. The performance of CaSED strongly depends on the choice of the retrieval database, with potential issues in retrieving concepts that are not well represented in the latter. Moreover, if the domain of application contains fine-grained concepts, a generic database might not be suitable. Without any prior information on the test labels, it is hard to predict the performance of a database a priori. On the other hand, CaSED can flexibly mitigate this issue by incrementally including new concepts in the database (even domain-specific ones) from textual corpora, without retraining. Future research may explore strategies to automatically select/extend a database based on test samples and/or pre-computing the best database to use in the absence of test label information.

Additionally, as CaSED lacks control over the classes contained in the database, its predictions might reflect potential biases. Improvements in mitigating biases and data quality control would reduce this issue. Another limitation is that CaSED does not keep track of its output history. This may lead to inconsistent predictions, i . e . assigning slightly different labels to images of the same semantic concept ( e . g . cassowary vs Casuarius ). Equipping CaSED with a memory storing the predicted labels may address this issue. Finally, CaSED does not deal with different class granularities: e . g . an image of a cassowary can be as well predicted as a bird . Future works may disambiguate such cases by explicitly integrating the user needs within VIC models.

Broader impact. CaSED addresses VIC in a scalable and training-free manner. We believe that our new problem formulation, metrics, and the effectiveness of CaSED will encourage future research in this topic, overcoming the limiting assumption of VLM-based classification and allowing the power of VLMs to benefit dynamic and unconstrained scenarios.

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS , 2022.

Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Retrieval-augmented diffusion models. NeurIPS , 2022.

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In ICML , 2022.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In ECCV 2014 , 2014.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR , 2021.

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR , 2023.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR , 2014.

Common Crawl. Common crawl, 2023. Accessed May 17, 2023. https://commoncrawl.org/ .

Yufeng Cui, Lichen Zhao, Feng Liang, Yangguang Li, and Jing Shao. Democratizing contrastive languageimage pre-training: A clip benchmark of data, model, and supervision. arXiv , 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In CVPR , 2021.

Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image-text data created by the people, for the people. arXiv , 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshop , 2004.

Enrico Fini, Pietro Astolfi, Adriana Romero-Soriano, Jakob Verbeek, and Michal Drozdzal. Improved baselines for vision-language pre-training, 2023.

FreeBSD. Web2 dictionary (revision 326913), 2023. Accessed May 17, 2023. https://svnweb.freebsd. org/base/head/share/dict/web2?view=markup&pathrev=326913 .

Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In ECCV , 2022.

Lluis Gomez, Yash Patel, Marçal Rusinol, Dimosthenis Karatzas, and CV Jawahar. Self-supervised learning of visual features through embedding images into text topic spaces. In CVPR , 2017.

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In ICML , 2020.

Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statistics. In ICLR , 2019.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Selected Topics in Applied Earth Observations and Remote Sensing , 2019.

Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. In CVPR , 2022.

Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory. In CVPR , 2023.

Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In ICCV , 2019.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , 2021.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , 2021.

Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. Transactions on Big Data , 2019.

Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In ECCV , 2016.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshops , 2013.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV , 2017.

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision , 2020.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledgeintensive nlp tasks. NeurIPS , 2020.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv , 2023.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML , 2022.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS , 2021.

Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. NeurIPS , 2022.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In CVPR , 2019.

Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. In CVPR , 2022.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv , 2013.

George A Miller. Wordnet: a lexical database for english. Communications of the ACM , 1995.

Roberto Navigli and Simone Paolo Ponzetto. Babelnet: Building a very large multilingual semantic network. In ACL , 2010.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing , 2008.

Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. NeurIPS , 2011.

Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer learning. arXiv , 2021.

Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In ECCV , 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Rita Ramos, Desmond Elliott, and Bruno Martins. Retrieval-augmented image captioning. In EACL , 2023.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP-IJCNLP , 2019.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurIPS , 2022.

Christoph Schuhmann, Robert Kaczmarczyk, Aran Komatsuzaki, Aarush Katta, Richard Vencu, Romain Beaumont, Jenia Jitsev, Theo Coombes, and Clayton Mullis. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. In NeurIPS Workshop Datacentric AI , 2021.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Annual Meeting of the Association for Computational Linguistics , 2018.

Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. arXiv , 2022.

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR , 2022.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv , 2012.

Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipediabased image text dataset for multimodal multilingual machine learning. In Research and Development in Information Retrieval , 2021.

Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM , 2016.

Hugo Touvron, Alexandre Sablayrolles, Matthijs Douze, Matthieu Cord, and Hervé Jégou. Grafit: Learning fine-grained image representations with coarse labels. In ICCV , 2021.

Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In ECCV , 2020.

Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In ICLR , 2022.

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Largescale scene recognition from abbey to zoo. In CVPR , 2010.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv , 2022.

Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR , 2021.

Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. In ICML , 2022.

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR , 2017.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV , 2022.

Appendix

A Candidates filtering

With the closest captions retrieved from the external database given the input image (please refer to Sec. 4 of the main manuscript for further details), we post-process them to filter out a set of candidate category names. We first create a set of all words that are contained in the captions. Then we apply sequentially three different groups of operations to the set to (i) remove noisy candidates, (ii) standardize their format, and (iii) filter them.

With the first group of operations, we remove all the irrelevant textual contents, such as tokens ( i . e . "<PERSON>"), URLs, or file extensions. Note that, for the file extensions, we remove the extension but retain the file name as it might contain candidate class names. We also remove all the words that are shorter than three characters and split compound words by underscores or dashes. Finally, we remove all those terms containing symbols or numbers and meta words that are irrelevant to the classification task, such as "image", "photo", or "thumbnail". As shown in Table 5, when compared to having no operation for candidate filtering (first row), this set of operations removes inappropriate content and increases the accuracy of clusters by +6 . 4% and improves the semantic IoU by +0 . 3 . However, we can observe a drop in semantic similarity by -1 . 5 . This might be due to the removal of unnatural words that could still describe well the content of the image, i . e . underlineor dash-separated words, or URLs since they are longer w.r.t. natural words.

The second group of operations standardize the candidate names by aligning words that refer to the same semantic class to a standard format, reducing class redundancy. For example, "cassowary" and "Cassowary" will be considered as a single class instead of two. To this end, we perform two operations-lowercase conversion and singular form conversion. With such standardizing conversions, we observe a sizeable boost in terms of performances when compared to the results obtained by applying only removal-related operations. As shown in Table 5, we achieve higher results across all three metrics, leading to a relative improvement of +7 . 2% , +0 . 7 , and +1 . 2 in terms of cluster accuracy, semantic similarity, and semantic IoU, respectively.

The last group of operations considers two forms of filtering, where the first aims to filter out entire categories of words via Part-Of-Speech (POS) tag and the second aims to filter out rare and noisy contents based on the word occurrences. We select these two operations since common dataset class names do not contain terms that carry no semantics e . g . articles and pronouns, and since [9] showed that CLIP performs better when exposed to a smaller amount of unique tokens. The POS tagging 5 categorizes words into groups, such as adjectives, articles, nouns, or verbs, enabling us to filter all the terms that are not semantically meaningful for a classification task. Regarding the occurrence filtering, we first count how often a word appears in the retrieved captions and then we remove words that appear only once to make the candidate list less noisy. We can see from Table 5 that the inclusion of this final set of operations scores the best among all three metrics when compared to the results obtained when only the previous two groups of operations are applied.

Number of captions vs number of selected candidates.

To complement the previous analysis, in Table 6, we report the number of unique candidates extracted by the candidate filtering procedure, averaged over the ten datasets, and with an increasing number of retrieved captions, i.e., 1, 2, 5, 10, and 20. In the table, we show both the number of candidates extracted and the number of selected words. As the number of retrieved captions increases, the unique number of candidate words also increases, i.e., from 3849 with 1 caption to 28781 with 20. However, the number of selected words stabilizes around 800 as soon as we retrieve more than 1 caption. Having more captions reduces the noises in the selected words, something that might be present when relying on a single caption.

B Computational cost

We analyze the computational efficiency of CaSED versus BLIP-2 performing VQA and captioning, and report their respective number of parameters and inference time in Table 7. Notably, the methods

using external databases are consistently faster than BLIP-2. For instance, CaSED achieves a speedup of 1.5x with respect to both the largest captioning model and the largest VQA model, while also achieving better performance. Overall, the fastest method is Closest Caption, which exploits the external database to retrieve a single caption and does not consider any candidate extraction pipeline. Conversely, our method retrieves the ten most similar captions and post-processes them to extract class name, resulting in a increase in inference time of approximately 3 times. Compared with the CLIP upper bound, our method considers multiple additional steps, each adding extra inference time. First, our method retrieves candidates from the external database to then extract the class names. Second, we have to forward the class names through the text encoder for each sample, while CLIP can forward them once and cache the features for later reuse.

When using an external database, note that an increase in database size implies a minimal variation in retrieval time. This is demonstrated by the computational cost required by retrieving from the large LAION-400M [51] database. As the results show, inference time is comparable between CaSED and CaSED (LAION-400M) despite the latter being approximately 8 times larger.

C Additional ablation study

In this section, we show the performance of our model with different backbones, and other commonly used databases for retrieval.

Backbone architecture. To answer this natural question about whether the outcome of our model depends on the backbone architecture, we further extend our main results with a CLIP model with a ResNet50 architecture. We report this additional ablation in Table 8, Table 9, and Table 10 for the cluster accuracy, the semantic IoU, and the semantic similarity, respectively. We can see that the performance with the CLIP ResNet50 is lower across all the metrics compared to CLIP ViTL/14. This is expected since ResNet50 is a a smaller architecture, thus with a reduced capacity for semantic representation learning as compared to ViT-L/14. Nevertheless, our method with ResNet50 is still competitive against BLIP-2 models while using 40x fewer parameters (note that our ViT-L implementation uses 10x fewer parameters).

Retrieval database. We analyze the impact of retrieving captions from databases of different scales, expanding our main results with the four unused databases of PMD, including COCO [37], SBU Captions [44], Localized Narratives [46], and Visual Genome [30]. Moreover, we evaluate our method on other three databases, namely Ade20K [65], OpenImages [31], and LAION-400M [51]. These databases further extend the range of database sizes, with Ade20k containing only 0.07M captions and LAION-400M having 413M. We report the results in Table 11, ordering the rows by the database size. Differently from the tables reported in the main results, the results in Table 11

are obtained on ImageNet. We first discuss the databases belonging to the PMD superset, and then analyze the results obtained with Ade20K, OpenImages, and LAION-400M. In the table, we also show the POS tag distribution (e.g. nouns, adjectives, verbs) of each dataset. Moreover, we report a metric, the semantic similarity to the closest caption in the dataset of each image (C.C. S-Sim) to check how close a database is to the target one (i.e. ImageNet).

By comparing the nine databases of PMD, we can perceive a non-uniform variation in performance across databases, with a remarkable gap between the top-5 (highlighted in blue ) and the rest. The performance of the fifth-performing ( i . e . WIT) and the sixth database ( i . e . SBU Captions) differs by 12 . 8% on cluster accuracy, while the difference between the best-performing ( i . e . CC12M) and the fifth is 10 . 9% . This observation motivates us to use the top-5 databases from PMD instead of the full superset to be efficient. We expand the best database with the other top-5 to ensure better coverage of class names in the textual descriptions.

With the additional databases, i . e . Ade20K, OpenImages, and LAION-400M, we further notice that there is a correlation between the size of a database and its performance. This was also noted in our results reported in the main manuscript. Ade20K, the smallest dataset, comprises only about 0.07M captions and obtains the lowest performance among all metrics. As the size of databases increase, the performance generally improves. However, it is important to note that dataset size alone is not the only requirement for good results. LAION-400M is of a much larger size than our split of PMD, yet the performance with LAION-400M is worse, which we attributes to the impact of much noisier retrieval from database of such scale. Instead, datasets such as CC12M and Redcaps perform better than LAION-400M despite being approximately 40x smaller. This finding suggests that the quality of the databases could be of a higher importance than its size.

Another observation pertains to the differences between captioning and visual question answering (VQA) datasets ( e . g . COCO, Localized Narratives, and Visual Genome) w.r.t. image-text datasets collected from the web ( e . g . CC12M, Redcaps, and LAION-400M). In general, it appears that textual descriptions from captioning and VQA datasets perform worse than datasets scraped from the web. This difference in performance may be due to the type of description provided, as the former often describes specific regions of the image, while the latter provides a general description of the image as a whole. Since our prime objective is to understand the class name of the subject in the image (whether it be a location, a pet, or a car model), captions describing secondary objects can lead to sub-optimal results.

Interestingly, we found that the average performance on the 10 datasets of a chosen textual database generally correlates with the accuracy on ImageNet and with the closest caption semantic similarity (C.C. S-Sim in Table 11). Examples are the subset of PMD and CC12M, achieving the best average results on the 10 datasets according to all metrics (Tab. 4b of the main paper) while being the best on the ImageNet validation set w.r.t. both the closest caption semantic similarity and the ViC metrics. The ImageNet validation set can thus be used as a proxy to pick the textual database in case of a lack of priors on the test set.

For what concerns the POS tag distribution, it is hard to extract any pattern that justifies the performance of different databases. For instance, CC3M is the only database without an extreme imbalance for nouns (i.e., 57.9% compared to >96% for the other in the top-5) while still achieving good performance. Moreover, while the number of concepts depends on the size of the database, there is no clear correlation between this value and the achieved scores. As an example, Localized Narratives has a comparable amount of concepts w.r.t. CC3M but a gap of >20% in cluster accuracy, and of 10 points in semantic similarity and IoU. Finally, it is worth noting that most of the datasets have an average of 12.6 words per caption, while Visual Genome has an average of only 5.1 words per caption. This characteristic may explain why Visual Genome behaves differently in terms of the database scaling rule of increasing performance with the database size.

D Qualitative results

Last, we report some qualitative results of our method applied on three different datasets, namely Caltech-101 (Fig. 5), Food101 (Fig. 6), and SUN397 (Fig. 7), where the first is coarse, and the last two are fine-grained, focusing on food plates and places respectively. For each, we present a batch of five images, where the first three represent success cases and the last two show interesting failure cases. Each sample shows the image we input to our method with the top-5 candidate classes.

From the results, we can see that for many success cases, our method not only generates the correct class name and selects it as the best matching label, but it also provides valid alternatives for classification. For example, the third image in Fig 5 or the second image in Fig 6, where CaSED provides the names "dessert" for the cheesecake and the label "bird" for the ibis. This phenomenon also happens in failure cases, where e . g . the last sample in Fig 6 provides both the name "pizza" and the name "margherita" for the dish, despite selecting the wrong name from the set.

Another interesting observation is that our method provides names for different objects in the same scene. For instance, the third and fourth samples in Fig 6 contain labels for both "guacamole" and "tortillas" for the first, and for "mozzarella", "insalata", and "balsamic" for the second. A further detail on the latter case is the ability of CLIP to reason in multiple languages since "insalata" translates to "salad" from Italian to English.

Regarding failure cases, it is interesting to note that the candidate names and the predicted label often describe well the input image despite being wrong w.r.t the dataset label. For instance, the two failure cases in Fig. 7 select "stadium" and "dumpsite" when the ground-truth class names are "football" and "garbage site". In addition, for the first case, the exact name "football" is still available among the best candidate names, but our method considers "stadium" as a better fit. Another example is the last failure case in Fig 5, where the model assigns the name "nokia" to a Nokia 3310 cellphone, while the ground-truth class name is "cellphone". Also in this case, the ground-truth label is present in the candidate list but our method considers "nokia" a more fitting class.

chandelier

dalmatian

ibis

gramophone

cellphone

chandelier

dalmatian

ibis

phonograph

nokia

candelabra

6op

stork

gramophone

phone

wrought

cruella

bird

victrola

smartphone

antique

bandana

opop

hmv

cellphone

baroque

png

species

audio

mobile

bibimbap

cheesecake

guacamole

capresesalad

pizza

bibimbap

cheesecake

guacamole

caprese

foodspotting

seoul

dessert

guac

insalata

pizza

korean

cake

molcajete

mozarella

lunch

served

tortilla

balsamic

pizzeria

kimchi

coulis

mexican

mozzarella

italian

shed

restaurant

football

garbagedump

shed

restaurant

stadium

dumpsite

kitchen

playhouse

lodge

field

landfill

onboard

gambrel

terrace

nfl

trash

catamaran

barn

dining

football

garbage

cabin

garage

interior

seating

rubbish

Finally, we notice the discovery of correlations between terms in the reasoning of our model. In the provided examples, it happens multiple times that the candidate class names do not describe objects in the scene but rather a correlated concept to the image. For instance, the third example in Fig 5 shows a Dalmatian, and among the candidate names there is "cruella", which is the name of the villain of the movie "The Hundred and One Dalmatians". Another instance of this appears in the first example of Fig 6, where the model correctly associates the "bibimbap" dish to its place of origin, Korea, with the candidate name "korean".

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the task of vocabulary-free image classification -  image classification without a predefined set of categories (which is required in zero-shot classification). The paper further proposes the first method for this task - Category Search from External Databases (CaSED). For each given image, it first retrieves captions from an external text corpus, e.g. a large-scale caption dataset. The category names for the classification task are extracted from the retrieved captions via text parsing and filtering. For the classification, a scores fusion is conducted between 1) image-to-text matching score between the input image and the category names, and 2) the text-to-text matching score between the average embedding of retrieved captions and the embeddings of category names. \nThe technical contributions are\n1) The paper proposes the task of vocabulary-free image (VIC), to overcome the constraint of requirement of a predefined set of class names\n2) The paper proposes an approach for VIC task via extracting category names from an external large-scale text corpus\n\nSTRENGTHS:\n1) The paper is clearly written and easy to follow \n2) Vocabulary-free image classification is an interesting task as it requires no prior knowledge of semantic class space of visual inputs.\n\nWEAKNESSES:\n1) Some details of baselines and evaluation metric are missing. See questions. \n2) The choice of the caption database (as the text corpus) might have a big impact on the performance. Guidances / criteria of selecting a good text database are missing. \n3) A baseline is missing: WordNet and English words contain different semantic words than words covered by captions in the subset of PMD. It would be interesting to extract the words from the captions in the subset PMD, and apply CLIP for image-text matching. \n4) In Table 4(b) using YFCC100M (29.9M) as the text corpus leads to worse results than the case of CC12M (10.3M) and Redcaps (7.9M). Using CC12M alone leads to better results than using all the five datasets (54.8M). Further explanations are required. \nIt would be interesting to report, besides the number of caption,   a) the number of objects / nouns covered in each database, and b) the relevance of captions in a database to the image dataset, e.g. by reporting the semantic similarities between captions and semantic labels of the image dataset,  semantic similarities between captions and the visual contents of the image dataset.\n\nMinor:\n1) Table 4(b):  asterisk meaning missing (I assume it means a subset of YFCC100M)\n\nQUESTIONS:\n1) In Sec 5.2 line 275, the paper mentions that “applying CLIP on top of a pre-defined large vocabulary is not effective” based on the inferior performance on WordNet (117k words) and the English Words (234k words). However, as the first step, CaSED uses CLIP to retrieve K closest captions to an input image from a large-scale text corpus (54.8M captions). Why is it effective here? \n2) For the cluster accuracy (line 242), is it necessary to make sure that the number of predicted labels (number of clusters) should be equal to the number of ground truth labels? How to make sure? What happens if these two are not equal? \n3) For the baselines of BLIP-2 captioning (line 253), each image has one description. How to compute the semantic relevance and the cluster accuracy? How to determine the predicted label from this description? Is the description directly treated as a predicted label?\n\nLIMITATIONS:\nOne missing limitation is that the method requires selection of a text database that is semantically relevant to the contents of the query image dataset. The choice of text corpus might have a big impact on the performance.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces Vocabulary-free Image Classification (VIC), a task that aims to assign an image to a class in a large, evolving semantic space without a known vocabulary. The proposed method, Category Search from External Databases (CaSED), utilizes a vision-language model and an external database to extract candidate categories and assign the best matching category to the image. Experimental results demonstrate the superiority of CaSED over other frameworks, offering efficiency and promising future research opportunities.\n\nSTRENGTHS:\nThe strengths of this work can be summarized as follows:\n1.\tExploration of Vocabulary-free Image Classification (VIC) as a task, which overcomes the limitations of existing Vision-Language Models (VLM) for image classification.\n2.\tProposal of CaSED, a training-free method for VIC that utilizes large captioning databases. CaSED does not require additional parameter tuning or fine-tuning of textual and visual encoders.\n3.\tConsistent outperformance of CaSED over a more complex VLM, BLIP-2, on VIC. CaSED achieves superior performance while utilizing significantly fewer parameters.\n4.\tIntroduction of specific evaluation metrics for VIC, providing a valuable reference for future research and benchmarking in this domain.\n\nWEAKNESSES:\n1.\tThe most interesting part of this paper is the new task: vocabulary-free image classification. However, it is already studied in the existing work: https://openreview.net/forum?id=sQ0TzsZTUn [1]. The major differences between this paper and the existing work are as follows:\na)\tThe authors of the current paper extend the evaluation by using a larger vocabulary set -- BabelNet, compared to the existing work.\nb)\tThe authors adapt the text-to-text score to find the best matching class.\nFrom the perspective of the reviewer, these two differences may not be considered major advancements. Further analysis and comparison with the existing work are required to fully understand the extent of novelty and contributions of the current paper.\n2.\tTo enhance the clarity and completeness of the paper, it would be beneficial for the authors to provide additional information on how to calculate the classical Cluster Accuracy, particularly considering the open vocabulary nature of the predicted labels. Given that the vocabulary is unconstrained, it is important to explain how the accuracy metric accounts for potential variations in the predicted labels.\n3.\tIt is worth considering that most of the class names in vocabulary-free image classification may consist solely of nouns. Consequently, retrieving information from image captions, which typically contain more diverse linguistic elements, could be inefficient and computationally expensive. This raises a valid concern about the practicality of such an approach, as it may not yield meaningful results and could potentially waste computational resources. The authors should address this issue by discussing alternative strategies or potential optimizations to mitigate the computational burden associated with retrieving information from image captions.\n\n[1] Han, Kai, et al. \"What's in a Name? Beyond Class Indices for Image Recognition.\" arXiv preprint arXiv:2304.02364 (2023).\n\nQUESTIONS:\nPlease address my comments in the weakness section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCurrent approaches to zero shot classification are limited by the assumption of an existing set of candidate classes. This paper presents a novel task, Vocabulary-free Image Classification, where semantic categories need to be automatically mined. This is important when generalising to a new domain, or the domain evolves over time. To address this task, the paper proposes to search for categories from an external database. While conceptually simple, this proves to be an effective approach, and performs better than using captioning models or large lexical databases like  WordNet. The paper also proposes metrics to evaluate on this new problem.\n\nSTRENGTHS:\n- The paper addresses an important limitation of prior zero-shot classification approaches. \n- The importance of the proposed task, the motivation behind the method, and the method itself, are clearly laid out. \n- The components of the method are well ablated\n- Multiple baselines are proposed and analysed\n- It presents an avenue for future research on how to discover semantic concepts in the V+L space\n\nWEAKNESSES:\n1. Text-to-text score. Using CLIP as a scoring function for image-to-text makes sense. However, it is not obvious that a CLIP text encoder is optimal for the text-to-text score. L207-208 say that the CLIP text encoder ”focuses on the visual elements of the caption”. This could be backed by showing inferior performance when using a text-only model for the text-to-text score. Furthermore, an addition to the semantic space representation experiments (L136-170) would be a model that uses CLIP to retrieve captions, and then a text-only encoder to match the captions to a  semantic class. \n\n2. Reproducibility. The paper does not mention how BLIP was prompted to generate class names\n\n3. Table 4a. The numbers on the last row of table 4a (retrieval with vis + lang scoring) correspond to the avg. values on Tables 1, 2 and 3. However, the first row (BLIP2) does not correspond to any row on these tables. (Related to 2. Reproducibility -- it not clear how the different settings differ)\n\n4. Multimodal scoring results. Table 4 a) shows that scoring with the text-to-text score performs better than image-to-text score, and adding the image-to-text contributes a small amount. This means that the average caption is the most informative signal. That makes the statement on L192-193 \"The visual information is the most reliable source for scoring the candidate categories\" invalid. It is indeed an interesting finding and some intuition behind that would be useful. It would be interesting to see the performance of an ever simpler baseline that only uses text - select the class that has the highest number of occurrences in the retrieved captions. \n\n5. Candidate classes. What is the distribution of the number of extracted candidate classes? And how does it differ as you change the number of retrieved captions?\n\n6. Generalization: One of the motivations behind this work is the applicability of ZS classifiers to domains that might not have a closed set of classes available. However, the current approach relies entirely on datasets of image-text pairs, from which captions are mined. For a domain that is not sufficiently represented in such datasets (e.g. medical images), the proposed method will fail. A discussion of how text-only databases can be used to handle that would be nice to see.\n\n7. Database. In Table 4b you show that simply using the CC12M set of captions performs better than the contracted one (Ours). Why not use that?\n\nQUESTIONS:\n1. Cluster Accuracy. When measuring cluster accuracy, one can imagine that in the vocabulary free case, there might be many more predicted labels that in the ground truth (up to the number of samples of the dataset). How is that handled?\n\n+ Please see weaknesses\n\nLIMITATIONS:\nThen limitations look good. A limitation not discussed is Weaknesses 6. Generalization", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method to perform image classification by using the similarity text and image embeddings of a large vision-language model (VLM). The method is called \"vocabulary-free\" because the large vision-language model doesn't have a particular set of classes (vocabulary) in mind, and (contrary to zero-shot approaches), the user does not need to provide a list of prompts. In particular, a set of k closest captions to the image embedding is retrieved from the VLM, and this candidate set is later scored to predict the image class. The underlying VLM is a ViT-L CLIP model.\n\nSTRENGTHS:\n- The paper shows very good results when compared with different reasonable baselines (see Table 1, in particular). The experiments use several popular image classification datasets, and use different metrics, and the proposed method is significantly better than the baselines in virtually all cases, and certainly on average.\n- The figures in the paper help understanding the proposed approach (Figure 3) in detail and the difference between \"vocabulary-free\" and zero-shot image classification.\n- The paper contains results from ablation studies used to justify certain design choices (e.g. number of candidate captions retrieved from the database), the importance of the visual and language scores, etc (see section 5.3). I found these experiments very interesting (modulo some questions that were unanswered, see later comments).\n\nWEAKNESSES:\n- The term \"vocabulary-free\" is a bit misleading, since the VLM does have a vocabulary built-in, i.e. the so-called \"semantic space\". It's impossible to classify an image representing an object outside this semantic space. However, it's true that the semantice space is far larger than the class space of any of the datasets used for evaluation (although not necessarily a superset). What the authors mean by \"vocabulary-free\" is precisely defined in the paper (section 3), so this weakness is not critical, but as mentioned before, I find the term not accurate. \n-  My main concern with the paper is regarding the use of the datasets used to train the VLM model used by the proposed method. Section 5.1 mentions that the nearest-neighbours are searched over a collection of 5 large datasets, and the authors ablate the impact of each dataset in section 5.3 (table 4b). However, it's unclear the impact of the dataset used to train the VLM when compared with the baselines. To be more specific: is the proposed CaSED method than BLIP-2 VQA because the dataset used to perform knn is bigger, because the embeddings are better, or because knn+scoring is better than VQA (regardless of the dataset)?\n- Lines 164-165 refer to the number of parameters as a reason for better/worse speed. This is technically wrong. The number of parameters can be (extremely highly) correlated with the speed, but it's not a direct cause of it necessarily. Thus, if arguments about speed are to be made, real runtime/query (on a given hardware) or -at the very least- FLOP/query (amortized over a given batch size) should be given instead.\n\n---------\nUpdate after rebuttal: The authors have (quite successfully) addressed my comments in their rebuttal. Thus, I'm slightly increasing my score.\n\nQUESTIONS:\n- Have you tried with other strategies to compute the final candidate scores? For instance, why not a log-linear average between the text-to-text and image-to-text scores?\n\nLIMITATIONS:\nThe authors have adequately adressed the limitations and broader impact of the paper in section 6.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}