SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alessandro Conti<sup>1</sup> Enrico Fini<sup>1</sup> Massimiliano Mancini<sup>1</sup>
Paolo Rota<sup>1</sup> Yiming Wang<sup>2</sup> Elisa Ricci<sup>1,2</sup>
<sup>1</sup>University of Trento <sup>2</sup>Fondazione Bruno Kessler (FBK)
Recent advances in large vision-language models have revolutionized the image classification paradigm. Despite showing impressive zero-shot capabilities, a pre-defined set of categories, a.k.a. the vocabulary, is assumed at test time for composing the textual prompts. However, such assumption can be impractical when the semantic context is unknown and evolving. We thus formalize a novel task, termed as Vocabulary-free Image Classification (VIC), where we aim to assign to an input image a class that resides in an unconstrained language-induced semantic space, without the prerequisite of a known vocabulary. VIC is a challenging task as the semantic space is extremely large, containing millions of concepts, with hardly-discriminate fine-grained categories. In this work, we first empirically verify that representing this semantic space by means of an external vision-language database is the most effective way to obtain semantically relevant content for classifying the image. We then propose Category Search from External Databases (CaSED), a method that exploits a pre-trained vision-language model and an external vision-language database to address VIC in a training-free manner. CaSED first extracts a set of candidate categories from captions retrieved from the database based on their semantic similarity to the image, and then assigns to the image the best matching candidate category according to the same vision-language model. Experiments on benchmark datasets validate that CaSED outperforms other complex vision-language frameworks, while being efficient with much fewer parameters, paving the way for future research in this direction<sup>1</sup>.
Large-scale Vision-Language Models (VLMs) [47, 62, 34] enabled astonishing progress in computer vision by aligning multimodal semantics in a shared embedding space. This paper focuses on their use for image classification, where models such as CLIP [47] demonstrated strength in zero-shot transfer. While we witnessed advances in VLM-based classification in many directions, e.g. prompt learning [53, 66], scaling up to larger models and datasets [25, 45, 6], or by jointly considering captioning task [62, 34], they all assume a finite set of target categories, i.e. the vocabulary, to be pre-defined and static (as shown in Fig. 1a). However, this assumption is fragile, as it is often violated in practical applications (e.g. robotics, autonomous driving) where semantic categories can either differ from the development/training to the deployment/testing or evolve dynamically over time.
In this work, we remove this assumption and study the new task of Vocabulary-free Image Classification (VIC). The objective of VIC is to assign an image to a class that belongs to an unconstrained language-induced semantic space at test time, without a vocabulary, i.e. without a pre-defined set of categories (as shown in Fig. 1b). The unconstrained nature of the semantic space makes VIC a challenging problem. First, the search space is extremely large, with a cardinality on the order
1Code and demo is available at https://github.com/altndrr/vic
(a) VLM-based classification
(b) Vocabulary-free Image Classification
Figure 1: Vision-Language Model (VLM)-based classification (a) assumes a pre-defined set of target categories, i.e. the vocabulary, while our novel task (b) lifts this assumption by directly operating on the unconstrained language-induced semantic space, without a known vocabulary.  $f_{\mathrm{VLM}}^v$  and  $f_{\mathrm{VLM}}^t$  denote the pre-trained vision and text models of a VLM, respectively.
of millions of semantic concepts $^2$ , way larger than any existing image classification benchmark (e.g. ImageNet-21k [10]). Second, it also includes very fine-grained concepts that might be hard to discriminate by the model. Third, as the categories encountered at test time are undefined beforehand, VIC calls for classification methods that do not rely on any vocabulary-aware supervision.
Recent web-scale Vision-Language Databases (VLDs) [51, 54], offer a unique opportunity to address VIC, as they cover a wide set of semantic concepts, ranging from general to highly specific ones. We empirically show that such external databases allow identifying semantic content that is more relevant to the target category than the captioning-enhanced VLMs supporting visual queries (e.g. BLIP-2 [33]). Motivated by this observation, we propose a training-free method for VIC, Category Search from External Databases (CaSED), which jointly leverages the discriminative multimodal representations derived from CLIP [47] and the information provided by recent VLDs (e.g. PMD [54]). CaSED operates in two steps: it first coarsely estimates a set of candidate categories for the test image, then it predicts the final category via multimodal matching. Specifically, we first retrieve the captions from the database that are semantically closer to the input image, from which we extract candidate categories via text parsing and filtering. We then estimate the similarity score between the input image and each candidate category via CLIP, using both visual and textual information, predicting as output the best matching candidate. CaSED exploits the pre-trained CLIP without further training, thus being flexible and computationally efficient.
We experiment on several datasets, considering both coarse- (e.g. Caltech-101 [14], UCF101 [55]) and fine-grained (e.g. FGVC-Aircraft [40], Flowers-102 [43]) classification tasks. To quantitatively assess the performance of methods addressing VIC, we also propose a set of metrics to measure how well the predicted class matches the semantics of the ground-truth label. Across all tasks and metrics, CaSED consistently outperforms all baselines, including VLMs as complex as BLIP-2 [33]. We believe that thanks to its simplicity and effectiveness, our training-free CaSED can serve as a competitive baseline for future works aiming to address the challenging VIC task.
To summarize, this work provides the following contributions:
- We explore the task of Vocabulary-free Image Classification where the goal is to assign a class to an image over an unconstrained set of semantic concepts, overcoming the fundamental assumption of existing VLM-based methods for image classification. We formalize this task and suggest specific evaluation metrics that can be used as a reference for future works.
- We propose CaSED, the first method to address VIC thanks to the adoption of large captioning databases. Notably, CaSED is training-free, not requiring any additional parameter nor finetuning of the network's textual and visual encoders.
- Our large-scale evaluation demonstrates that CaSED consistently outperforms a more complex VLM such as BLIP-2 [33] on VIC, while requiring much fewer parameters.
2We estimate the number of concepts using BabelNet [42], which is close to 4 million for English.
Vision-Language Models. Leveraging large-scale datasets with image-text pairs [51, 50, 54], recent works train models by mapping the two modalities into a shared representation space [28, 18, 11, 47, 26, 35, 15]. A notable example is CLIP [47] which, using modality-specific encoders and a contrastive objective to align their output representations, showed remarkable performance on zero-shot classification. Subsequent works improved CLIP by e.g. connecting the two modalities via cross-modal attention [35], multi-object representation alignment [64], learning from weak-supervision [60] or unaligned data [54].
Another line of works improved vision-language pre-training for complex vision-language tasks, such as image captioning and visual question answering (VQA) [62, 22, 34, 1]. In this context, BLIP [34] exploits web data and generated captions to supervise pre-training of a multimodal architecture, outperforming existing VLMs on both captioning and VQA. The current state-of-the-art method BLIP-2 [33] trains a module connecting the two modalities on top of a frozen visual encoder and a frozen large-language model, enabling instructed zero-shot image-to-text generation.
In this work, we challenge a fundamental assumption of zero-shot classification with VLMs: the set of target classes is known a priori. We propose a new task, VIC, which sidesteps this assumption, performing classification in a language-induced open-ended space of semantic categories. We show that even BLIP-2 struggles in this scenario while external multimodal databases provide valuable priors for inferring the semantic category of an image. As a final note, VIC differs from open-vocabulary recognition (e.g. [63, 17]) since the latter assumes that the list of target classes is known and available to the model during inference.
Retrieval augmented models. In natural language processing, multiple works showed the benefit of retrieving information from external databases, improving the performance of large language models [19, 32, 3]. In computer vision, such a paradigm has been used mostly to deal with the imbalanced distribution of classes. Examples are [38, 39], addressing long-tail recognition by learning to retrieve training samples [38] or image-text pairs from an external database [39]. Similarly, [58] retrieves images from a given dataset to learn fine-grained visual representations. More recently, retrieval-augmentation has been extended to various types of sources for visual question answering [23], as well as to condition the generative process in diffusion models [2], or image captioning [48]. Our work is close in spirit to [39], as we exploit an external database. However, [39] assumes a pre-defined set of classes (and data) available for training a retrieval module, something we cannot have for the extremely large semantic space of VIC. In CaSED, retrieval is leveraged to first create a set of candidate classes, and to then perform the final class prediction. Moreover, we assume the database to contain only captions, and not necessarily paired image-text data, thus being less memory-demanding.
Finally, the performance of retrieval models is largely affected by the employed database. In the context of VLMs, researchers collected multiple open datasets to study vision-language pretraining. Two notable examples are LAION-5B [51], collected by filtering Common Crawl [8] via CLIP, and the Public Multimodal Datasets (PMD) [54], collecting image-text pairs from different public datasets, such as Conceptual Captions [52, 5], YFCC100M [57], Wikipedia Image Text [56], and Redcaps [12]. In our experiments, we use a subset of PMD as database and investigate how classification performance varies based on the size and quality of the database.
3 Vocabulary-free Image Classification
Preliminaries. Given the image space  $\mathcal{X}$  and a set of class labels  $C$ , a classification model  $f: \mathcal{X} \to C$  is a function mapping an image  $\boldsymbol{x} \in \mathcal{X}$  to its corresponding class  $\boldsymbol{c} \in C$ . While  $C$  may be represented by indices that refer to semantic classes, here we focus on cases where  $C$  consists of a set of concept names that correspond to real entities. Note that  $C$  is a finite set whose elements belong to the semantic space  $S$ , i.e.  $C \subset S$ . In standard image classification,  $C$  is given a priori, and  $f$  is learned on a dataset of image-label pairs.
Recently, the introduction of contrastive-based VLMs [47, 25], which learn a cross-modal aligned feature space, revised this paradigm by defining a function  $f_{\mathrm{VLM}}$  that infers the similarity between an
Figure 2: Results of our preliminary study, showing the top-1 accuracy when matching semantic descriptions to ground-truth class names in ten different datasets. We compare BLIP-2 (VQA) and BLIP-2 (Captioning) with Closest Caption and Captions Centroid, i.e. the average representation of the retrieved captions. We additionally highlight the Upper bound for zero-shot CLIP. Representing the large semantic space as VLDs and retrieving captions from it produces semantically more similar outputs to ground-truth labels w.r.t. querying outputs from VQA-enabled VLMs, while requiring 10 times fewer parameters compared to the latter.
image and a textual description  $t \in \mathcal{T}$ , i.e.  $f_{\mathrm{VLM}}: \mathcal{X} \times \mathcal{T} \to \mathbb{R}$ , with  $\mathcal{T}$  being the language space. Given this function, classification can be performed with:
$$
f (\boldsymbol {x}) = \underset {\boldsymbol {c} \in C} {\arg \max } f _ {\mathrm {V L M}} (\boldsymbol {x}, \phi (\boldsymbol {c})) \tag {1}
$$
where  $\phi(c)$  is a string concatenation operation, combining a fixed text template, i.e. a prompt, with a class name. This definition allows for zero-shot transfer, i.e. performing arbitrary classification tasks by re-defining the set  $C$  at test time, without re-training the model. However, such zero-shot transfer setup still assumes that the set  $C$  is provided. In this work, Vocabulary-free Image Classification (VIC) is formulated to surpass this assumption.
Task definition. VIC aims to assign a class  $c$  to an image  $x$  without prior knowledge on  $C$ , thus operating on the semantic class space  $S$  that contains all the possible concepts. Formally, we want to produce a function  $f$  mapping an image to a semantic label in  $S$ , i.e.  $f: \mathcal{X} \to S$ . Our task definition implies that at test time, the function  $f$  has only access to an input image  $x$  and a large source of semantic concepts that approximates  $S$ . VIC is a challenging classification task by definition due to the extremely large cardinality of the semantic classes in  $S$ . As an example, ImageNet-21k [10], one of the largest classification benchmarks, is 200 times smaller than the semantic classes in BabelNet [42]. This large search space poses a prime challenge for distinguishing fine-grained concepts across multiple domains as well as ones that naturally follow a long-tailed distribution.
Semantic space representation. As the main challenge of VIC, how to represent the large semantic space plays a fundamental role in the method design. We can either model the multimodal semantic space directly with a VLM equipped with an autoregressive language decoder [34] or via image-text retrieval from VLDs. Consequently, we can approach VIC either via VQA-enabled VLMs by querying for the candidate class given the input image, or by retrieving and processing data from an external VLD to obtain the candidate class.
To investigate the two potential strategies, we perform a preliminary experimental analysis to understand how well the output of a method semantically captures the image category, or in other words, to assess the alignment of class boundaries in the visual and textual representations. Specifically, we compare the semantic accuracy of querying VQA VLMs and of retrieving from VLDs w.r.t. the ground-truth class labels. We consider the output of a method as correct if its closest textual embedding among the target classes of the dataset corresponds to the ground-truth class of the test sample<sup>3</sup>. We exploit the text encoder of CLIP (ViT-L) [47] to obtain textual embeddings.
Note that this metric is not the standard accuracy in image classification as we use distances in the embedding space to ground predictions from the unconstrained semantic space to the set of classes in a specific dataset.
Regarding experimented methods, we select BLIP-2 [33] to represent VQA-enabled VLMs for its state-of-the-art performance in VQA benchmarks, while we use a subset of PMD [54] as the VLD. In particular, we compare the following methods: i) BLIP-2 VQA, which directly queries BLIP-2 for the image category; ii) BLIP-2 Captioning, which queries BLIP-2 for the image caption; iii) Closest Caption, which is the closest caption to the image, as retrieved from the database; iv) Caption Centroid, which averages the textual embeddings of the 10 most similar captions to the input image. As we use CLIP embeddings, if visual and textual representations perfectly align, the performance would be the same as zero-shot CLIP with given target classes. We thus report zero-shot CLIP to serve as the upper bound for retrieval accuracy.
We experiment on a variety of test datasets for both coarse- and fine-grained classification (see details in Sec. 5), and report the results in Fig. 2. The average textual embedding of the retrieved captions (i.e. Caption Centroid) achieves the best semantic accuracy for 9 datasets out of 10, consistently surpassing methods based on BLIP-2. On average, the accuracy achieved by Caption Centroid is  $60.47\%$ , which is  $+17.36\%$  higher than the one achieved by BLIP-2 Captioning  $(43.11\%)$ . Moreover, Captions Centroid achieves results much closer to the CLIP upper bound  $(67.17\%)$  than the other approaches. Notably, such VLD-based retrieval is also computationally more efficient, faster ( $\sim 4$  second for a batch size of 64 on a single A6000 GPU), and requires fewer parameters (approximately 10 times less) than BLIP-2 (see Tab. 7 in Appendix B).
The results of this preliminary study clearly suggest that representing the large semantic space with VLDs can produce (via retrieval) semantically more relevant content to the input image, in comparison to querying VQA-enabled VLMs, while being computationally efficient. Based on this conclusion, we develop an approach, Category Search from External Databases (CaSED), that searches for the semantic class from the large semantic space represented in the captions of VLDs.
4 CaSED: Category Search from External Databases
Our proposed method CaSED finds the best matching category within the unconstrained semantic space by multimodal data from large VLDs. Fig. 3 provides an overview of our proposed method. We first retrieve the semantically most similar captions from a database, from which we extract a set of candidate categories by applying text parsing and filtering techniques. We further score the candidates using the multimodal aligned representation of the large pre-trained VLM, i.e. CLIP [47], to obtain the best-matching category. We describe in detail each process in the following.
4.1 Generating candidate categories
We first restrict the extremely large classification space to a few most probable candidate classes. Let  $f_{\mathrm{VLM}}$  be the pre-trained VLM and  $D$  be the external database of image captions. Given an input  $\mathbf{x}$ , we retrieve the set  $D_{\mathbf{x}} \subset D$  of  $K$  closest captions to the input image via
$$
D _ {\boldsymbol {x}} = \sup  _ {\boldsymbol {d} \in D} \operatorname {k} f _ {\mathrm {V L M}} (\boldsymbol {x}, \boldsymbol {d}) = \sup  _ {\boldsymbol {d} \in D} \left\langle f _ {\mathrm {V L M}} ^ {v} (\boldsymbol {x}), f _ {\mathrm {V L M}} ^ {t} (\boldsymbol {d}) \right\rangle , \tag {2}
$$
where  $f_{\mathrm{VLM}}^v:\mathcal{X}\to \mathcal{Z}$  is the visual encoder of the VLM,  $f_{\mathrm{VLM}}^{t}:\mathcal{T}\rightarrow \mathcal{Z}$  is the textual encoder, and  $\mathcal{Z}$  is their shared embedding space. The operation  $\langle \cdot ,\cdot \rangle$  indicates the computation of the cosine similarity. Note that our approach is agnostic to the particular form of  $D$ , and that it can accommodate a flexible database size by including captions from additional resources.
From the set  $D_{\mathbf{x}}$ , we then extract a finite set of candidate classes  $C_{\mathbf{x}}$  by performing simple text parsing and filtering techniques, e.g. stop-words removal, POS tagging. Details on the filtering procedure can be found in Appendix A.
4.2 Multimodal candidate scoring
Among the small set  $C_x$ , we score each candidate by accounting for both visual and textual semantic similarities using the VLM encoders, in order to select the best-matching class for the input image.
Image-to-text score. As the image is the main driver for the semantic class we aim to recognize, we use the visual information to score the candidate categories. We denote  $s_c^v$  as the visual score of each
Figure 3: Overview of CaSED. Given an input image, CaSED retrieves the most relevant captions from an external database filtering them to extract candidate categories. We classify image-to-text and text-to-text, using the retrieved captions centroid as the textual counterpart of the input image.
candidate category  $c$  and compute it as the similarity between the visual representation of the input image and the textual representation of the candidate name:
$$
s _ {\boldsymbol {c}} ^ {v} = \left\langle f _ {\mathrm {V L M}} ^ {v} (\boldsymbol {x}), f _ {\mathrm {V L M}} ^ {t} (\boldsymbol {c}) \right\rangle . \tag {3}
$$
The higher value of  $s_c^v$  indicates a closer alignment between the target image and the candidate class.
Text-to-text score. While the image-to-text score  $s_c^v$  is effective, there exists a well-known modality gap in the space  $\mathcal{Z}$ , harming the performance of zero-shot models [36]. As suggested by Fig. 2 in Sec. 3, the semantic relevance of the retrieved captions, and their centroid in particular, is high w.r.t. the underlying ground-truth label. We are therefore motivated to exploit such attributes and introduce a unimodal text-to-text scoring to mitigate the modality gap of cross-modal scoring.
Formally, we define the centroid  $\bar{d}_{\bar{x}}$  of the retrieved captions as:
$$
\bar {d} _ {\boldsymbol {x}} = \frac {1}{K} \sum_ {\boldsymbol {d} \in D _ {\boldsymbol {x}}} f _ {\mathrm {V L M}} ^ {t} (\boldsymbol {d}), \tag {4}
$$
where  $K$  is the number of retrieved captions. We then define the text-based matching score  $s_c^t$  as the similarity between the centroid and the candidate category:
$$
s _ {\boldsymbol {c}} ^ {t} = \langle \bar {d} _ {\boldsymbol {x}}, f _ {\mathrm {V L M}} ^ {t} (\boldsymbol {c}) \rangle . \tag {5}
$$
A higher value of  $s_c^t$  means a higher alignment between the caption centroid and the candidate embedding. Note that the semantic relevance of the caption centroid is an inherent property of the text encoder of VLMs (i.e. CLIP). Since CLIP is trained with an image-text alignment loss, its text encoder focuses on the visual elements of the caption, discarding parts that are either non-visual or non-relevant to the visual content. This improves the model's robustness to noisy candidates.
Final predicted candidate. To predict the final candidate, we merge the two scores, obtaining the final score  $s_c$  for each candidate  $c$  as:
$$
s _ {\boldsymbol {c}} = \alpha \sigma \left(s _ {\boldsymbol {c}} ^ {v}\right) + (1 - \alpha) \sigma \left(s _ {\boldsymbol {c}} ^ {t}\right) \tag {6}
$$
where  $\sigma(\cdot)$  is the softmax operation on the two scores of each candidate class, and  $\alpha$  is a hyperparameter regulating the contribution of the two modalities. Finally we obtain the output category as  $f(x) = \arg \max_{c \in C_x} s_c$ . Notably, CaSED respects the VIC task definition, performing classification without known class priors, while being training-free with the use of a pre-trained and frozen VLM. This makes the approach flexible and applicable to a variety of architectures and databases.
We evaluate CaSED in comparison to other VLM-based methods on the novel task Vocabulary-free Image Classification with extensive benchmark datasets covering both coarse-grained and fine-grained classification. We first describe the experimental protocol in terms of the datasets, the proposed evaluation metrics, and the baselines applicable to this task (Sec. 5.1). We then discuss the quantitative results regarding the comparison between our method and baselines (Sec. 5.2). Finally, we present a thorough ablation to justify the design choices of CaSED (Sec. 5.3). In addition, we provide a cost comparison between the baseline methods and CaSED in Appendix B. We also offer further ablation of our method in Appendix C regarding architecture and database, and showcase qualitative results of predicted categories from multiple datasets in Appendix D.
5.1 Experimental protocol