SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets. We follow existing works [53, 66] and use ten datasets that feature both coarse-grained and fine-grained classification in different domains: Caltech-101 (C101) [14], DTD [7], EuroSAT (ESAT) [21], FGVC-Aircraft (Airc.) [40], Flowers-102 (Flwr) [43], Food-101 (Food) [4], Oxford Pets (Pets), Stanford Cars (Cars) [29], SUN397 (SUN) [61], and UCF101 (UCF) [55]. Additionally, we used ImageNet [10] for hyperparameters tuning.
Evaluation metrics. Due to the unconstrained nature of the semantic space, evaluating the effectiveness of methods for VIC is not trivial. In this paper, we propose to use two main criteria, namely semantic relevance, i.e. the similarity of the predicted class w.r.t. the ground-truth label, and image grouping, i.e. the quality of the predicted classes for organizing images into clusters. For semantic, we consider i) Semantic Similarity, i.e. the similarity of predicted/ground-truth labels in a semantic space, and ii) Semantic IoU, i.e. the overlap of words between the prediction and the true label. More formally, given an input  $\mathbf{x}$  with ground-truth label  $\mathbf{y}$  and prediction  $\hat{\mathbf{c}} = f(\mathbf{x})$ , we compute the Semantic Similarity as  $\langle g(\hat{\mathbf{c}}), g(\mathbf{y}) \rangle$ , where  $g: \mathcal{T} \to \mathcal{V}$  is a function mapping text to an embedding space  $\mathcal{V}$ . Since we want to model free-form text, we use Sentence-BERT [49] as  $g$ . For Semantic IoU, given a predicted label  $\mathbf{c}$ , and assuming  $\mathbf{c}$  being a set of words, we compute the Semantic IoU as  $|c \cap \mathbf{y}| / |c \cup \mathbf{y}|$ , where  $\mathbf{y}$  is the set of words in the ground-truth label. To assess grouping, we measure the classic Cluster Accuracy by first clustering images according to their predicted label, and then assigning each cluster to a ground-truth label with Hungarian matching. Sometimes, this mapping is resolved with a many-to-one match, where a predicted cluster is assigned to the most present ground-truth label. This evaluation draws inspiration from the protocols used for deep visual clustering [59, 24, 20].
Baselines. We consider three main groups of baselines for our comparisons. The most straightforward baselines consist of using CLIP with large vocabularies, such as WordNet [41] (117k names) or the English Words (234k names [16]). As an upper bound, we also consider CLIP with the perfect vocabulary, i.e. the ground-truth names of the target dataset (CLIP upper bound). Due to lack of space, we only report results for CLIP with ViT-L [13], while results with other architectures are reported in Appendix C. The second group of baselines consists of captioning methods, as captions can well describe the semantic content of images. We consider two options: captions retrieved from a database and captions generated by a pre-trained image captioning model. For the former we exploit a large collection of textual descriptions, retrieving the most fitting caption for the given image. For the latter, we exploit BLIP-2 [33] — a VLM with remarkable performance on a variety of tasks, including image captioning — to provide a description for the image. The last group of baselines consists of using a VQA model to directly predict the class name associated with the image. Again, we consider BLIP-2 [33], since being highly effective also in VQA. We evaluate BLIP-2 over both ViT-L and ViT-g throughout the experiments<sup>4</sup>.
Implementation details. Our experiments were conducted using NVIDIA A6000 GPUs with mixed-bit precision. As database, we use a subset of PMD [54], containing five of its largest datasets: Conceptual Captions (CC3M) [52], Conceptual Captions 12M (CC12M) [5], Wikipedia Image Text (WIT) [56], Redcaps [12], and a subset of [57] used for PMD (YFCC100M*). Further details on the
4Following the BLIP-2 [33] demo, for captioning, we used the prompt "Question: what's in the image? Answer:". For VQA, we used "Question: what's the name of the object in the image? Answer: a".
Method	Cluster Accuracy (%)↑
C101	DTD	ESAT	Airc.	Flwr	Food	Pets	SUN	Cars	UCF	Avg.
CLIP	WordNet	34.0	20.1	16.7	16.7	58.3	40.9	52.0	29.4	18.6	39.5	32.6
English Words	29.1	19.6	22.1	15.9	64.0	30.9	44.4	24.2	19.3	34.5	30.4
Caption	Closest Caption	12.8	8.9	16.7	13.3	28.5	13.1	15.0	8.6	20.0	17.8	15.5
BLIP-2 (ViT-L)	26.5	11.7	23.3	5.4	23.6	12.4	11.6	19.5	14.8	25.7	17.4
BLIP-2 (ViT-g)	37.4	13.0	25.2	10.0	29.5	19.9	15.5	21.5	27.9	32.7	23.3
VQA	BLIP-2 (ViT-L)	60.4	20.4	21.4	8.1	36.7	21.3	14.0	32.6	28.8	44.3	28.8
BLIP-2 (ViT-g)	62.2	23.8	22.0	15.9	57.8	33.4	23.4	36.4	57.2	55.4	38.7
CaSED	51.5	29.1	23.8	22.8	68.7	58.8	60.4	37.4	31.3	47.7	43.1
CLIP upper bound	87.6	52.9	47.4	31.8	78.0	89.9	88.0	65.3	76.5	72.5	69.0
Table 1: Cluster Accuracy on the ten datasets. Green is our method, gray shows the upper bound.
Method	Semantic Similarity (x100) ↑
C101	DTD	ESAT	Airc.	Flwr	Food	Pets	SUN	Cars	UCF	Avg.
CLIP	WordNet	48.6	32.7	24.4	18.9	55.9	49.6	53.7	44.9	28.8	44.2	40.2
English Words	39.3	31.6	19.1	18.6	43.4	38.0	44.2	36.0	19.9	34.7	32.5
Caption	Closest Caption	42.1	23.9	23.4	29.2	40.0	46.9	40.2	39.8	49.2	40.3	37.5
BLIP-2 (ViT-L)	57.8	31.4	39.9	24.4	36.1	44.6	29.0	45.3	46.4	38.0	39.3
BLIP-2 (ViT-g)	63.0	33.1	36.2	24.3	45.2	51.6	31.6	48.3	61.0	44.6	43.9
VQA	BLIP-2 (ViT-L)	70.5	34.9	29.7	29.1	48.8	42.0	40.0	50.6	52.4	48.6	44.7
BLIP-2 (ViT-g)	73.5	36.5	31.4	30.8	59.9	52.1	43.9	53.3	65.1	55.1	50.1
CaSED	65.7	40.0	32.0	30.3	55.5	64.5	62.5	52.5	47.4	54.1	50.4
CLIP upper bound	90.8	69.8	67.7	66.7	83.4	93.7	91.8	80.5	92.3	83.3	82.0
Table 2: Semantic Similarity on the ten datasets. Values are multiplied by x100 for readability. Green highlights our method and gray indicates the upper bound.
selection are left for Appendix C. We speed up the retrieval process by embedding the database via the text encoder  $f_{\mathrm{VLM}}^t$  and using fast indexing technique, i.e. FAISS [27]. We tuned the  $\alpha$  hyperparameter of Eq. (6) and the number of retrieved captions  $K$  of our method on the ImageNet dataset, finding that  $\alpha = 0.7$  and  $K = 10$  led to the best results. We use these values across all experiments.
5.2 Quantitative results
The results of CaSED and the baselines are presented in Table 1, Table 2, and Table 3 for Cluster Accuracy  $(\%)$ , Semantic Similarity, and Semantic IoU  $(\%)$ , respectively. CaSED consistently outperforms all baselines in all metrics on average and in most of the datasets. Notably, CaSED surpasses BLIP-2 (VQA) over ViT-g by  $+4.4\%$  in Cluster Accuracy and  $+1.7\%$  on Semantic IoU, while using much fewer parameters (i.e. 102M vs 4.1B). The gap is even larger over the same visual backbone, with CaSED outperforming BLIP-2 on ViT-L (VQA) by  $+14.3\%$  on Cluster Accuracy,  $+5.7$  in Semantic Similarity, and  $+6.8\%$  on Semantic IoU. These results highlight the effectiveness of CaSED, achieving the best performance both in terms of semantic relevance and image grouping.
An interesting observation from the tables is that simply applying CLIP on top of a pre-defined, large vocabulary, is not effective in VIC. This is due to the challenge of classifying over a huge search space, where class boundaries are hard to model. This is confirmed by the results of CLIP with English Words having a larger search space but performing consistently worse than CLIP with WordNet across all metrics (e.g. -7.7 on Semantic Similarity, -3.2% on Semantic IoU).
A final observation relates to the captioning models. Despite their ability to capture the image semantic even in challenging settings (e.g. 39.3 Semantic Similarity of BLIP-2 ViT-L on ESAT), captions exhibit high variability across images of the same category. This causes the worst performance on Clustering and Semantic IoU across all approaches (e.g. almost  $-20\%$  less than CaSED on average in terms of Cluster Accuracy), thus demonstrating that while captions can effectively describe the content of an image, they do not necessarily imply better categorization for VIC.
Method	Semantic IoU (%) ↑
C101	DTD	ESAT	Airc.	Flwr	Food	Pets	SUN	Cars	UCF	Avg.
CLIP	WordNet	15.0	3.0	1.3	0.5	31.3	7.8	14.7	9.0	4.8	3.8	9.1
English Words	8.0	2.0	0.0	1.1	16.4	2.0	17.2	8.1	2.7	1.8	5.9
Caption	Closest Caption	4.5	0.8	1.3	1.9	5.9	3.1	3.0	2.3	11.4	1.0	3.5
BLIP-2 (ViT-L)	13.4	1.4	4.8	0.0	7.5	4.7	1.7	4.7	11.6	1.1	5.1
BLIP-2 (ViT-g)	16.8	1.8	4.1	0.1	13.9	7.9	2.9	5.7	24.7	1.9	8.0
VQA	BLIP-2 (ViT-L)	36.1	1.8	7.0	0.1	21.5	3.7	5.7	11.5	18.9	2.5	10.9
BLIP-2 (ViT-g)	41.5	2.4	7.5	2.0	38.0	8.6	10.2	13.8	33.2	2.8	16.0
CaSED	35.4	5.1	2.3	4.8	33.1	19.4	35.1	17.2	16.2	8.4	17.7
CLIP upper bound	86.0	52.2	51.5	28.6	75.7	89.9	88.0	66.6	84.5	71.3	69.4
Table 3: Semantic IoU on the ten datasets. Green is our method, gray shows the upper bound.
Candidates Generation	Scoring Vis. Lang.	CA	S-Sim.	S-IoU
Generative [33]	✓	23.3	47.1	11.9
Retrieval	✓	41.7	49.3	17.0
✓	42.7	50.3	17.0
✓	✓	43.1	50.4	17.7
(a) Ablation on candidate generation and scoring.
Database	Size	CA	S-Sim.	S-IoU
CC3M	2.8M	34.2	47.9	13.1
WIT	4.8M	34.6	42.9	12.1
Redcaps	7.9M	42.0	49.5	17.2
CC12M	10.3M	44.0	51.3	18.3
YFCC100M*	29.9M	40.7	48.8	17.1
All	54.8M	43.1	50.4	17.7
(b) Ablation on the database.
Table 4: Ablation studies. Metrics are averaged across the ten datasets. Green highlights our setting. Bold represents best, underline indicates second best.
In this section, we present the ablation study associated with different components of our approach. We first analyze the impact of our retrieval-based candidate selection and multimodal scoring. We then show the results of CaSED for different databases, and how the number of retrieved captions impacts the performance.
Candidates generation. We consider two options to generate the set of candidate classes. The first uses BLIP-2 (ViT-g), asking for multiple candidate labels for the input image. The second is our caption retrieval and filtering strategy. Table 4a shows the results where, for fairness, we score both sets with the same VLM (i.e. CLIP ViT-L). Our approach consistently outperforms BLIP-2 across all metrics (e.g. 41.7 vs 23.3 for Cluster Accuracy). This confirms the preliminary results of our study in Fig. 2, with retrieved captions providing better semantic priors than directly using a powerful VLM.
Multimodal scoring. The second ablation studies the effect of our proposed multimodal scoring vs its unimodal counterparts. As Table 4a shows, multimodal candidate scoring provides the best results across all metrics, with clear improvements over the visual modality alone (i.e.  $+1.4\%$  of Cluster Accuracy). Notably, scoring via language only partially fills the gap between visual and multimodal scoring (e.g.  $+1\%$  on Cluster Accuracy and  $+1$  on Semantic Similarity), confirming that caption centroids contain discriminative semantic information.
Retrieval database. We analyze the impact of retrieving captions from different databases, using five public ones, i.e. CC3M, WIT, Redcaps, CC12M, and a subset of YFCC100M. The databases have different sizes (e.g. from 2.8M captions of CC3M to 29.9M captions of the YFCC100M subset), and different levels of noise. As shown in Table 4b, the results tend to improve as the size of the database increases (e.g.  $+8.9\%$  on Cluster Accuracy over CC3M). However, the quality of the captions influences the performance, with CC12M and Redcaps alone achieving either comparable or slightly better results than the full database. These results suggest that while performance improves with the size of the database, the quality of the captions has a higher impact than the mere quantity.
Number of captions. Finally, we check how performance varies w.r.t. the number of retrieved captions. As shown in Fig. 4, all metrics consistently improve as the number of captions increases from 1 to 10. After that, performance tends to saturate, with a slight decrease in terms of Semantic Similarity for  $K = 20$ . These results suggest that while a minimum number of captions is needed to fully capture the semantics of the image, the possible interference of less-related (or even noisy) captions may impact the final performance. Future research may further improve the performance on VIC by focusing on how to deal with noisy retrieval results.
Figure 4: Ablation on the number of retrieved captions. We report Cluster accuracy  $(\%)$ , Semantic similarity, and Semantic IoU  $(\%)$ .
6 Discussion and conclusions
In this work, we proposed a new task, VIC, which operates on an unconstrained semantic space, without assuming a pre-defined set of classes, a brittle assumption of VLM-based classification. We experimentally verified that multimodal databases provide good semantic priors to restrict the large search space, and developed CaSED, an efficient training-free approach that retrieves the closest captions to the input image to extract candidate categories and scores them in a multimodal fashion. On different benchmarks, CaSED consistently achieved better results than more complex VLMs.
Limitations and future works. The performance of CaSED strongly depends on the choice of the retrieval database, with potential issues in retrieving concepts that are not well represented in the latter. Moreover, if the domain of application contains fine-grained concepts, a generic database might not be suitable. Without any prior information on the test labels, it is hard to predict the performance of a database a priori. On the other hand, CaSED can flexibly mitigate this issue by incrementally including new concepts in the database (even domain-specific ones) from textual corpora, without retraining. Future research may explore strategies to automatically select/extend a database based on test samples and/or pre-computing the best database to use in the absence of test label information.
Additionally, as CaSED lacks control over the classes contained in the database, its predictions might reflect potential biases. Improvements in mitigating biases and data quality control would reduce this issue. Another limitation is that CaSED does not keep track of its output history. This may lead to inconsistent predictions, i.e. assigning slightly different labels to images of the same semantic concept (e.g. cassowary vs Casuarius). Equipping CaSED with a memory storing the predicted labels may address this issue. Finally, CaSED does not deal with different class granularities: e.g. an image of a cassowary can be as well predicted as a bird. Future works may disambiguate such cases by explicitly integrating the user needs within VIC models.
Broader impact. CaSED addresses VIC in a scalable and training-free manner. We believe that our new problem formulation, metrics, and the effectiveness of CaSED will encourage future research in this topic, overcoming the limiting assumption of VLM-based classification and allowing the power of VLMs to benefit dynamic and unconstrained scenarios.
This work was supported by the MUR PNRR project FAIR - Future AI Research (PE00000013) and ICSC National Research Centre for High Performance Computing, Big Data and Quantum Computing (CN00000013), funded by the NextGenerationEU. E.R. is partially supported by the PRECRISIS, funded by the EU Internal Security Fund (ISFP-2022-TFI-AG-PROTECT-02-101100539), the EU project SPRING (No. 871245), and by the PRIN project LEGO-AI (Prot. 2020TA3K9N). The work was carried out in the Vision and Learning joint laboratory of FBK and UNITN. We also thank the Deep Learning Lab of the ProM Facility for the GPU time.
[55] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv, 2012.
[56] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Research and Development in Information Retrieval, 2021.
[57] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 2016.
[58] Hugo Touvron, Alexandre Sablayrolles, Matthijs Douze, Matthieu Cord, and Hervé Jégou. Grafit: Learning fine-grained image representations with coarse labels. In ICCV, 2021.
[59] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In ECCV, 2020.
[60] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In ICLR, 2022.
[61] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
[62] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv, 2022.
[63] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR, 2021.
[64] Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. In ICML, 2022.
[65] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
[66] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 2022.
A Candidates filtering
With the closest captions retrieved from the external database given the input image (please refer to Sec. 4 of the main manuscript for further details), we post-process them to filter out a set of candidate category names. We first create a set of all words that are contained in the captions. Then we apply sequentially three different groups of operations to the set to (i) remove noisy candidates, (ii) standardize their format, and (iii) filter them.
With the first group of operations, we remove all the irrelevant textual contents, such as tokens (i.e. "<PERSON>"), URLs, or file extensions. Note that, for the file extensions, we remove the extension but retain the file name as it might contain candidate class names. We also remove all the words that are shorter than three characters and split compound words by underscores or dashes. Finally, we remove all those terms containing symbols or numbers and meta words that are irrelevant to the classification task, such as "image", "photo", or "thumbnail". As shown in Table 5, when compared to having no operation for candidate filtering (first row), this set of operations removes inappropriate content and increases the accuracy of clusters by  $+6.4\%$  and improves the semantic IoU by  $+0.3$ . However, we can observe a drop in semantic similarity by  $-1.5$ . This might be due to the removal of unnatural words that could still describe well the content of the image, i.e. underline- or dash-separated words, or URLs since they are longer w.r.t. natural words.
The second group of operations standardize the candidate names by aligning words that refer to the same semantic class to a standard format, reducing class redundancy. For example, "cassowary" and "Cassowary" will be considered as a single class instead of two. To this end, we perform two operations—lowercase conversion and singular form conversion. With such standardizing conversions, we observe a sizeable boost in terms of performances when compared to the results obtained by applying only removal-related operations. As shown in Table 5, we achieve higher results across all three metrics, leading to a relative improvement of  $+7.2\%$ ,  $+0.7$ , and  $+1.2$  in terms of cluster accuracy, semantic similarity, and semantic IoU, respectively.
The last group of operations considers two forms of filtering, where the first aims to filter out entire categories of words via Part-Of-Speech (POS) tag and the second aims to filter out rare and noisy contents based on the word occurrences. We select these two operations since common dataset class names do not contain terms that carry no semantics e.g. articles and pronouns, and since [9] showed that CLIP performs better when exposed to a smaller amount of unique tokens. The POS tagging $^5$  categorizes words into groups, such as adjectives, articles, nouns, or verbs, enabling us to filter all the terms that are not semantically meaningful for a classification task. Regarding the occurrence filtering, we first count how often a word appears in the retrieved captions and then we remove words that appear only once to make the candidate list less noisy. We can see from Table 5 that the inclusion of this final set of operations scores the best among all three metrics when compared to the results obtained when only the previous two groups of operations are applied.
Number of captions vs number of selected candidates.
To complement the previous analysis, in Table 6, we report the number of unique candidates extracted by the candidate filtering procedure, averaged over the ten datasets, and with an increasing number of retrieved captions, i.e., 1, 2, 5, 10, and 20. In the table, we show both the number of candidates extracted and the number of selected words. As the number of retrieved captions increases, the unique number of candidate words also increases, i.e., from 3849 with 1 caption to 28781 with 20. However, the number of selected words stabilizes around 800 as soon as we retrieve more than 1 caption. Having more captions reduces the noises in the selected words, something that might be present when relying on a single caption.
B Computational cost
We analyze the computational efficiency of CaSED versus BLIP-2 performing VQA and captioning, and report their respective number of parameters and inference time in Table 7. Notably, the methods
We use the NLP library flair (https://github.com/flairNLP/flair).
Operations	CA	S-Sim.	S-IoU
Remove	Standardize	Filter
27.7	48.8	15.0
✓	34.1	47.3	15.3
✓	✓	41.3	48.0	16.5
✓	✓	✓	43.1	50.4	17.7
Table 5: Ablation on the candidate filtering operations. Metrics are averaged across the ten datasets.
Num. captions	CA	S-Sim.	S-IoU	Candidates	Selected
1	30.9	43.2	12.4	3849	1047
2	35.0	46.4	14.6	6867	854
5	42.3	50.6	17.5	14548	775
10	43.1	50.4	17.7	22475	794
20	42.9	49.3	17.1	28781	802
Table 6: Extended ablation on the number retrieved captions. Green represents our selected configuration. We expand the results of Fig. 4 in the main manuscript to show the number of unique words extracted from captions (i.e., "candidates") and the number of words selected by CaSED for classification (i.e., "selected"). Results are averaged across the ten datasets.
using external databases are consistently faster than BLIP-2. For instance, CaSED achieves a speed-up of  $1.5\mathrm{x}$  with respect to both the largest captioning model and the largest VQA model, while also achieving better performance. Overall, the fastest method is Closest Caption, which exploits the external database to retrieve a single caption and does not consider any candidate extraction pipeline. Conversely, our method retrieves the ten most similar captions and post-processes them to extract class name, resulting in an increase in inference time of approximately 3 times. Compared with the CLIP upper bound, our method considers multiple additional steps, each adding extra inference time. First, our method retrieves candidates from the external database to then extract the class names. Second, we have to forward the class names through the text encoder for each sample, while CLIP can forward them once and cache the features for later reuse.
When using an external database, note that an increase in database size implies a minimal variation in retrieval time. This is demonstrated by the computational cost required by retrieving from the large LAION-400M [51] database. As the results show, inference time is comparable between CaSED and CaSED (LAION-400M) despite the latter being approximately 8 times larger.
C Additional ablation study
In this section, we show the performance of our model with different backbones, and other commonly used databases for retrieval.