SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We consider 10 existing image retrieval methods as baselines and investigate their image embedding qualities. According to how the embedding is built, these methods can be categorized into 3 groups.
Bottom-up This strategy builds a global image embedding based on local image features. The related methods include: (1) BoW [2]. This method extracts RootSIFT [2] local features from the given image, which are then quantized using a codebook and finally assembled into a sparse feature vector, i.e., image embedding. Since BoW only relies on handcrafted low-level image features, the produced embedding tends to have better generalization ability than learning-based ones that fit to certain domains. (2) FIRe [47], which extracts mid-level image features and then aggregates them in a manner similar to BoW. However, different from BoW, FIRe is deep learning-based and the feature extraction needs to be learned with certain training data, e.g., SfM-120k [39].
Top-down Contrary to the bottom-up approach, this strategy learns to extract local image features through image-level supervision on global image embeddings. The local features typically correspond to the convolutional feature maps and are used for feature matching or reranking. In contrast, the global image embeddings are used in the first stage of a retrieval system to efficiently select the most similar images. In our experiment, we consider one representative approach, DELG [4], which jointly extracts deep local features and global image embeddings.
Top-only This strategy performs image retrieval with learned global image embeddings directly, without the need of extracting and using local image features. The global image embeddings are typically produced from a deep model that is trained on a large dataset, in a supervised or self- / weakly-supervised manner. In the experiment, we consider the following state-of-the-art methods: (1) CLIP [40]; (2) SLIP [28]; (3) BLIP [23]; (4) BLIP2 [22]; (5) DINO [5]; (6) DINoV2 [32]; (7) DiHT [36]. Note that apart from the design differences, another major distinction among these methods lies in their training data; see Table 3 for more details. In Table 6 we also show the specific neural network model used in each method.
It is worth mentioning that for some top-only methods, their training data may overlap with our FORB benchmark. In particular, we find a few images from FORB are also included in LAION-5B [42], and therefore training data based on the subset of LAION-5B (e.g., LAION-438M [42] and 129M [23]) may also share duplicate images with FORB. In addition, since the training set of CLIP are collected from web, it may overlap with FORB as well. This test set overlap issue has been discussed in previous works [40, 42] and is considered to have little impact on the validity of performance evaluations. In the supplementary material we perform extra experiments on a deduplicated version of FORB and observe the evaluation results closely resemble those from the original FORB (see Section A.4)
Table 4: Comparison of different image retrieval methods on our FORB benchmark. Bolded numbers indicate the best results.  $\dagger$  means the model training data may overlap with FORB and the retrieval accuracy can be interpreted as an "upper bound" performance.
mAP@5 (%)	t-mAP@5 (%)
Method	Overall	Easy	Medium	Hard	Overall	Easy	Medium	Hard
BoW [2]	78.44	90.38	79.78	52.65	62.49	78.29	63.61	35.00
BoW (+ rerank) [2]	80.38	92.69	81.77	53.70	67.83	81.95	69.04	41.03
FIRE [47]	88.08	98.48	90.14	56.58	77.50	90.38	79.41	44.97
DELG [4]	48.81	79.45	48.11	24.48	34.92	65.44	33.79	15.04
DELG (+ rerank) [4]	58.74	87.96	58.43	31.91	39.47	70.64	38.45	17.74
CLIP†[40]	89.36	98.23	90.00	73.84	67.23	87.10	67.48	44.27
SLIP [28]	39.01	64.45	38.58	17.22	24.43	50.27	23.42	8.07
BLIP†[23]	74.11	94.67	74.65	47.53	49.98	81.31	49.58	21.89
BLIP2†[22]	81.73	94.28	82.72	58.85	57.11	81.59	57.43	28.77
DINO [5]	55.20	85.08	55.28	23.79	42.28	74.51	41.75	14.56
DINOv2 [32]	68.86	92.85	69.53	37.51	48.21	72.04	48.44	21.39
DiHT†[36]	84.77	96.56	85.47	65.55	60.54	83.79	61.06	31.43
4.2 Implementation
In the experiment, we resize the query and database images to standardize the inputs, ensuring that the longest side is no more than 480 while maintaining the original aspect ratio. For the baseline methods, we implement BoW in Python according to [2], while for the others we adapt their open source implementations to image retrieval task. Specifically, for both BoW and FIRe, we build the codebook using 10k images randomly sampled from the database images. For DELG, we follow its default protocols and extract multi-scale local and global features for both query and database images. For all top-only methods, we produce multi-scale feature representations as well. To be specific, we firstly build an image pyramid by resizing the input image and then center cropping. In our implementation, to strike a balance between accuracy and inference speed, we use 3 scales,  $\{\frac{1}{\sqrt{2}}, 1, \sqrt{2}\}$ , for query images, and 7 scales [30] for database images. Next, we compute the global image features at each scale and apply  $L_{2}$  normalization to them. Finally, we aggregate all the features by average-pooling, followed by another  $L_{2}$  normalization step. Such multi-scale features mitigate the issue of lacking scale invariance for top-only methods. In practice, we observe much improved accuracy of multi-scale features compared to the single-scale ones.
The source code for all the implementations is available at https://github.com/pxiangwu/FORB/, and licensed under the MIT license.
In Table 4 we report image retrieval accuracy for different methods in terms of mAP@5 and  $t$ -mAP@5 (see supplementary material for more results). It can be observed that:
(1) Image embeddings built from handcrafted low-level features can be more universal than many learning-based global image descriptors. In particular, while BoW was introduced decades ago and manually designed, it still outperforms DELG and many top-only methods on our FORB benchmark, demonstrating its strong generalization ability. Moreover, from  $t$ -mAP it can be observed that BoW is better at separating true positives from irrelevant candidates, giving a larger matching score margin.
(2) Mid-level image features are more discriminative than low-level descriptors, and their induced global image embeddings exhibit a superior generalization ability over OOD domains. In Table 4 we investigate one baseline method, i.e., FIRe, which builds embeddings from mid-level features. It can be observed that FIRe overall achieves the best performance among all baselines, with the highest  $t$ -mAP while giving an mAP on par with CLIP. To extract mid-level features, FIRe needs a model training procedure. Surprisingly, although FIRe was trained on 3D landmark images, it can still work well on 2D flat object domains. This could be because in principle the mid-level features of FIRe are similar to the low-level ones, but they typically cover a larger image region and thus incorporate more semantic information, leading to much improved discriminative ability.
Table 5: Retrieval accuracies on diverse objects. We report overall mAP and  $t$ -mAP. Bolded numbers indicate the best results.  $\dagger$  means the model training data may overlap with FORB and the retrieval accuracy can be interpreted as an "upper bound" performance.
Method	mAP@5 (%)/t-mAP@5 (%)
Animated Card	Photorealistic Card	Book Cover	Painting	Currency	Logo	Packaged Goods	Movie Poster
BoW [2]	85.93 / 70.42	79.82 / 62.98	87.92 / 72.68	73.33 / 53.64	70.79 / 52.64	29.98 / 20.20	88.57 / 70.93	73.40 / 53.19
BoW (+ rerank) [2]	89.68 / 76.65	84.58 / 70.55	89.57 / 77.25	77.94 / 62.31	73.46 / 59.81	20.06 / 16.26	82.10 / 70.68	76.90 / 61.31
FIRE [47]	93.92 / 83.50	95.69 / 85.17	90.55 / 80.40	88.61 / 78.24	81.57 / 69.72	42.50 / 33.32	92.69 / 81.21	85.35 / 71.03
DELG [4]	53.86 / 43.42	43.78 / 24.95	58.83 / 39.66	29.75 / 16.08	65.64 / 47.77	13.45 / 7.92	69.88 / 46.37	42.09 / 25.10
DELG (+ rerank) [4]	64.95 / 50.42	55.63 / 28.38	67.91 / 42.50	39.83 / 18.24	73.94 / 50.93	19.17 / 9.91	76.23 / 48.32	49.80 / 27.01
CLIP†[40]	91.93 / 72.91	74.26 / 54.00	99.17 / 71.48	93.12 / 63.43	87.30 / 73.95	85.29 / 53.90	98.14 / 79.10	86.99 / 53.92
SLIP [28]	34.15 / 24.84	47.51 / 34.30	45.50 / 22.34	55.93 / 26.71	29.74 / 24.20	14.92 / 3.25	64.12 / 29.64	38.28 / 19.52
BLIP†[23]	64.87 / 51.64	74.22 / 58.61	93.43 / 55.50	79.60 / 45.10	68.93 / 50.17	82.86 / 29.25	96.37 / 51.04	69.51 / 32.64
BLIP2†[22]	78.21 / 64.47	78.23 / 57.77	96.44 / 61.86	84.43 / 42.36	78.32 / 55.11	80.59 / 31.74	97.70 / 63.32	73.53 / 33.82
DINO [5]	52.75 / 41.27	80.16 / 63.26	48.78 / 33.20	65.90 / 51.90	53.49 / 41.27	6.15 / 3.14	77.15 / 56.19	55.47 / 41.11
DINOv2 [32]	70.00 / 45.29	87.76 / 76.93	68.56 / 37.68	79.92 / 57.40	65.30 / 55.83	6.62 / 1.44	92.26 / 68.55	65.04 / 35.88
DiHT†[36]	83.02 / 67.74	78.22 / 62.49	95.66 / 65.24	93.25 / 47.97	84.41 / 59.14	78.09 / 26.77	98.38 / 73.18	80.33 / 37.85
Table 6: Architectures and inference speeds (seconds / query) of different methods.
Method	Architecture	Speed	Method	Architecture	Speed
BoW [2]	-	0.410	SLIP [28]	ViT-L/16	0.209
BoW (+ rerank) [2]	-	0.418	BLIP [23]	ViT-L/16	0.211
FIRE [47]	ResNet-50	0.124	BLIP2 [22]	ViT-g/14 + QFormer	0.341
DELG [4]	ResNet-50	0.376	DINO [5]	ViT-S/8	0.177
DELG (+ rerank) [4]	ResNet-50	6.015	DINOv2 [32]	ViT-L/14	0.222
CLIP [40]	ViT-L/14@336px	0.513	DiHT [36]	ViT-L/14@336px	0.357
(3) For top-only methods, their retrieval accuracies on OOD domains improve with increasing size of model and training data. For example, since the training data of DINO and SLIP are relatively smaller than others, the generalization ability of their image embeddings is inferior to that of CLIP and DiHT, which employ larger model and training set.
(4) Image embeddings based on low- and mid-level features cannot adequately distinguish feature-scarce images. As shown in Table 5, both BoW and FIRe fail to accurately recognize logos, which typically consist of simple patterns and contain sparse features. In contrast, the top-only methods are better at handling logos, probably because they describe images based on their high-level semantics and thus suffer less from the lack of lower level features.
In addition to the retrieval accuracies, we also show the inference speeds of different methods in Table 6. We measure the speed on a machine with 120 GB RAM, an NVIDIA T4 GPU and 32 Intel Xeon CPUs (@2.30GHz). Notably, although CLIP achieves the highest mAP among all the methods, it is not efficient since the feature extraction is computationally expensive. In contrast, FIRe runs at a much faster speed, with a similar mAP and even better  $t$ -mAP. This further demonstrates the advantages of bottom-up strategy and mid-level features.
As shown in Table 4, while being effective on certain object domains, the embeddings from most of the baseline methods are not universal enough to generalize to diverse open-world objects. This affirms the need for the proposed FORB benchmark to further strengthen the research in the generalization ability of image embeddings. In addition, our benchmark results show that even trained with 3D landmark images, embeddings produced by FIRe can still well distinguish images from OOD domains, indicating the great potential of mid-level features in retrieval tasks. In particular, given the advantages and weaknesses of mid-level features, one future direction would be to develop a retrieval method that jointly leverages the mid- and high-level image features, giving image embeddings that share the benefits of both sides.
We present FORB, a benchmark for flat object retrieval and matching. Essentially FORB supplements existing image retrieval benchmarks, and more importantly, it serves as a test bed for evaluating the generalization abilities of image embeddings on OOD domains. Our experiments on FORB show
that embeddings based on low- and mid-level image features overall are more universal than those constructed from high-level semantics. Notably, we observe that the mid-level features introduced by FIRE are surprisingly general and give the best overall retrieval performance, even if the model is trained on 3D landmarks. However, despite the overall inferiority, embeddings of high-level semantics are usually more effective for images that contain sparse features. These findings suggest that one potential future direction would be to develop methods that jointly leverage the mid- and high-level image features and combine the strengths of both.
Limitations and future work. In our experiment, we compare baselines which have different model sizes and are trained on various datasets. As a result, the comparisons among these methods and their corresponding embeddings could be unfair to some extent. In addition, our FORB benchmark currently only considers distractors from the same domain as the index images. To improve the diversity and challenges of our benchmark, in the future we plan to collect more distractors from other domains. In addition, to further enrich the OOD queries, we also plan to curate queries beyond the domains of index images, a practice which is similar to GLDv2 [49]. In this way we can better measure the matching score margins of different methods with  $t$ -mAP. Despite these limitations, our benchmark still serves as a supportive dataset for further research in the task of image retrieval. We hope our work would facilitate the understanding of different image embeddings and promote the design of new methods.
[1] Xiang An, Jiankang Deng, Kaicheng Yang, Jaiwei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Unicom: Universal and compact representation learning for image retrieval. 2023.
[2] Relja Arandjelović and Andrew Zisserman. Three things everyone should know to improve object retrieval. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2911-2918. IEEE, 2012.
[3] Artem Babenko and Victor Lempitsky. Aggregating local deep features for image retrieval. In Proceedings of the IEEE international conference on computer vision, pages 1269-1277, 2015.
[4] Bingyi Cao, Andre Araujo, and Jack Sim. Unifying deep local and global features for image search. In European conference on computer vision, pages 726-743. Springer, 2020.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[7] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021.
[8] Gabriella Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski, and Cedric Bray. Visual categorization with bags of keypoints. In Workshop on statistical learning in computer vision, ECCV, volume 1, pages 1-2. Prague, 2004.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. 2020.
[11] Albert Gordo, Jon Almazán, Jerome Revaud, and Diane Larlus. Deep image retrieval: Learning global representations for image search. In European conference on computer vision, pages 241-257. Springer, 2016.
[12] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar,
[47] Philippe Weinzaepfel, Thomas Lucas, Diane Larlus, and Yannis Kalantidis. Learning superfeatures for image retrieval. International conference on learning representations, 2022.
[48] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
[49] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2575-2584, 2020.