SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pengxiang Wu, Siman Wang, Kevin Dela Rosa, Derek Hao Hu
{pwu, swang7, kevin.delarosa, hao.hu}@snap.com
Image retrieval is a fundamental task in computer vision. Despite recent advances in this field, many techniques have been evaluated on a limited number of domains, with a small number of instance categories. Notably, most existing works only consider domains like 3D landmarks, making it difficult to generalize the conclusions made by these works to other domains, e.g., logo and other 2D flat objects. To bridge this gap, we introduce a new dataset for benchmarking visual search methods on flat images with diverse patterns. Our flat object retrieval benchmark (FORB) supplements the commonly adopted 3D object domain, and more importantly, it serves as a testbed for assessing the image embedding quality on out-of-distribution domains. In this benchmark we investigate the retrieval accuracy of representative methods in terms of candidate ranks, as well as matching score margin, a viewpoint which is largely ignored by many works. Our experiments not only highlight the challenges and rich heterogeneity of FORB, but also reveal the hidden properties of different retrieval strategies. The proposed benchmark is a growing project and we expect to expand in both quantity and variety of objects. The dataset and supporting codes are available at https://github.com/pxiangwu/FORB/.
Image retrieval is a fundamental and long-standing task in computer vision. Given a query image, this task aims to search for the most similar images from a large database. Recent methods have achieved remarkable performance on certain domains, such as 3D landmark [37, 49] and clothes [25]. To perform image retrieval, the prevailing practice is to map the query image into a compact embedding space, where similar images are close to each other while dissimilar ones are separated away. This embedding space can be handcrafted and one classic design is the Bag of Words (BoW) [2, 8]. A more effective idea is to learn the embedding automatically, based on deep neural netowrkst [11, 3, 29, 38]. However, all these methods have only been evaluated on a limited number of domains (e.g., 3D landmarks), and as a result, it remains unclear if the embedding of one method is more general than the others. In particular, for learning-based methods, since they are usually trained on a specific restricted dataset with limited object classes (e.g., ImageNet [9] and Open Images [21]), their feature embeddings could be not universal enough to generalize to various open-world objects. Therefore, it is necessary to have benchmarks supplementary to the existing ones for a more comprehensive evaluation of the embeddings, especially in terms of their out-of-distribution (OOD) generalization ability.
In particular, existing image retrieval benchmarks mainly involve domains of 3D objects. Examples of the commonly considered objects include 3D landmarks, clothes, natural living things and online products. While many recent benchmarks that curate the images of these objects have sufficiently large query image sets, they are typically limited to a small number of object categories or instances. Moving beyond 3D objects, there are several datasets focusing on 2D flat objects. However, these
(a) Database
(b) Query (easy)
(c) Database
(d) Query (Medium)
(e) Database
(f) Query (medium)
(g) Database
(h) Query (hard)
(i) Database
(j) Query (medium)
(k) Database
(1) Query (medium)
(m) Database
(n) Query (hard)
(o) Database
Figure 1: Example database and query images from our FORB benchmark. For each query image, we show its corresponding index image and the retrieval difficulty. The images are from different content domains: (a)(b) photorealistic trading card; (c)(d) book cover; (e)(f) painting; (g)(h) logo; (i)(j) packaged goods; (k)(l) movie poster; (m)(n) animated trading card; (o)(p) currency.
(p) Query (medium)
datasets are mostly small in size and related to one particular type of object, i.e., logo [41, 45]. Besides, their query images tend to be in canonical pose without much distraction from the background, making the retrieval less challenging.
In order to fill the domain gap of existing benchmarks and to encourage future research in this area, we present a Flat Object Retrieval Benchmark (FORB) which contains diverse flat objects with different query difficulties. The flat objects are those with 2D surface only, which bears the textures and patterns of the object (e.g., painting and logo; see Figure 1). Despite being one dimension less than 3D objects, such flat surfaces still pose many challenges for image retrieval. In particular, there can be large variations between the query and database images, due to surface and color distortions, perspective transformation, view occlusion, and illumination change. Our benchmark takes into account all these challenges and covers objects with a variety of textures (see Section 3). Notably, these objects are common in daily life and our benchmark could benefit diverse real-world visual search applications, such as recognizing logos for brand promotion, augmenting artwork exhibits in a museum, online shopping and more.
To understand how different image embeddings perform on our benchmark, we evaluate the retrieval accuracy from two perspectives: (1) Candidate rank, which corresponds to the sorted order of database images based on their similarities to the query image. The correctness of ranks reflects the discriminative ability of image embedding and can be measured with mean Average Precision (mAP). (2) Matching score margin. For a query image, ideally its matching scores against ground-truth database images should be high (e.g., assuming cosine similarity), while the scores against non-relevant images should be low. Therefore, the degree of compliance with this ideal margin also delineates the quality of image embedding, a viewpoint which is largely ignored by previous works. To measure this margin, we propose to query the given image against distractor images, giving false positive candidates. By thresholding the matching scores, we can compute a specific false positive rate (FPR) and an updated mAP, which together quantify the margin of image embeddings. In particular, an ideal embedding should have a low FPR while keeping a high mAP.
To establish baselines on our benchmark, we evaluate a series of representative methods, including both learning-based models and handcrafted designs. Our results reveal intriguing properties of
Table 1: Comparison of our benchmark against existing image retrieval datasets.
Dataset	Domain	# Query	# Database	Has distractor	Has difficulty label
Oxford [33]	3D landmark	55	5K	✘	✘
Paris [35]	3D landmark	55	6K	✘	✘
R-Oxford [37]	3D landmark	70	5K + 1M	✓	✓
R-Paris [37]	3D landmark	70	5K + 1M	✓	✓
GLD [30]	3D landmark	118K	1.1M	✓	✘
GLDv2 [49]	3D landmark	118K	762K	✓	✘
CUB [48]	Bird	6K	6K	✘	✘
Cars196 [20]	Car	8K	8K	✘	✘
SOP [31]	3D product	60K	60K	✘	✘
DeepFashion [25]	Clothes	14K	13K	✘	✘
VehicleID [24]	Vehicle	35.6K	4.8K	✘	✘
iNaturalist [46]	Plant & Animal	136K	136K	✘	✘
FlickrLogos [41]	Flat object (logo)	4K	320	✓	✘
FORB	Flat object	14K	54K	✓	✓
embeddings built from different feature levels. Specifically, we show that even a model is trained on 3D objects, its embedding induced from low- or mid-level image features can still be universal enough to distinguish diverse flat objects. Moreover, for feature-scarce images, embeddings based on high-level features tend to achieve better accuracy.
Our contributions include: (1) We introduce FORB, a new visual search benchmark for evaluating image embeddings on flat objects. FORB supplements the commonly used 3D object benchmarks and essentially provides a platform for assessing the OOD generalization ability of an embedding method. (2) We propose a new evaluation metric motivated by matching score margin. This metric is complementary to mAP and offers a new perspective on image embedding quality. (3) We conduct comprehensive comparisons for different representative methods, providing solid baselines for future method developments. (4) Our evaluation results reveal the hidden properties of different retrieval strategies as well as their limitations, providing insights into the development of new techniques.
2.1 Existing Datasets and Benchmarks for Image Retrieval
There has been a long history of developing benchmarks for image retrieval. For example, to promote research in instance-level recognition and search, Oxford [33] and Paris [35] datasets were introduced and have motivated a wealth of innovations in this field. With a similar motivation, researchers curated CUB [48] and Cars196 [20] to facilitate fine-grained object matching. Despite the popularity of these datasets, they are small in size and only involve a limited number of instances and categories. To further enrich the object domains for image retrieval and increase the size and complexity of the task, several more challenging datasets were constructed, such as SOP [31], DeepFashion [25], VehicleID [24], iNaturalist [46], and Google Landmarks dataset v2 (GLDv2) [49]. In particular, GLDv2 has gained widespread attention since being introduced due to its significant scale and variability, and serves as a solid benchmark for testing emerging retrieval techniques.
One limitation of these datasets is that they only focus on the task of 3D object retrieval, involving a restricted number of object domains (e.g., 3D landmarks). In fact, compared to 3D objects, there exist few benchmarks on other domains, especially 2D flat objects. In real-world visual search applications, flat objects also make up a large fraction of queries. However, there are only few benchmarks on such objects and most of them are for logo [41, 45]. To fill this domain gap, our FORB benchmark includes a variety of flat objects and supplements existing 3D object benchmarks. In particular, FORB effectively serves as an OOD query set for evaluating the embeddings trained on 3D objects. In Table 1 we compare FORB against existing image retrieval datasets in detail.
It is worth mentioning that there exists another similar benchmark for assessing the generalization abilities of image embeddings, i.e., Google Universal Image Embedding Challenge<sup>1</sup>. However, this benchmark mainly involves 3D objects and its evaluation data is kept private. We believe our FORB
<sup>1</sup>https://www.kaggle.com/competitions/google-universal-image-embedding
supplements this benchmark and will facilitate the development of visual search applications, such as organizing photo collections, visual commerce and more.
2.2 Out-of-Distribution Query
Most existing benchmarks only have "on-topic" queries without considering the out-of-distribution ones. As a result, they fail to present real-world challenges and are not enough to fully evaluate the quality of an image embedding. Notably, in a generic visual search app, the system tends to be queried with a large number of irrelevant queries, i.e., OOD queries, for which it is expected to not yield any results. Therefore, OOD queries provide an additional important view into the robustness of image embeddings. This issue of lacking OOD queries in existing benchmarks was recognized in GLDv2 [49] and addressed with plenty of non-landmark queries. In practice, to assess the discriminative ability of image embeddings between true positive and false positive candidates, GLDv2 employs micro Average Precision ( $\mu$ AP), which both measures ranking performance and penalizes false positive predictions. Our FORB benchmark shares a similar motivation to GLDv2, but with a few key differences: (1) We do not provide additional OOD queries with respect to the database images. Instead, we split database into index images and distractors, and query the images against distractors. In this way we effectively turn all the query images into OOD queries. (2) Instead of using  $\mu$ AP, we propose a new metric,  $t$ -mAP, which computes an averaged mAP over different confidence thresholds. The thresholds are determined through quantiles of false positive rates. Compared to  $\mu$ AP and mAP, our  $t$ -mAP takes into account the matching score margin, which directly reflects the discriminability of image embeddings.
2.3 Universal Image Embedding
The quality of image embeddings determines the performance of modern image retrieval methods. Based on the design of image features, existing embeddings can be divided into two categories: handcrafted and learning-based. The former one builds image embeddings based on handcrafted low-level features (e.g., SIFT [26]), using a bag of words (BoW). This design paradigm dominates many classic methods, such as [34, 27, 2, 18, 44], and usually leads to embeddings that generalize well over various domains. With the rapid advancement of deep learning, such handcrafted embeddings have been replaced with the learning-based ones in the community. The learning of image embeddings is commonly conducted in a supervised manner, on crowd-labeled datasets [15, 17, 16, 10]. However, supervised learning is not scalable since manual annotation of large-scale training data is time-consuming and costly. As a result, the training data usually contains limited pre-defined object classes (e.g., ImageNet [9] and Open Images [21]), and embeddings learned from these data are not universal enough to generalize to various open-world objects [1]. In recent years, self- and weakly-supervised learning have gained extensive attention due to their less reliance on labeled data. By designing appropriate pre-text tasks and training strategies (e.g., image-text matching), these learning paradigms can easily leverage a large number of unlabeled or noisy data, producing image embeddings of greater generality than supervised learning [14, 13, 7, 6, 12, 40, 19].
3 The FORB Benchmark
Our FORB benchmark only provides testing query images without training data. It serves as a testbed supplementary to existing benchmarks, with the following goals.
Goals Our proposed benchmark aims to enrich the object domains considered in image retrieval tasks and measure the generalization ability of embedding models with respect to out-of-distribution queries. Besides, we also seek to understand the effects of image features from different levels on the embedding quality, thereby shedding light on future development of embedding models.
There are 8 different types of flat objects involved in our benchmark: (1) Animated trading card. We consider one particular type of card, i.e., Pokemon trading card. (2) Photorealistic trading card. We consider cards for different sports, such as baseball, basketball, and football. (3) Book cover, which comes from books in different languages, such as English and Chinese. (4) Painting, which involves
Table 2: Overview of the proposed FORB benchmark.
Object Type	# Query	# Index	# Distractor	# Easy	# Medium	# Hard
Animated trading card	6,025	1,392	11,137	714	4,868	443
Photorealistic trading card	2,187	484	521	67	2,039	81
Book cover	1,461	470	10,739	66	1,277	118
Painting	988	430	615	119	710	159
Currency	758	395	1,188	112	576	70
Logo	1170	535	174	24	957	189
Packaged goods	800	476	2,382	24	727	49
Movie poster	512	403	23,094	49	426	37
Total	13,901	4,585	49,850	1,175	11,580	1,146
various styles, such as impressionism and baroque, etc. (5) Currency, which involves banknotes of modern and antique designs from different countries. We consider both the front and back of a banknote. (6) Logo. We consider common logos (e.g., Nike) as well as long-tailed logos (e.g., brands of local small businesses). (7) Packaged goods. We only consider products for which the corresponding index images are displayed on flat surface. (8) Movie poster. We consider posters from different countries, such as America and Japan. In Figure 1 we show examples for each object. As can be seen, these objects have diverse textures, involving animation and artificial patterns, etc, and thus offer various retrieval challenges. Also, they are common in daily life and retrieving such objects serves as a practical use case in real applications. For example, eBay builds an image retrieval system<sup>2</sup> for trading cards to facilitate the sales of cards.
To build our benchmark, we collected the query and index images mainly via Google Images. Specifically, before collecting images, for each type of objects we firstly curated a list of object names. Their names can be obtained from dedicated websites, such as TCGplayer<sup>3</sup> for animated trading card and Wikimedia Commons<sup>4</sup> for painting. Next, we queried Google Images with each of the names and retrieved the corresponding query and index images. The returned results were typically noisy and we manually filtered out the irrelevant images as well as those that could be copyright protected. In this way, we effectively matched each index image with diverse query images, giving image-level ground truths. Note that our collected query images are in the wild whereas the database images are in canonical pose (see Figure 1). Besides Google Images, we also leveraged some other sources to further augment the benchmark, such as Google Lens API, eBay, and Amazon.
To increase retrieval difficulty and challenge, similar to previous works [37, 49] we also introduced distractors to the benchmark. The distractors are images that share similar semantics, contents, or textures with the index images. They can be from the same domains as the index images, or from other domains. Distractors are primarily introduced to increase the retrieval difficulty, as they would bring perplexing features that deceive retrieval algorithms and reduce the accuracy of retrieval results. Ideally, a strong retrieval algorithm should be robust against distractors. In our benchmark, the distractor images were all from the 8 object domains and crawled from different specific websites, such as TCGplayer and Wikimedia Commons. See supplementary material for some examples. The details of our benchmark can be found in Table 2.
3.2 Data Annotation and Metadata
As mentioned above, we provide image-level retrieval ground truths for each query image. To enable a more detailed evaluation on the quality of image embeddings, we also offer annotations on the retrieval difficulties for each query image. Specifically, we break down difficulty into three levels: easy, medium, and hard. The specific difficulty level for a query image is subject to the following factors: (1) occlusion; (2) blur; (3) truncation; (4) color distortion; (5) perspective distortion; (6) texture complexity; (7) area of the object in the query image. For example, if the target object only occupies a small area in the image, we tag "hard" for the given query image due to the distraction of background; see Figure 1(h)(n). Similarly, if the object does not bear severe perspective distortion or truncation, we tag "easy"; see Figure 1(b). In practice, assigning difficulty levels to query images can
2https://pages.ebay.com/scantolist/
<sup>3</sup>https://www.tcgplayer.com/
<sup>4</sup>https://commons.wikimedia.org/
be a subjective process. To reduce bias and ensure precise difficulty assessment, we involve different annotators in manually labeling the difficulty of each image and then use majority voting to determine the final difficulty level. As shown in Table 4, the annotated difficulty levels are quite consistent with retrieval accuracies for all methods, i.e., the accuracies are high on easy queries, whereas they are low on hard queries.
We store the annotations with a newline delimited JSON file, where each line contains the metadata corresponding to a query image. Specifically, each line is comprised of the following information: (1) query image ID; (2) the file name of query image; (3) the source URL of query image; (4) the file names of ground-truth index images; (5) the source URLs of ground-truth index images; (6) difficulty level. Here the source URL corresponds to where we downloaded the image. Apart from this annotation information, we also provide newline delimited JSON files for tracking the set of query and database images, respectively, where each line contains information regarding the image file name and source URL.
We host the metadata files at https://github.com/pxiangwu/FORB/, which is publicly accessible. As for the query and database images, they can be downloaded via the provided source image URLs. Alternatively, these images are also accessible from a Google drive, where we snapshot all the images from source URLs. Both the metadata and image files are licensed under CC BY-NC-SA.
3.3 Metrics
Our FORB benchmark uses the commonly adopted mAP metric, as well as a new one that takes into account the matching score margin.
mAP The mean Average Precision metric considers both the true positives and false positives in the ranked retrieval results. The metric is defined as follows:
$$
\mathrm {m A P} @ k = \frac {1}{Q} \sum_ {q = 1} ^ {Q} \mathrm {A P} @ k (q), \quad \mathrm {A P} @ k (q) = \frac {1}{\min  \left(m _ {q} , k\right)} \sum_ {k = 1} ^ {\min  \left(n _ {q}, k\right)} \mathrm {P} _ {q} (k) \operatorname {r e l} _ {q} (k), \tag {1}
$$
where  $Q$  is the total number of query images;  $m_q$  is the number of ground-truth index images matched with query image  $q$ ;  $n_q$  is the number of predictions made by the retrieval method;  $\mathrm{P}_q(k)$  is the precision at rank  $k$  for query image  $q$ ; and  $\mathrm{rel}_q(k)$  is a relevance indicator function which equals 1 if the result at rank  $k$  is relevant and equals to 0 otherwise. Note that for some query images (e.g., OOD images) they do not have associated index images to retrieve, and mAP does not penalize the method even if it retrieves some results for the query images.
$t$ -mAP To take into account OOD queries and false positive results, we introduce thresholded mAP, i.e.,  $t$ -mAP. This metric measures the matching score margin with the aid of OOD queries, and is computed as below:
$$
t - \mathrm {m A P} = \frac {1}{\tau (1)} \int_ {0} ^ {\tau (1)} \mathrm {m A P} (t) d t, \tag {2}
$$
where  $\tau(x)$  is the threshold that leads to a false positive rate of  $1 - x$  on OOD queries after thresholding the retrieved candidates with respect to their matching scores; formally,  $\tau(x) = \min\{\tilde{x} \mid \mathrm{FPR}(\tilde{x}) = 1 - x\}$ , where  $\mathrm{FPR}(\tilde{x})$  is the false positive rate at threshold  $\tilde{x}$ .  $\mathrm{mAP}(t)$  is the mAP computed after the retrieval results are suppressed at threshold  $t$ . Note that  $\mathrm{mAP}(t)$  tends to decrease with increasing threshold  $t$ . However, for an ideal universal image embedding, it is expected to still have a high mAP even at threshold  $\tau(1)$ , due to its strong discriminability between true positives and false positives.
In practice, to numerically compute Equation (2), we uniformly sample 11 thresholds and average  $\mathrm{mAP}(t)$  over them:
$$
t - \mathrm {m A P} = \frac {1}{1 1} \sum_ {t \in \{0, \tau (0. 1), \dots , \tau (1. 0) \}} \mathrm {m A P} (t). \tag {3}
$$
As can be seen,  $t$ -mAP takes value from [0, 1], with higher value indicating better performance.
Table 3: The training data used by different image retrieval methods. "Web images" means the training data are sourced from the Internet and typically comprise various 3D objects along with some flat objects. We use the generic term "3D objects" to indicate the training data involve diverse 3D objects, such as 3D landmarks, plants, and animals, etc.
Method	Training data	Domain	# images	Method	Training data	Domain	# images
BoW [8]	-	-	-	BLIP [23]	129M [23]	3D objects + web images	129M
FIRE [47]	SfM-120k [39]	3D landmark	120K	BLIP2 [22]	129M [23]	3D objects + web images	129M
DELG [4]	GLD [30]	3D landmark	960K	DINO [5]	ImageNet [9]	3D objects	1M
CLIP [40]	Proprietary 400M	Web images	400M	DINOv2 [32]	LVD-142M [32]	3D objects	142M
SLIP [28]	YFCC15M [43]	Web images	15M	DiHT [36]	LAION-438M [42]	Web images	438M
In this section, we evaluate several representative image retrieval methods on our FORB benchmark. Based on the evaluation results, we also provide a detailed analysis on the behavior of different image embeddings and their intriguing properties.