SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qianqian Shen $^{1,*}$ , Yunhan Zhao $^{2,*}$ , Nahyun Kwon $^{3}$ , Jeeun Kim $^{3}$ , Yanan Li $^{1,\dagger}$ , Shu Kong $^{3,4,5,\dagger}$
$^{1}$ Zhejiang Lab  $^{2}$ UC-Irvine  $^{3}$ Texas A&M University
$^{4}$ Institute of Collaborative Innovation  $^{5}$ University of Macau
shenqq@zhejianglab.com, yunhaz5@ics.uci.edu, {nahyunkwon, jeeun.kim, shu}  $@$  tamu.edu,
liyn@zhejianglab.com, skong@um.edu.mo
Dataset and open-source code
Instance detection (InsDet) is a long-lasting problem in robotics and computer vision, aiming to detect object instances (predefined by some visual examples) in a cluttered scene. Despite its practical significance, its advancement is overshadowed by Object Detection, which aims to detect objects belonging to some predefined classes. One major reason is that current InsDet datasets are too small in scale by today's standards. For example, the popular InsDet dataset GMU (published in 2016) has only 23 instances, far less than COCO (80 classes), a well-known object detection dataset published in 2014. We are motivated to introduce a new InsDet dataset and protocol. First, we define a realistic setup for InsDet: training data consists of multi-view instance captures, along with diverse scene images allowing synthesizing training images by pasting instance images on them with free box annotations. Second, we release a real-world database, which contains multi-view capture of 100 object instances, and high-resolution  $(6\mathrm{k}\times 8\mathrm{k})$  testing images. Third, we extensively study baseline methods for InsDet on our dataset, analyze their performance and suggest future work. Somewhat surprisingly, using the off-the-shelf class-agnostic segmentation model (Segment Anything Model, SAM) and the self-supervised feature representation DINOv2 performs the best, achieving  $>10$  AP better than end-to-end trained InsDet models that repurpose object detectors (e.g., FasterRCNN and RetinaNet).
Instance detection (InsDet) requires detecting specific object instances (defined by some visual examples) from a scene image [12]. It is practically important in robotics, e.g., elderly-assistant robot robots need to fetch specific items (my-cup vs. your-cup) from a cluttered kitchen [40], micro-fulfillment robots for the retail need to pick items from mixed boxes or shelves [4].
Motivation. InsDet receives much less attention than the related problem of Object Detection (ObjDet), which aims to detect all objects belonging to some predefined classes [28, 37, 29, 48]. Fig. 1 compares the two problems. One major reason is that there are not large-enough InsDet datasets by today's standards. For example, the popular InsDet dataset GMU (published in 2016) [15] has only 23 object instances while the popular ObjDet dataset COCO has 80 object classes (published in 2014) [28]. Moreover, there are no unified protocols in the literature of InsDet. The current InsDet literature mixes multiple datasets to simulate training images and testing scenarios [12]. Note that the
*Equal contributions; † Corresponding authors.
Object detection (ObjDet): detecting objects belonging to
these classes:
Figure 1: Object detection (ObjDet) vs. instance detection (InsDet). ObjDet aims to detect all objects belonging to some predefined classes, whereas InsDet requires detecting specific object instances defined by some visual examples. Loosely speaking, InsDet treats a single object instance as a class compared to ObjDet. Please refer to Fig. 2-right for the challenge of InsDet, which is the focus of our work.
Instance Detection (InsDet): detecting instances with
training protocol of InsDet does not follow that of ObjDet, which has training images annotated with bounding boxes. Differently, for  $\mathrm{InsDet}^2$ , its setup should have profile images of instances (cf. right in Fig. 1) and optionally diverse background images not containing such instances [12]. We release a new dataset and present a unified protocol to foster the InsDet research.
Overview of our dataset is presented in Fig. 2. In our dataset, profile images (3072x3072) of object instances and testing images (6144x8192) are high-resolution captured by a Leica camera (commonly used in today's cellphones). This inexpensive camera is deployable in current or future robot devices. Hence, our dataset simulates real-world scenarios, e.g., robotic navigation in indoor scenes. Even with high-resolution images, objects in testing images appear small, taking only a tiny region in the high-res images. This demonstrates a clear challenge of InsDet in our dataset. Therefore, our dataset allows studying InsDet methods towards real-time operation on high-res (as future work).
Preview of technical insights. On our dataset, we revisit existing InsDet methods [26, 12, 16]. Perhaps the only InsDet framework is cut-paste-learn [12], which cuts instances from their profile images, pastes them on random background images (so being able to derive "free" bounding boxes annotations), and trains InsDet detectors on such data by following that of ObjDet (e.g., FasterRCNN [37]). We study this framework, train different detectors, and confirm that the state-of-the-art transformer-based detector DINO [48] performs the best, achieving 27.99 AP, significantly better than CNN-based detector FasterRCNN (19.52 AP). Further, we present a non-learned method that runs off-the-shelf proposal detectors (SAM [23] in our work) to generate object proposals and use self-supervised learned features  $(\mathrm{DINO}_f[8]^3$  and  $\mathrm{DINOv2}_f$  [33]) to find matched proposals to instances' profile images. Surprisingly, this non-learned method resoundingly outperforms end-to-end learning methods, i.e., SAM+DINOv2  $^f$  achieves 41.61 AP, much better than DINO (27.99 AP) [48].
Contributions. We make three major contributions.
1. We formulate the InsDet problem with a unified protocol and release a challenging dataset consisting of both high-resolution profile images and high-res testing images.
2. We conduct extensive experiments on our dataset and benchmark representative methods following the cut-paste-learn framework [12], showing that stronger detectors perform better.
3. We present a non-learned method that uses an off-the-shelf proposal detector (i.e., SAM [23]) to produce proposals, and self-supervised learned features (e.g., DINOv2f [33]) to find instances (which are well matched to their profile images). This simple method significantly outperforms the end-to-end InsDet models.
Instance detection (InsDet) is a long-lasting problem in computer vision and robotics [49, 12, 32, 3, 15, 21, 4], referring to detecting specific object instances in a scene image. Traditional InsDet methods use keypoint matching [34] or template matching [19]; more recent ones train deep neural networks to approach InsDet [32]. Some others focus on obtaining more training samples by rendering realistic instance examples [22, 21], data augmentation [12], and synthesizing training images by cutting
2In real-world applications (e.g., robot learning), it is infeasible to place objects in diverse scenes, take scene photos, then annotate instances using boxes towards training images (cf. training data in object detection).
3We add subscript  $f$  to indicate that  $\mathrm{DINO}_f$  [8] is the self-supervised learned feature extractor; distinguishing it from a well-known object detector DINO [48].
Figure 2: Overview of our instance detection dataset. Left: It contains 100 distinct object instances. For each of them, we capture 24 profile photos from multiple views. We paste QR code images beneath objects to allow relative camera estimation (e.g., by COLMAP [41]), just like other existing datasets [20, 5]. Middle: We take photos in random scenes (which do not contain any of the 100 instances) as background images. The background images can be optionally used to synthesize training data, e.g., pasting the foreground instances on them towards box-annotated training images [26, 12, 16] as used in the object detection literature [28]. Right: high-resolution  $(6\mathrm{k}\times 8\mathrm{k})$  testing images of clutter scenes contain diverse instances, including some of the 100 predefined instances and other uninterested ones. The goal of InsDet is to detect the predefined instances in these testing images. From the zoom-in regions, we see the scene clutters make InsDet a rather challenging problem.
instances as foregrounds and pasting them to background images [26, 12, 16]. Speaking of InsDet datasets, [15] collects scene images from 9 kitchen scenes with RGB-D cameras and defines 23 instances of interest to annotate with 2D boxes on scene images; [21] creates 3D models of 29 instances from 6 indoor scenes, and uses them to synthesize training and testing data; [4] creates 3D mesh models of 100 grocery store objects, renders 80 views of images for each instance, and uses them to synthesize training data.
As for benchmarking protocol of InsDet, [12] synthesizes training data from BigBird [43] and UW Scenes [25] and tests on the GMU dataset [15]; [21] trains on their in-house data and test on LM-O [5] and Rutgers APC [38] datasets. Moreover, some works require hardware-demanding setups [4], some synthesize both training and testing data [21, 26], while others mix existing datasets for benchmarking [12]. Given that the modern literature on InsDet lacks a unified benchmarking protocol (till now!), we introduce a more realistic unified protocol along with our InsDet dataset, allowing fairly benchmarking methods and fostering research of InsDet.
Object detection (ObjDet) is a fundamental computer vision problem [13, 28, 37], requiring detecting all objects belonging to some predefined categories. The prevalent ObjDet detectors adopt convolutional neural networks (CNNs) as a backbone and a detector-head for proposal detection and classification, typically using bounding box regression and a softmax-classifier. Approaches can be grouped into two categories: one-stage detectors [36, 30, 35, 46] and two-stage detectors [17, 6]. One-stage detectors predict candidate detection proposals using bounding boxes and labels at regular spatial positions over feature maps; two-stage detectors first produce detection proposals, then perform classification and bounding box regression for each proposal. Recently, the transformer-based detectors transcend CNN-based detectors [7, 51, 48], yielding much better performance on various ObjDet benchmarks. Different from ObjDet, InsDet requires distinguishing individual object instances within a class. Nevertheless, to approach InsDet, the common practice is to repurpose ObjDet detectors by treating unique instances as individual classes. We follow this practice and benchmark various ObjDet methods on our InsDet dataset.
Pretrained models. Pretraining is an effective way to learn features from diverse data. For example, training on the large-scale ImageNet dataset for image classification [10], a neural network can serve as a powerful feature extractor for various vision tasks [11, 42]. Object detectors trained on the COCO dataset [28] can serve as a backbone allowing finetuning on a target domain to improve detection performance [27]. Such pretraining requires human annotations which can be costly. Therefore, self-supervised pretraining has attracted increasing attention and achieved remarkable progress [9, 18, 8, 33]. Moreover, the recent literature shows that pretraining on much larger-scale data can serve as a foundation model for being able to perform well across domains and tasks. For example, the Segment Anything Model (SAM) pretrains a class-agnostic proposal detector on web-scale data and shows an impressive ability to detect and segment diverse objects in the wild [23]. In this work, with our high-res InsDet dataset, we explore a non-learned method by using publicly available pretrained models. We show that such a simple method significantly outperforms end-to-end learned InsDet detectors.
3 Instance Detection: Protocol and Dataset
In this section, we formulate a realistic unified InsDet protocol and introduce the new dataset. We release our dataset under the MIT License, hoping to contribute to the broader research community.
3.1 The Protocol
Our InsDet protocol is motivated by real-world indoor robotic applications. In particular, we consider the scenario that assistive robots must locate and recognize instances to fetch them in a cluttered indoor scene [40], where InsDet is a crucial component. Realistically, for a given object instance, the robots should see it only from a few views (at the training stage), and then accurately detect it in a distance in any scenes (at the testing stage). Therefore, we suggest the protocol specifying the training and testing setups below. We refer the readers to Fig. 2 for an illustration of this protocol.
- Training. There are profile images of each instance captured at different views and diverse background images. The background images can be used to synthesize training images with free 2D-box annotations, as done by the cut-paste-learn methods [26, 12, 16].
- Testing. InsDet algorithms are required to precisely detect all predefined instances from real-world images of cluttered scenes.
Evaluation metrics. The InsDet literature commonly uses average precision (AP) at IoU=0.5 [12, 2, 32]; others use different metrics, e.g., AP at IoU=0.75 [21], mean AP [3, 15], and F1 score [4]. As a single metric appears to be insufficient to benchmark methods, we follow the literature of ObjDet that uses multiple metrics altogether [28].
- AP averages the precision at IoU thresholds from 0.5 to 0.95 with the step size 0.05. It is the primary metric in the most well-known COCO Object Detection dataset [28].
-  $\mathbf{AP}_{50}$  and  $\mathbf{AP}_{75}$  are the precision averaged over all instances with IoU threshold as 0.5 and 0.75, respectively. In particular,  $\mathbf{AP}_{50}$  is the widely used metric in the literature of InsDet.
- AR (average recall) averages the proposal recall at IoU threshold from 0.5 to 1.0 with the step size 0.05, regardless of the classification accuracy. AR measures the localization performance (excluding classification accuracy) of an InsDet model.
Moreover, we tag hard and easy scenes in the testing images based on the level of clutter and occlusion, as shown by the right panel of Fig. 2.
3.2 The Dataset
We introduce a challenging real-world dataset of indoor scenes (motivated by indoor assistive robots), including high-resolution photos of 100 distinct object instances, and high-resolution testing images captured from 14 indoor scenes where there are such 100 instances defined for InsDet. Table 1 summarizes the statistics compared with existing datasets, showing that our dataset is larger in scale and more challenging than existing InsDet datasets. Importantly, object instances are located far from the camera in cluttered scenes; this is realistic because robots must detect objects in a distance before approaching them [1]. Perhaps surprisingly, only a few InsDet datasets exist in the literature. Among them, Grocery [4], which is the latest and has the most instances like our dataset, is not publicly available.
Our InsDet dataset contains 100 object instances. When capturing photos for each instance, inspired by prior arts [43, 20, 5], we paste a QR code on the tabletop, which enables pose estimation, e.g., using COLMAP [41]. Yet, we note more realistic scenarios can be hand-holding instances for capturing [24], which we think of as future work. In Fig. 3, we plot the per-instance frequency in the testing set. Each instance photo is of  $3072 \times 3072$  pixel resolution. For each instance, we capture 24 photos from multiple views. The left panel of Fig. 2 shows some random photos for some instances. For the testing set, we capture high-resolution images ( $6144 \times 8192$ ) in cluttered scenes, where some instances are placed in reasonable locations, as shown in the right panel of Fig. 2. We tag these images as easy or hard based on scene clutter and object occlusion levels. When objects are placed sparsely, we tag the testing images as easy; otherwise, we tag them as hard. Our InsDet dataset also contains 200 high-res background images of indoor scenes (cf. Fig. 2-middle). These indoor scenes are not included in testing images. They allow using the cut-paste-learn framework to synthesize training images [26, 12, 16]. Following this framework, we segment foreground instances using GrabCut [39]
Table 1: Comparison of our dataset to existing ones. Several datasets are used in the InsDet literature although they are designed for different tasks. For example, BigBird and LM are designed to study algorithms of object recognition and object pose estimation, hence they contain instances that are close to the camera. Naively repurposing them for InsDet leads to saturated performance, impoverishing the exploration space of InsDet. Instead, ours is more challenging as instances are placed far from the camera, simulating realistic scenarios where robots must detect instances at a distance. Importantly, our dataset contains far more instances than other publicly available InsDet datasets.
for what task	publicly available	#instances	#scenes	published year	resolution
BigBird [43]	recognition	✓	100	N/A	2014	1280x1024
RGBD [26]	scene label.	✓	300	14	2017	N/A
LM [20]	6D pose est.	✓	15	1	2012	480x640
LM-O [5]	6D pose est.	✓	20	1	2017	480x640
RU-APC [38]	3D pose est.	✓	14	1	2016	480x640
GMU [15]	InsDet	✓	23	9	2016	1080x1920
AVD [1]	InsDet	✓	33	9	2017	1080x1920
Grocery [4]	InsDet	X	100	10	2021	unknown
Ours	InsDet	✓	100	14	2023	6144x8192
Figure 3: Imbalanced distribution of instances in test-set. Yet, instances have the same number of profile images in training and the metrics average over all instances. So, the evaluation is unbiased.
size	bounding box area
small	< 2002
medium	2002 - 4002
large	> 4002
Table 2: Following the spirit of COCO dataset, we tag objects with different sizes by small, medium, and large, respectively.
Figure 4: Distribution of objects w.r.t their bounding box area in testing images. We split them into small, medium, and large subgroups to allow breakdown analysis.
to paste them on background images. It is worth noting that the recent vision foundation model SAM [23] makes interactive segmentation much more efficient. Yet, this work is made public after we collected our dataset. Following the COCO dataset [28], we further tag testing object instances as small, medium, and large according to their bounding box area, as in Table 2. To determine their size tags, we plot the distribution of their sizes in Fig. 4, showing an intuitive way to tag them.
4.1 The Strong Baseline: Cut-Paste-Learn
Cut-Paste-Learn serves as a strong baseline that synthesizes training images with 2D-box annotations [12]. This allows one to train InsDet detectors in the same way as training normal ObjDet detectors, by simply treating the  $K$  unique instances as  $K$  distinct classes. It cuts and pastes foreground instances at various aspect ratios and scales on diverse background images, yielding synthetic training images, as shown in Fig. 5. Cut-paste-learn is model-agnostic, allowing one to adopt any state-of-the-art detector architecture. In this work, we study five popular detectors, covering the two-stage detector FasterRCNN [37], and one-stage anchor-based detector RetinaNet [29], and one-stage anchor-free detectors CenterNet [49], and FCOS [45]; and the transformer-based detector DINO [48]. There are multiple factors in the cut-paste-learn framework, such as the number of inserted objects in each background image, their relative size, the number of generated training images and blending methods. We conduct comprehensive ablation studies and report results using the best-tuned choices. We refer interested readers to the supplement for the ablation studies.
(a) Box
(b) Gaussian blurring
Figure 5: Synthetic training images for cut-paste-learn methods. We use different blending methods to paste object instances on the same background. We recommend that interested readers refer to the supplement for an ablation study using different blending methods.
(c) Motion
(d) Naive pasting
4.2 The Simple, Non-Learned Method
We introduce a simple, non-learned InsDet method by exploiting publicly available pretrained models. This method consists of three main steps: (1) proposal generation on testing images, (2) matching proposals and profile images, (3) selecting the best-matched proposals as the detected instances.
Proposal generation. We use the recently released Segment Anything Model (SAM) [23] to generate proposals. For a proposal, we define a minimum bounding square box encapsulating the masked instance, and then crop the region from the high-resolution testing image. SAM not only achieves high recall (Table 4) on our InsDet dataset but detects objects not belonging to the instances of interest. So the next step is to find interested instances from the proposals.
Feature representation of proposals and profile images. Intuitively, among the pool of proposals, we are interested in those that are well-matched to any profile images of any instance. The well-matched ones are more likely to be predefined instances. To match proposals and profile images, we use off-the-shelf features to represent them. In this work, we study two self-supervised learned models as feature extractors, i.e.  $\mathrm{DINO}_f$  [8], and  $\mathrm{DINOv2}_f$  [33]. We feed a square crop (of a proposal) or a profile image to the feature extractor to obtain its feature representation. We use cosine similarity over the features as the similarity measure between a proposal and a profile image.
Proposal matching and selection. As each instance has multiple profile images, we need to design the similarity between a proposal and an instance. For a proposal, we compute the cosine similarities of its feature to all the profile images of an instance and use the maximum as its final similarity to this instance. We then filter out proposals and instances if they have similarities lower than a threshold, indicating that they are not matched to any instances or proposals. Finally, we obtain a similarity matrix between all remaining proposals and all remaining instances. Over this matrix, we study two matching algorithms to find the best match (hence the final InsDet results), i.e. Rank & Select, and Stable Matching [14, 31]. The former is a greedy algorithm that iteratively selects the best match (highest cosine similarity) between a proposal and an instance and removes the corresponding proposal until no proposal/instance is left. The latter produces an optimal list of matched proposals and instances, such that there exist no pair of instances and proposals which both prefer each other to their current correspondence under the matching.
Synthesizing training images for cut-paste-learn baselines. Our baseline method trains state-of-the-art ObjDet detectors on data synthesized using the cut-paste-learn strategy [12]. For evaluating on our InsDet dataset, we generate 19k training examples and 6k validation examples. For each example, various numbers of foreground objects ranging from 25 to 35 are pasted to a randomly selected background image. The objects are randomly resized with a scale from 0.15 to 0.5. We use four blending options [12], including Gaussian blurring, motion blurring, box blurring, and naive pasting. Fig. 5 shows some random synthetic images. The above factors have a notable impact on the final performance of trained models, and we have conducted a comprehensive ablation study. We refer interested readers to the supplement for the study.
Implementation details. We conduct all the experiments based on open-source implementations, such as Detector2 [47] (for FasterRCNN and RetinaNet), CenterNet [50], FCOS [44] and DINO [48]. The CNN-based end-to-end detectors are initialized with pretrained weights on COCO [28]. We fine-tune CNN-based models using SGD and the transformer-based model using AdamW with a